Anyone Running RTE in z/OS zCX container?

Hi,

I am hoping someone on here has played with RTE using z/OS zCX containers.    I currently have RTE WE 10.1.5.0065 running successfully under z/OS V2.5 in USS with IBM supplied NODE.JS version 18.14.2.

My question is about running RTE in a z/OS ZCX container as documented in the administrators guide. For the zCX container, I have s390x/node.js V20 docker image downloaded s390x/node - Docker Image | Docker Hub here. However, what is not clear in the admin guide is whether or not I need to use the extracted download image for z/OS USS which is the larger .pax file on your website, or do I use the extracted .tar file instead? 

I think the Linux .tar download is what I need, and have been using to create the docker image.

I actually had the docker build run successfully with the .tar version, but had issues because the first s390x/node.js package was the "slim" version, which i hadnt noticed, and doesnt contain everything needed. RTE startup had errors, and i couldnt connect to it, so I assumed it was the wrong version of node.js. Once I downloaded the full version, and imported that, is when the problems below started.

I ask because the docker build is failing with:

dockerfile:

FROM s390x/node:latest

# Create app directory

RUN mkdir -p /home/rte

# Bundle app source

COPY ./bzwapps /home/rte/bzwapps

# Expose the HTTP port

EXPOSE 8543

# Start the application

WORKDIR /home/rte/bzwapps/bin

CMD [ "sh", "nodeServer-docker.sh" ]


Fails with

admin@ctsatest:/tmp/RocketRTEinstall$ docker build . -t bluezone:latest

Sending build context to Docker daemon 369.2MB

Step 1/6 : FROM s390x/node:latest

 ---> 9350c47e2c80

Step 2/6 : RUN mkdir -p /home/rte

 ---> Running in 22e5e482b372

failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown

Hi Dave, you are correct. zCX container use the .tar version. RTE Web deployment in zCX works for me. It seems the working directory is incorrect. Please check if this path is correct: WORKDIR /home/rte/bzwapps/bin.

WORKDIR /home/rte/bzwapps/bzwapps/bin?


Thanks,
Limei

------------------------------
Limei Wu
Rocket Internal - All Brands
Dalian CN
------------------------------

Limei,

thanks for the clarification on which version of code to use.   I found my problems.   First, I will say that zCX is new territory for me, so learning as I go.  When i installed the required node.js, I did a docker import of the .tar file, instead of a docker load.   Once I got past that, the docker build worked successfully.    The second hurdle where I couldnt connect to the RTE server, was because I did not start the container with the -p option to publish the port.   Once I did that I am able to connect.

Thanks again!

Dave

Dave, great to hear that you have figure this out.

Limei,

If you dont mind, another question.    When a ZCX instance is shutdown, do the containers running inside it come down nicely?  Is there a method to have the RTE shutdown.sh script run automatically at shutdown?    I shutdown ZCX container, and restarted, and it seems as though data was lost or corrupted, because I can no longer logon to the RTE instance, and am having to start over it seems?

In the same thought, is there an automated method to start the container as well?

Thanks in advance.

Hi Dave, when you shutdown the zCX container, the application inside it will not be shutdown but cannot access too. When you start the container again, you can access the application inside it again. If you just shutdown/stop the container and then start it again, the data should not be lost.  How do you shutdown and start the container?

docker stop xx

docker start xx

If you use the docker run command to start the container, it will create a new container and then start it. The data will be lost since it is a new container. To keep the data, you need to mount the data out, for example docker run -p 8543:8543 -v /home/rte/bzwapps/: /opt/bzwapps --name container-name image-name. I believe there are several persistence storage solution for container data, like using volume .etc.

Thank-you Limei,

Your pointers helped a lot.  I did create a volume, and started new container with:

docker run -d -p 8543:8543 --volume bluezone_data:/home/rte/bzwapps --name bluezone --restart unless-stopped bluezone:10.1.5

One other question if you do not mind, but in reading the RTE administrator guide when doing upgrades, i see references for copying the ./deploy folder to ./migrate, and then upgrade.    How would this be accomplished in a docker environment?

Hi Dave, I think you can go to the volumn/mounted data storage, and copy the ./deploy folder to ./migrate and then move to the RTE bzwapps folder. Then start the new version's container with a new volumn. After the upgrading test successfully, you can remove the old volumn data.

Hi Limei,

I'm hoping you can assist me again.   I have an open case I hope you can take a peek at.  It is 00929750.    In summary, I have RTE running in ZCX containers on two different hosts, and am trying to get clustering working.    So far no luck.   I suspect it is a zCX docker configuration issue.   Ive rebuilt the container to expose the http, https, and the cluster listening ports, and started a new container in both locations(same run command, but different IP/host names)

docker run -d --ip 192.168.34.37 -p 8543:8543 -p 8544:8544 -p 8643:8643 -h 192.168.34.37 -v bluezone_deploy:/home/rte/bzwapps/deploy -v bluezone_migrate:/home/rte/bzwapps/migrate  --name bluezone --restart unless-stopped bluezone:10.2.0.1

trying to add the peer from the master fails with: 

Failed to add peer

Would you be willing to give me some pointers on this subject?

Hi Dave, could you check the ipv4 showing in the peer metadata? You can copy it out or click the eye icon to view it. By default, I think it is the container's ip.  So that is why it is failed to add the peer since it cannot access another container's ip. How to communicate between different host's containers? One possible solution is using the overlay network driver.

Overlay networks are distributed networks that span multiple Docker hosts. The network allows all the containers running on any of the hosts to communicate with each other without requiring OS-level routing support.

Here is a reference for this:  https://spacelift.io/blog/docker-networking.

Hi Dave, could you check the ipv4 showing in the peer metadata? You can copy it out or click the eye icon to view it. By default, I think it is the container's ip.  So that is why it is failed to add the peer since it cannot access another container's ip. How to communicate between different host's containers? One possible solution is using the overlay network driver.

Overlay networks are distributed networks that span multiple Docker hosts. The network allows all the containers running on any of the hosts to communicate with each other without requiring OS-level routing support.

Here is a reference for this:  https://spacelift.io/blog/docker-networking.

Hi Dave, could you check the ipv4 showing in the peer metadata? You can copy it out or click the eye icon to view it. By default, I think it is the container's ip.  So that is why it is failed to add the peer since it cannot access another container's ip. How to communicate between different host's containers? One possible solution is using the overlay network driver.

Overlay networks are distributed networks that span multiple Docker hosts. The network allows all the containers running on any of the hosts to communicate with each other without requiring OS-level routing support.

Here is a reference for this:  https://spacelift.io/blog/docker-networking.

MOre questions....  I looked at the doc you referenced.   Can I just create a new SWARM overlay?  I know its somewhat outside the scope of RTE, but you support clustering, its just the ZCX docker portion that is somewhat difficult.   are you able to provide me with any examples/samples to get me going?

Hi Dave, the peer metadata is generated when the RTE Web server starts up. It is getting the container's ip. I am not sure if it can get the host ip. The peers nodes communicate with each other through p2p network to share and access resources directly. I think the docker networking driver can solve this. We also need to investigate and try this. I will talk with our team to see if we can provide some examples.

Thanks,

Limei

Hi Dave, the peer metadata is generated when the RTE Web server starts up. It is getting the container's ip. I am not sure if it can get the host ip. The peers nodes communicate with each other through p2p network to share and access resources directly. I think the docker networking driver can solve this. We also need to investigate and try this. I will talk with our team to see if we can provide some examples.

Thanks,

Limei

Limei,

I just wanted to pass along that I think I have had some success.    I was able to add the node with the meta-data simply by overtyping the 172.18 IP's in the metadata with the host IP's when I added the machine to the cluster.

{"id":"12D3KooWK4CXRvdvzhnnhJ9SAcfw7jgruSrjsHpSsqQcEVmT4dHE","peerId":{"id":"12D3KooWK4CXRvdvzhnnhJ9SAcfw7jgruSrjsHpSsqQcEVmT4dHE","pubKey":"CAESIIlEaO9OBJBLf7pNGFsnZexVqlYKfHzoIFSyrUjHt+bV"},"port":8643,"version":{"major":2,"minor":0,"patch":3},"multiaddrs":["/ip4/127.0.0.1/tcp/8643/p2p/12D3KooWK4CXRvdvzhnnhJ9SAcfw7jgruSrjsHpSsqQcEVmT4dHE","/ip4/172.18.0.3/tcp/8643/p2p/12D3KooWK4CXRvdvzhnnhJ9SAcfw7jgruSrjsHpSsqQcEVmT4dHE"],"serverURL":"http://192.168.34.37:8543","localIp":"172.18.0.3","timestamp":1699027365293}

Wow, that's great, Dave. Could you see if the data syncronization works between clustering nodes? If yes, replacing with host ip is a simple solution too. If any further issue, please let us know.

Thanks,

Limei

Yes, everything sync'd right up.   It would be great if I didnt have to do that, though.  Could this be a considered a "bug" in the RTE code?   Pulling local IP works everwhere except containers, whether on zcx or x86 I presume?