Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

My first glance at MiniConda. It appears to be customized on a single system installation concept.

Historically we have installed the OpenSource tools easily into each products own ZFS dataset mounted on the maintenance path. Propagation to all the other systems occurred via DFDSS backup of said ZFS dataset to be restored and renamed as required on each target system.

In simple terms can anyone document how this install/maintain/propagate process could be performed using MiniConda in Z/OS.

I have started reading the User Manuals @ https://docs.conda.io and my initial impression is that it is similar in concept to APT in DEBIAN LINUX.

Additionally what firewall rules would I need to get this working?

Regards
Bruce

------------------------------
Bruce Hewson
Citibank
------------------------------

Hello Bruce,

Not sure if all this qualifies for 'simple terms'...

As long as installation paths (i.e. mount points) on different systems are the same, you can use your old approach with Miniconda. You can install Miniconda into a separate zFS, then copy that zFS to other LPARs, and mount the copies there. Whenever you need to update the tools, you would do 'conda update' on the master system and then re-deploy the zFS data set to other LPARs again. Simple as that.

However, certain things need to be thought over before implementing it. In particular, you need to decide whether zFS copies are read-only or if they allow write access. This affects the location of environments created by conda users on your systems - if anyone would want to use conda itself and not just the tools shipped with it.

1. If write access is allowed, a new environment is by default placed into the 'envs' sub-directory within Miniconda installation, becomes 'globally visible', and can be used or modified by other users.

The zFS data set must allow extra space for new environments. Also, new environments will be destroyed (and users upset) each time you update zFS data sets.

2. If Miniconda's directory is read-only, an attempt to create a new environment will place it into the user's home directory under '$HOME/.conda/envs'. Any user can still have personal environments tuned to their wishes; this also allows you to pre-define some 'canned' environments within Miniconda (on the master system) and be sure they remain unmodified on the target LPARs.

The downside is that configuration (e.g. tailoring files in the 'etc' directory) can only be done on the master system.

There are other options, e.g. you can tell conda where to create environments by setting the 'envs_dirs' parameter in the .condarc file.

Another important thing to keep in mind is that tools shipped with Miniconda require an environment to be activated for them to work properly. The 'conda activate' command sets up the shell variables these tools need - mostly PATH and LIBPATH, but many tools have extra requirements. This means, users on all LPARs will need a 'conda activate' command in their profile. This command replaces a bunch of 'export' statements that you needed earlier when you were installing the tools separately.

Regarding firewall setup, I would imagine you need to allow HTTP/HTTPS access to the conda server if you plan to install packages from the server, and that's it. Conda also supports so-called 'file channels' where you place packages into the local filesystem and therefore can disable internet access at all.

Unfortunately I'm not familiar with Debian's 'apt' and can't comment on this one, except that package managers in general look somewhat similar.

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Software
------------------------------

Hello Vladimir,

still trying to gain an understanding on this whole concept.

I got into this when I cam to download the latest Python 3.7

Last time it was simple, download the file, install to a specified directory, on which was mounted a ZFS dataset named as Python.V370, update our global /etc/profile, and then we could ship those ZFS datasets out to other Z/OS systems, and IPL. Then everything was completed. For 100+ systems.

As Rocket now requires us to use MiniConda to do this process, I am required to massage our existing process, with the complexities built into Miniconda, that hide what I used to do myself, and somehow end up with a simple, foolproof, set of guidelines for the installation and maintenance of the Open Source Tools and Languages supplied by Rocket.

Nothing I have read so far make it easy for me to make that conversion.

Sorry about that.

Hoping the experts at Rocket can assist.

Regards
Bruce

------------------------------
Bruce Hewson
Citibank
------------------------------

While not an answer to all your questions (to which we will follow up in a subsequent post), this 20 minute video may help as an introduction to using conda on z/OS.

------------------------------
Peter Fandel
Rocket Software
------------------------------

While not an answer to all your questions (to which we will follow up in a subsequent post), this 20 minute video may help as an introduction to using conda on z/OS.

------------------------------
Peter Fandel
Rocket Software
------------------------------

Hello,

Next step is to make holes in our firewalls. That is in progress. But how can I check if my company has a Support Contract so that I can get an ID to access the Rocket Secured Server?

Once I get to actually install one of the Open Source Languages or Tools via MiniConda, I should be able to look much more closely at what is required for product propagation to all our sites.

Regards
Bruce

------------------------------
Bruce Hewson
Citibank
------------------------------

I'm trying to wrap my head around this new conda process. I only want to deploy a select package or packages (e.g. git) without all the conda stuff. How can I do that? I don't see a way to install a package (with co/pre-requisites) to a specific directory (filesystem) but that would be create if there were one.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Hello Lionel,

'conda create -p /path/to/env' is probably what you're looking for. With this command, you can create an environment in whatever location you need, e.g. on a separate filesystem. Please note, however, that deploying it separately from conda means one would need to manually set up environment variables the same way conda does.

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Software
------------------------------

I'm trying to wrap my head around this new conda process. I only want to deploy a select package or packages (e.g. git) without all the conda stuff. How can I do that? I don't see a way to install a package (with co/pre-requisites) to a specific directory (filesystem) but that would be create if there were one.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Hi Lionel,

The old download and install system involved about 40 steps to install Git. Once you have z/OS Miniconda set up on at least one system, these 40 or so steps are replaced with a single command. The full advantages of using conda are explained and demonstrated in this video.

Rocket has moved to conda both for ease of use as per above and also for security reasons explained in the above-referenced video.

Many of our customers will separate admin from developer role with respect to download and install whereby admin will install z/OS Miniconda and use it to set up environments which can then be deployed to developer systems obviating the need for developers to learn any conda commands. Others may install z/OS Miniconda and then allow and/or expect in-house developers to issue conda commands to download what they need. Still others may expect each developer to set up their own z/OS Miniconda as well as use it for obtaining and deploying open source ports. Any way you slice it however, conda will be required at some point in the process.

Another video demonstrates how to install and set up z/OS Miniconda here.

Regards,
-Peter

------------------------------
Peter Fandel
Rocket Software
------------------------------

Hello Lionel,

'conda create -p /path/to/env' is probably what you're looking for. With this command, you can create an environment in whatever location you need, e.g. on a separate filesystem. Please note, however, that deploying it separately from conda means one would need to manually set up environment variables the same way conda does.

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Software
------------------------------

I tried the create -p /path and tried to install git to it but i was told it was already installed in a previous environment as I had already installed it but not to that path.

After installing to a specific path where/how do i find the environment setup? I trust it is in a file somewhere?

------------------------------
Lionel Dyck
lbdyck
------------------------------

I'm trying to wrap my head around this new conda process. I only want to deploy a select package or packages (e.g. git) without all the conda stuff. How can I do that? I don't see a way to install a package (with co/pre-requisites) to a specific directory (filesystem) but that would be create if there were one.

------------------------------
Lionel Dyck
lbdyck
------------------------------

I agree Lionel. I can't tell you how pleased I was when I found out I had to learn yet another way to install a product on z/OS. Remember the days of SMP/E for IBM products and IEBCOPY for ISVs. Things were so much simpler back then. :-)

------------------------------
Gary Freestone
IBM Australia
------------------------------

I tried the create -p /path and tried to install git to it but i was told it was already installed in a previous environment as I had already installed it but not to that path.

After installing to a specific path where/how do i find the environment setup? I trust it is in a file somewhere?

------------------------------
Lionel Dyck
lbdyck
------------------------------

Lionel,
Could you post here exact commands and outputs? It would help us to assist you.

------------------------------
Tatiana Balaburkina
Rocket Software
------------------------------

As there is no command history that I can find I am unable to do this. I will try to recreate.

------------------------------
Lionel Dyck
lbdyck
------------------------------

I agree Lionel. I can't tell you how pleased I was when I found out I had to learn yet another way to install a product on z/OS. Remember the days of SMP/E for IBM products and IEBCOPY for ISVs. Things were so much simpler back then. :-)

------------------------------
Gary Freestone
IBM Australia
------------------------------

Using the conda create -p .... I was able to create a unique directory for git 2.26 and have it installed.

Now - where do I find the information on how to setup my environment to use it? Does miniconda create a file somewhere that I can then copy all/part to place into my target systems /etc/profile ?

------------------------------
Lionel Dyck
lbdyck
------------------------------

I agree Lionel. I can't tell you how pleased I was when I found out I had to learn yet another way to install a product on z/OS. Remember the days of SMP/E for IBM products and IEBCOPY for ISVs. Things were so much simpler back then. :-)

------------------------------
Gary Freestone
IBM Australia
------------------------------

Gary - agreed. The old way was much easier to use (imho) as it was granular and easy to document for change management and auditors. With conda it's a black box that doesn't provide the audit history that change management or auditors seek. Conda also hides the environment setup which is something the auditors look at. When I worked at a US government site providing a copy of the /etc/profile and /etc/rc was key to the annual audit process.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Hi Lionel,

Rocket ported tools (incl. conda) are open sourced. Although you can always learn the code, you are absolutely don't need to! If you want to see, which environment variables were modified, you can run the following commands and compare outputs:

> env | sort #prints out current environment settings

> conda activate <env_name> #activates an environment and adjusts environment settings

> env | sort #prints out new environment settings

You are even not obliged to modify general config files (like /etc/profile). You can run "./<installation_path>/etc/profile.d/conda.sh" command manually, when you want to work with conda in your current session.

In terms of history, I find conda even much more convenient that the old way. Just one command "conda list" shows you all the installed packages with their versions and build numbers. So you don't need to guess where you should gather information which builds you are using.

------------------------------
Peter Fandel
Rocket Software
------------------------------

Hi Lionel,

Rocket ported tools (incl. conda) are open sourced. Although you can always learn the code, you are absolutely don't need to! If you want to see, which environment variables were modified, you can run the following commands and compare outputs:

> env | sort #prints out current environment settings

> conda activate <env_name> #activates an environment and adjusts environment settings

> env | sort #prints out new environment settings

You are even not obliged to modify general config files (like /etc/profile). You can run "./<installation_path>/etc/profile.d/conda.sh" command manually, when you want to work with conda in your current session.

In terms of history, I find conda even much more convenient that the old way. Just one command "conda list" shows you all the installed packages with their versions and build numbers. So you don't need to guess where you should gather information which builds you are using.

------------------------------
Peter Fandel
Rocket Software
------------------------------

True to a point - but something that is non-standard and thus known only to the select few for history/etc.

On the environment I did find them but in the path/conda/activate.d/xxx.sh where there is one for git and one for perl and one for .....

Using miniconda makes it difficult if I just want to roll out bash or perl compared to the previous installation process. And when installing to a specific path (create -p xxx) there is more than just the requested package - there is conda related directories which I presume can be ignored when deploying beyond the conda environment).

Miniconda does look to be very nice but it isn't z/OS and makes working with existing change management and audit controls challenging as well as the stated challenges of trying to deploy only a specific package.

Thank you for your help - at this point I think I've got what I need to proceed and I'm sure as I learn more about miniconda it will become more intuitive and the doc I will have to write up for others, including auditors, will make things less ivory towerish.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Hi Lionel,

We at Rocket are of the opinion that open source will rapidly gain adoption on z/OS and rather than Bash here and Perl there, mainframe users will be deploying much larger numbers of open source tools and making them a critical part of their tool chains and application development infrastructure.

We will continue to do our best to support you through this transition.

Best Regards,
-Peter

------------------------------
Peter Fandel
Rocket Software
------------------------------

Peter - you are correct that many open source tools will become standard but until they are deployed as part of the standard z/OS deployment they will not be pervasive. Witness zowe - IBM realized that it needed to provide a distribution using IBM channels for it to gain a broader audience and level of confidence by the customers. Many smaller shops don't have the sysprog bandwidth to deal with multiple vendors and multiple install methods - granted miniconda helps but it is yet another install tool (yait in open source parlence :) ).

------------------------------
Lionel Dyck
lbdyck
------------------------------

Hi Lionel,

Let's try to get it working for you. You've probably progressed a bit during this conversation, so I'm not sure how's everything going at the moment. Could you please tell us in more detail (if possible) what you're trying to achieve and what are the issues you're currently experiencing?

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Software
------------------------------

I have the target directory (filesystem) with git and all it's components. I found the xxx.sh to configure the profile and now need to find the time to collect each of those files into a consolidated file and then add it to my /etc/profile (initially for testing to my ~/.profile. I hope to get to that this weekend.

Thank you for all your help.

Btw. The xxxx.sh files use a variable for the conda directory and it would be easier in this case if the variable were resolved when creating the xxxx.sh files.

I'll update this weekend or early next week on my progress.

Lionel B. Dyck <sdg><

Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation. Character is what you are, reputation merely what others think you are." - John Wooden

I have the target directory (filesystem) with git and all it's components. I found the xxx.sh to configure the profile and now need to find the time to collect each of those files into a consolidated file and then add it to my /etc/profile (initially for testing to my ~/.profile. I hope to get to that this weekend.

Thank you for all your help.

Btw. The xxxx.sh files use a variable for the conda directory and it would be easier in this case if the variable were resolved when creating the xxxx.sh files.

I'll update this weekend or early next week on my progress.

Lionel B. Dyck <sdg><

Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation. Character is what you are, reputation merely what others think you are." - John Wooden

One question about combining those .sh files -

Do I understand correctly that the reason for this is that for audit / change management, you can only report /etc/profile and not some scripts 'external' to it?

Won't it work if you just 'source' them from your /etc/profile like this below? Also please keep in mind that PATH, LIBPATH and MANPATH are modified by conda itself and not by those activation scripts.

export CONDA_PREFIX=/path/to/your/env
export PATH=$CONDA_PREFIX/bin:$PATH
export LIBPATH=$CONDA_PREFIX/lib:$LIBPATH
export MANPATH=$CONDA_PREFIX/share/man:$CONDA_PREFIX/man:$MANPATH

for i in $CONDA_PREFIX/etc/conda/activate.d/*.sh; do . $i; done

------------------------------
Vladimir Ein
Rocket Software
------------------------------

I can't speak for Lionel but what I think he wants something similar to how we work at Rocket. He wants to download the tools into a sandbox and them deploy them to a shared file system that can be accessed by multiple LPARs.

------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------

@David Crayford - that is exactly what I'm looking for. Is there a Conda on z/OS best practices doc. Conda looks fine for an individual system, such as a personal computer, but doesn't appear granular enough for an environment of multiple LPARs and/or SYSPLEXes - at least not at 1st (or 2nd) look.

From my reading it appears you can only have one environment active at a time, which means each environment must have all of the tools within it.

The ideal is to be able to deploy release x of tool a and release y of tool b, each in their own filesystems so that either can be replaced by a newer release without impacting the other.

------------------------------
Lionel Dyck
lbdyck
------------------------------

@Lionel Dyck That is a perfectly legitimate requirement. I come from a System Programming background and would expect the same. I'm sure one of the PT guys will chip in with instructions on how we do it at Rocket.

------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------

When I used to do software installations, we had specific requirements for change control. Before changing a system we needed to submit information about exactly what modifications were being made, what were the symptoms of failure (e.g. a change to an environment variable that changed the behavior of something else would qualify here), and detailed instructions someone else could use to back out the change. I am unclear what those instructions would look like with Conda.

Also, many customers have their own procedures that have evolved over decades for migrating software between systems. Often when a vendor had a custom installation system it was necessary to install, analyze what it actually did, tear apart the result and fit the pieces into the existing process. A 40 step process to install git doesn't sound that bad because I understand exactly what changes have been made. My initial impression of the Conda installation is it would actually be more difficult because you can't see exactly what it is doing.

(Having viewed the videos on the installation maybe it isn't so bad - it looks like changes may be local to a directory, with some script changes to point to the updated location. So change documentation would need to list old and new directories, and the script changes.)

------------------------------
Andrew Rowley
Black Hill Software
------------------------------

The idea behind virtual environments like Conda and Poetry (Python, pyenv) is that you can create a virtual environment for your project with different versions of software, both libraries and the Python intepreter. It's actually much better for application stability but a PITA for global installations. Trade offs - but you can do both. Create a sandbox and roll out an installation wide environment. You have multiple sandboxes with different environments with easily downloadable updates.

Suck it and see. If you don't like it post constructive criticism on here and I'm sure the guys will do their best to help.

If you want paid support for the ported tools then I'm sure we can bundle together packages and provide on site support as needed.

------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------

Sign up

Please log in or register:

Welcome to the Rocket Forum!

Please log in or register:

Scanning file for viruses.

This file cannot be downloaded