Thank you for your document - I've developed an approach that seems to be working very well for me now.
Thoughts/comments/suggestions welcome.
1. conda create -n git226
2. conda activate git226
3. conda install git
4. conda deactivate git226
5. env | sort > before
6. conda activate git226
7. env | sort > after
8. conda deactivate git226
9. diff before after > diff
10. edit diff
- ignore > CONDA .... records
- ignore all records that don't have a > or <
- ignore PS2 and PWD
- ignore SHELL and SHLVL
- capture delta for remaining
- create exports for delta
export LIBPATH=/u/lbdyck/miniconda/envs/git226/lib:$LIBPATH
export MANPATH=/u/lbdyck/miniconda/envs/git226/share/man:/u/lbdyck/miniconda/envs/git226/man:$MANPATH
export PATH=/u/lbdyck/miniconda/envs/git226/bin:$PATH
- adjust the exports for the new path (e.g. /tools/git/)
export LIBPATH=/tools/git/lib:$LIBPATH
export MANPATH=/tools/git/share/man:/tools/git/man:$MANPATH
export PATH=/tools/git/bin:$PATH
11. Create a new zfs (e.g. git226.zfs)
12. mount it at /tools/git/
13. copy (cp) from /u/lbdyck/miniconda/envs/git226 to /tools/git
cp -R /u/lbdyck/miniconda/envs/git226/ /tools/git/
14. Add new exports to /etc/profile
I was hoping to find a document, or file, that explained the customizations required but was unable to do so and the activate script was more than I wanted to review - thus the use of the env command.
------------------------------
LionelDyck <><
https://github.com/zigihttps://github.com/lbdyck------------------------------
Original Message:
Sent: 05-05-2021 04:20
From: Tatiana Balaburkina
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Hi Lionel,
The process that you described can be also used with Miniconda. We have a document that describes copying one installation to different LPARs in details. I want to note that Miniconda is a flexible tool, and it can be adapted to different environments. We didn't want to limit customers to only the maintenance approach. If you feel that you need guidelines on how to adjust Miniconda to your environment, don't hesitate to reach out to us.
How to deploy a specific level of tools to one LPAR?
Let's start with understanding what an environment is. From the perspective of filesystem a conda environment is a directory that contains a specific collection of conda packages that you have installed. During installation process conda installs dependencies and confirms that there are no incompatible packages in the environment, so the end user doesn't need to worry about that.
From the user's perspective, a conda environment is a set of tools which will be used together (not necessarily simultaneously). We would recommend creating a separate environment for test purposes (see Early testing of updates) or when you need 2+ levels of tools on one LPAR.
So how to deploy a specific level of tools to one LPAR? Create an environment, activate it and install required packages (all verbs refer to conda commands).
Which files does miniconda modify?
miniconda-zos-X.Y-yyyy-mm-dd.run is kind of a self-extracting archive, which will be unpacked to the directory specified by user. It is called the Miniconda installation directory. When you install a package into an environment conda will modify an environment directory and a cache directory, which by default are placed into the Miniconda installation directory. Also, conda can create and modify $HOME/.condarc and $HOME/.conda. This file and directory are located outside of the installation directory.
Conda doesn't modify other external files or directories unless the user explicitly specifies that in configuration files, conda command options or any other way.
Note, that the tools that you install with conda can modify external files. This behavior is unrelated to the Miniconda installation approach.
Normally, environments are created within the Miniconda installation directory. Although it's possible to make an environment on a separate zFS, at the moment we don't recommend this approach, because it might lead to the further issues with updates.
Note, that you will need a Miniconda installation on each LPAR where you want to use environments.
Please, let us know if you need further assistance.
------------------------------
Tatiana Balaburkina
Rocket Software
Original Message:
Sent: 05-04-2021 22:06
From: David Crayford
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
No arguments from me Andrew, that's exactly how z/OS software is and should be managed. Conda doesn't prevent you from doing that.
We've had a chat about this internally and the Ported Tools engineering manager will post a reply with a link to a document that describes how to do that.
------------------------------
David Crayford
Senior Software Engineer
Rocket Software
Original Message:
Sent: 05-04-2021 19:48
From: Andrew Rowley
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Virtual environments with different versions of software is good for some types of software development, but I think most z/OS sites want everyone using the same thing. They don't want downloadable updates other than in their specific installation environment.
Sites want to:
- e.g. install bash and git and roll out bash to all systems but git to a subset.
- e.g. install an update to bash to fix a CVE and roll out to all systems while an update to git has been rolled out to a few systems
- roll out exactly what is installed on the test environment to the other systems
Installing things independently makes dependencies more complex to manage, but sites are used to doing that already.
I'm not working in this area any more, but that is my experience from when I did do this work.
------------------------------
Andrew Rowley
Black Hill Software
Original Message:
Sent: 05-04-2021 07:52
From: David Crayford
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
The idea behind virtual environments like Conda and Poetry (Python, pyenv) is that you can create a virtual environment for your project with different versions of software, both libraries and the Python intepreter. It's actually much better for application stability but a PITA for global installations. Trade offs - but you can do both. Create a sandbox and roll out an installation wide environment. You have multiple sandboxes with different environments with easily downloadable updates.
Suck it and see. If you don't like it post constructive criticism on here and I'm sure the guys will do their best to help.
If you want paid support for the ported tools then I'm sure we can bundle together packages and provide on site support as needed.
------------------------------
David Crayford
Senior Software Engineer
Rocket Software
Original Message:
Sent: 05-04-2021 07:26
From: Andrew Rowley
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
When I used to do software installations, we had specific requirements for change control. Before changing a system we needed to submit information about exactly what modifications were being made, what were the symptoms of failure (e.g. a change to an environment variable that changed the behavior of something else would qualify here), and detailed instructions someone else could use to back out the change. I am unclear what those instructions would look like with Conda.
Also, many customers have their own procedures that have evolved over decades for migrating software between systems. Often when a vendor had a custom installation system it was necessary to install, analyze what it actually did, tear apart the result and fit the pieces into the existing process. A 40 step process to install git doesn't sound that bad because I understand exactly what changes have been made. My initial impression of the Conda installation is it would actually be more difficult because you can't see exactly what it is doing.
(Having viewed the videos on the installation maybe it isn't so bad - it looks like changes may be local to a directory, with some script changes to point to the updated location. So change documentation would need to list old and new directories, and the script changes.)
------------------------------
Andrew Rowley
Black Hill Software
Original Message:
Sent: 05-04-2021 06:55
From: David Crayford
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
@Lionel Dyck That is a perfectly legitimate requirement. I come from a System Programming background and would expect the same. I'm sure one of the PT guys will chip in with instructions on how we do it at Rocket.
------------------------------
David Crayford
Senior Software Engineer
Rocket Software
Original Message:
Sent: 05-04-2021 06:37
From: Lionel Dyck
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
@David Crayford - that is exactly what I'm looking for. Is there a Conda on z/OS best practices doc. Conda looks fine for an individual system, such as a personal computer, but doesn't appear granular enough for an environment of multiple LPARs and/or SYSPLEXes - at least not at 1st (or 2nd) look.
From my reading it appears you can only have one environment active at a time, which means each environment must have all of the tools within it.
The ideal is to be able to deploy release x of tool a and release y of tool b, each in their own filesystems so that either can be replaced by a newer release without impacting the other.
------------------------------
Lionel Dyck
lbdyck
Original Message:
Sent: 05-04-2021 04:57
From: David Crayford
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
I can't speak for Lionel but what I think he wants is something similar to how we work at Rocket. He wants to download the tools into a sandbox and them deploy them to a shared file system that can be accessed by multiple LPARs.
------------------------------
David Crayford
Senior Software Engineer
Rocket Software
Original Message:
Sent: 01-15-2021 14:41
From: Vladimir Ein
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
One question about combining those .sh files -
Do I understand correctly that the reason for this is that for audit / change management, you can only report /etc/profile and not some scripts 'external' to it?
Won't it work if you just 'source' them from your /etc/profile like this below? Also please keep in mind that PATH, LIBPATH and MANPATH are modified by conda itself and not by those activation scripts.
export CONDA_PREFIX=/path/to/your/env
export PATH=$CONDA_PREFIX/bin:$PATH
export LIBPATH=$CONDA_PREFIX/lib:$LIBPATH
export MANPATH=$CONDA_PREFIX/share/man:$CONDA_PREFIX/man:$MANPATH
for i in $CONDA_PREFIX/etc/conda/activate.d/*.sh; do . $i; done
------------------------------
Vladimir Ein
Rocket Software
Original Message:
Sent: 01-15-2021 10:58
From: Lionel Dyck
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
I have the target directory (filesystem) with git and all it's components. I found the xxx.sh to configure the profile and now need to find the time to collect each of those files into a consolidated file and then add it to my /etc/profile (initially for testing to my ~/.profile. I hope to get to that this weekend.
Thank you for all your help.
Btw. The xxxx.sh files use a variable for the conda directory and it would be easier in this case if the variable were resolved when creating the xxxx.sh files.
I'll update this weekend or early next week on my progress.
Lionel B. Dyck <sdg><
Website: https://www.lbdsoftware.com
"Worry more about your character than your reputation. Character is what you are, reputation merely what others think you are." - John Wooden
Original Message:
Sent: 1/15/2021 10:53:00 AM
From: Vladimir Ein
Subject: RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Hi Lionel,
Let's try to get it working for you. You've probably progressed a bit during this conversation, so I'm not sure how's everything going at the moment. Could you please tell us in more detail (if possible) what you're trying to achieve and what are the issues you're currently experiencing?
Regards,
Vladimir
------------------------------
Vladimir Ein
Rocket Software
Original Message:
Sent: 01-15-2021 08:47
From: Lionel Dyck
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Peter - you are correct that many open source tools will become standard but until they are deployed as part of the standard z/OS deployment they will not be pervasive. Witness zowe - IBM realized that it needed to provide a distribution using IBM channels for it to gain a broader audience and level of confidence by the customers. Many smaller shops don't have the sysprog bandwidth to deal with multiple vendors and multiple install methods - granted miniconda helps but it is yet another install tool (yait in open source parlence :) ).
------------------------------
Lionel Dyck
lbdyck
Original Message:
Sent: 01-15-2021 08:40
From: Peter Fandel
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Hi Lionel,
We at Rocket are of the opinion that open source will rapidly gain adoption on z/OS and rather than Bash here and Perl there, mainframe users will be deploying much larger numbers of open source tools and making them a critical part of their tool chains and application development infrastructure.
We will continue to do our best to support you through this transition.
Best Regards,
-Peter
------------------------------
Peter Fandel
Rocket Software
Original Message:
Sent: 01-14-2021 12:39
From: Lionel Dyck
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
True to a point - but something that is non-standard and thus known only to the select few for history/etc.
On the environment I did find them but in the path/conda/activate.d/xxx.sh where there is one for git and one for perl and one for .....
Using miniconda makes it difficult if I just want to roll out bash or perl compared to the previous installation process. And when installing to a specific path (create -p xxx) there is more than just the requested package - there is conda related directories which I presume can be ignored when deploying beyond the conda environment).
Miniconda does look to be very nice but it isn't z/OS and makes working with existing change management and audit controls challenging as well as the stated challenges of trying to deploy only a specific package.
Thank you for your help - at this point I think I've got what I need to proceed and I'm sure as I learn more about miniconda it will become more intuitive and the doc I will have to write up for others, including auditors, will make things less ivory towerish.
------------------------------
Lionel Dyck
lbdyck
Original Message:
Sent: 01-14-2021 12:05
From: Peter Fandel
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Hi Lionel,
Rocket ported tools (incl. conda) are open sourced. Although you can always learn the code, you are absolutely don't need to! If you want to see, which environment variables were modified, you can run the following commands and compare outputs:
> env | sort #prints out current environment settings
> conda activate <env_name> #activates an environment and adjusts environment settings
> env | sort #prints out new environment settings
You are even not obliged to modify general config files (like /etc/profile). You can run "./<installation_path>/etc/profile.d/conda.sh" command manually, when you want to work with conda in your current session.
In terms of history, I find conda even much more convenient that the old way. Just one command "conda list" shows you all the installed packages with their versions and build numbers. So you don't need to guess where you should gather information which builds you are using.
------------------------------
Peter Fandel
Rocket Software
Original Message:
Sent: 01-14-2021 07:43
From: Lionel Dyck
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Gary - agreed. The old way was much easier to use (imho) as it was granular and easy to document for change management and auditors. With conda it's a black box that doesn't provide the audit history that change management or auditors seek. Conda also hides the environment setup which is something the auditors look at. When I worked at a US government site providing a copy of the /etc/profile and /etc/rc was key to the annual audit process.
------------------------------
Lionel Dyck
lbdyck
Original Message:
Sent: 01-14-2021 00:07
From: Gary Freestone
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
I agree Lionel. I can't tell you how pleased I was when I found out I had to learn yet another way to install a product on z/OS. Remember the days of SMP/E for IBM products and IEBCOPY for ISVs. Things were so much simpler back then. :-)
------------------------------
Gary Freestone
IBM Australia
Original Message:
Sent: 01-12-2021 13:57
From: Lionel Dyck
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
I'm trying to wrap my head around this new conda process. I only want to deploy a select package or packages (e.g. git) without all the conda stuff. How can I do that? I don't see a way to install a package (with co/pre-requisites) to a specific directory (filesystem) but that would be create if there were one.
------------------------------
Lionel Dyck
lbdyck
Original Message:
Sent: 12-15-2020 09:44
From: Bruce Hewson
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Hello,
Next step is to make holes in our firewalls. That is in progress. But how can I check if my company has a Support Contract so that I can get an ID to access the Rocket Secured Server?
Once I get to actually install one of the Open Source Languages or Tools via MiniConda, I should be able to look much more closely at what is required for product propagation to all our sites.
Regards
Bruce
------------------------------
Bruce Hewson
Citibank
Original Message:
Sent: 12-10-2020 09:50
From: Peter Fandel
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
While not an answer to all your questions (to which we will follow up in a subsequent post), this 20 minute video may help as an introduction to using conda on z/OS.
------------------------------
Peter Fandel
Rocket Software
Original Message:
Sent: 12-09-2020 21:47
From: Bruce Hewson
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Hello Vladimir,
still trying to gain an understanding on this whole concept.
I got into this when I cam to download the latest Python 3.7
Last time it was simple, download the file, install to a specified directory, on which was mounted a ZFS dataset named as Python.V370, update our global /etc/profile, and then we could ship those ZFS datasets out to other Z/OS systems, and IPL. Then everything was completed. For 100+ systems.
As Rocket now requires us to use MiniConda to do this process, I am required to massage our existing process, with the complexities built into Miniconda, that hide what I used to do myself, and somehow end up with a simple, foolproof, set of guidelines for the installation and maintenance of the Open Source Tools and Languages supplied by Rocket.
Nothing I have read so far make it easy for me to make that conversion.
Sorry about that.
Hoping the experts at Rocket can assist.
Regards
Bruce
------------------------------
Bruce Hewson
Citibank
Original Message:
Sent: 12-09-2020 14:37
From: Vladimir Ein
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
Hello Bruce,
Not sure if all this qualifies for 'simple terms'...
As long as installation paths (i.e. mount points) on different systems are the same, you can use your old approach with Miniconda. You can install Miniconda into a separate zFS, then copy that zFS to other LPARs, and mount the copies there. Whenever you need to update the tools, you would do 'conda update' on the master system and then re-deploy the zFS data set to other LPARs again. Simple as that.
However, certain things need to be thought over before implementing it. In particular, you need to decide whether zFS copies are read-only or if they allow write access. This affects the location of environments created by conda users on your systems - if anyone would want to use conda itself and not just the tools shipped with it.
1. If write access is allowed, a new environment is by default placed into the 'envs' sub-directory within Miniconda installation, becomes 'globally visible', and can be used or modified by other users.
The zFS data set must allow extra space for new environments. Also, new environments will be destroyed (and users upset) each time you update zFS data sets.
2. If Miniconda's directory is read-only, an attempt to create a new environment will place it into the user's home directory under '$HOME/.conda/envs'. Any user can still have personal environments tuned to their wishes; this also allows you to pre-define some 'canned' environments within Miniconda (on the master system) and be sure they remain unmodified on the target LPARs.
The downside is that configuration (e.g. tailoring files in the 'etc' directory) can only be done on the master system.
There are other options, e.g. you can tell conda where to create environments by setting the 'envs_dirs' parameter in the .condarc file.
Another important thing to keep in mind is that tools shipped with Miniconda require an environment to be activated for them to work properly. The 'conda activate' command sets up the shell variables these tools need - mostly PATH and LIBPATH, but many tools have extra requirements. This means, users on all LPARs will need a 'conda activate' command in their profile. This command replaces a bunch of 'export' statements that you needed earlier when you were installing the tools separately.
Regarding firewall setup, I would imagine you need to allow HTTP/HTTPS access to the conda server if you plan to install packages from the server, and that's it. Conda also supports so-called 'file channels' where you place packages into the local filesystem and therefore can disable internet access at all.
Unfortunately I'm not familiar with Debian's 'apt' and can't comment on this one, except that package managers in general look somewhat similar.
Regards,
Vladimir
------------------------------
Vladimir Ein
Rocket Software
Original Message:
Sent: 12-09-2020 02:10
From: Bruce Hewson
Subject: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
My first glance at MiniConda. It appears to be customized on a single system installation concept.
Historically we have installed the OpenSource tools easily into each products own ZFS dataset mounted on the maintenance path. Propagation to all the other systems occurred via DFDSS backup of said ZFS dataset to be restored and renamed as required on each target system.
In simple terms can anyone document how this install/maintain/propagate process could be performed using MiniConda in Z/OS.
I have started reading the User Manuals @ https://docs.conda.io and my initial impression is that it is similar in concept to APT in DEBIAN LINUX.
Additionally what firewall rules would I need to get this working?
Regards
Bruce
------------------------------
Bruce Hewson
Citibank
------------------------------