My first glance at MiniConda. It appears to be customized on a single system installation concept.
Historically we have installed the OpenSource tools easily into each products own ZFS dataset mounted on the maintenance path. Propagation to all the other systems occurred via DFDSS backup of said ZFS dataset to be restored and renamed as required on each target system.
In simple terms can anyone document how this install/maintain/propagate process could be performed using MiniConda in Z/OS.
I have started reading the User Manuals @ https://docs.conda.io and my initial impression is that it is similar in concept to APT in DEBIAN LINUX.
Additionally what firewall rules would I need to get this working?
Regards
Bruce
------------------------------
Bruce Hewson
Citibank
------------------------------
Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.
The idea behind virtual environments like Conda and Poetry (Python, pyenv) is that you can create a virtual environment for your project with different versions of software, both libraries and the Python intepreter. It's actually much better for application stability but a PITA for global installations. Trade offs - but you can do both. Create a sandbox and roll out an installation wide environment. You have multiple sandboxes with different environments with easily downloadable updates.
Suck it and see. If you don't like it post constructive criticism on here and I'm sure the guys will do their best to help.
------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------
1. Install git and related tools into /home/me/conda/git226
2. test there
3. copy ght git226 directory into a new filesystem (zfs)
4. copy new filesystem to other systems (or share it)
5. on test system mount (RO) it at the /xxx/git mount point for broader validation
6. on production systems mount (RO) it at /xxx/git mount point for production use
7. This requires that the /etc/profile be setup to reference /xxx/git as appropriate
With the pre-conda process I could do this. With miniconda I don't see how to do that (yet).
------------------------------
LionelDyck <><
https://github.com/zigi
https://github.com/lbdyck
------------------------------
The idea behind virtual environments like Conda and Poetry (Python, pyenv) is that you can create a virtual environment for your project with different versions of software, both libraries and the Python intepreter. It's actually much better for application stability but a PITA for global installations. Trade offs - but you can do both. Create a sandbox and roll out an installation wide environment. You have multiple sandboxes with different environments with easily downloadable updates.
Suck it and see. If you don't like it post constructive criticism on here and I'm sure the guys will do their best to help.
------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------
Sites want to:
- e.g. install bash and git and roll out bash to all systems but git to a subset.
- e.g. install an update to bash to fix a CVE and roll out to all systems while an update to git has been rolled out to a few systems
- roll out exactly what is installed on the test environment to the other systems
I'm not working in this area any more, but that is my experience from when I did do this work.
------------------------------
Andrew Rowley
Black Hill Software
------------------------------
- Rocketeer
Virtual environments with different versions of software is good for some types of software development, but I think most z/OS sites want everyone using the same thing. They don't want downloadable updates other than in their specific installation environment.
Sites want to:
I'm not working in this area any more, but that is my experience from when I did do this work.
------------------------------
Andrew Rowley
Black Hill Software
------------------------------
Sites want to:
- e.g. install bash and git and roll out bash to all systems but git to a subset.
- e.g. install an update to bash to fix a CVE and roll out to all systems while an update to git has been rolled out to a few systems
- roll out exactly what is installed on the test environment to the other systems
I'm not working in this area any more, but that is my experience from when I did do this work.
------------------------------
Andrew Rowley
Black Hill Software
------------------------------
No arguments from me Andrew, that's exactly how z/OS software is and should be managed. Conda doesn't prevent you from doing that.
We've had a chat about this internally and the Ported Tools engineering manager will post a reply with a link to a document that describes how to do that.
------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------
+1- Rocketeer
No arguments from me Andrew, that's exactly how z/OS software is and should be managed. Conda doesn't prevent you from doing that.
We've had a chat about this internally and the Ported Tools engineering manager will post a reply with a link to a document that describes how to do that.
------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------
Hi Lionel,
The process that you described can be also used with Miniconda. We have a document that describes copying one installation to different LPARs in details. I want to note that Miniconda is a flexible tool, and it can be adapted to different environments. We didn't want to limit customers to only the maintenance approach. If you feel that you need guidelines on how to adjust Miniconda to your environment, don't hesitate to reach out to us.
How to deploy a specific level of tools to one LPAR?
Let's start with understanding what an environment is. From the perspective of filesystem a conda environment is a directory that contains a specific collection of conda packages that you have installed. During installation process conda installs dependencies and confirms that there are no incompatible packages in the environment, so the end user doesn't need to worry about that.
From the user's perspective, a conda environment is a set of tools which will be used together (not necessarily simultaneously). We would recommend creating a separate environment for test purposes (see Early testing of updates) or when you need 2+ levels of tools on one LPAR.
So how to deploy a specific level of tools to one LPAR? Create an environment, activate it and install required packages (all verbs refer to conda commands).
Which files does miniconda modify?
miniconda-zos-X.Y-yyyy-mm-dd.run is kind of a self-extracting archive, which will be unpacked to the directory specified by user. It is called the Miniconda installation directory. When you install a package into an environment conda will modify an environment directory and a cache directory, which by default are placed into the Miniconda installation directory. Also, conda can create and modify $HOME/.condarc and $HOME/.conda. This file and directory are located outside of the installation directory.
Conda doesn't modify other external files or directories unless the user explicitly specifies that in configuration files, conda command options or any other way.
Note, that the tools that you install with conda can modify external files. This behavior is unrelated to the Miniconda installation approach.
Normally, environments are created within the Miniconda installation directory. Although it's possible to make an environment on a separate zFS, at the moment we don't recommend this approach, because it might lead to the further issues with updates.
Note, that you will need a Miniconda installation on each LPAR where you want to use environments.
Please, let us know if you need further assistance.
------------------------------
Tatiana Balaburkina
Rocket Software
------------------------------
Hi Lionel,
The process that you described can be also used with Miniconda. We have a document that describes copying one installation to different LPARs in details. I want to note that Miniconda is a flexible tool, and it can be adapted to different environments. We didn't want to limit customers to only the maintenance approach. If you feel that you need guidelines on how to adjust Miniconda to your environment, don't hesitate to reach out to us.
How to deploy a specific level of tools to one LPAR?
Let's start with understanding what an environment is. From the perspective of filesystem a conda environment is a directory that contains a specific collection of conda packages that you have installed. During installation process conda installs dependencies and confirms that there are no incompatible packages in the environment, so the end user doesn't need to worry about that.
From the user's perspective, a conda environment is a set of tools which will be used together (not necessarily simultaneously). We would recommend creating a separate environment for test purposes (see Early testing of updates) or when you need 2+ levels of tools on one LPAR.
So how to deploy a specific level of tools to one LPAR? Create an environment, activate it and install required packages (all verbs refer to conda commands).
Which files does miniconda modify?
miniconda-zos-X.Y-yyyy-mm-dd.run is kind of a self-extracting archive, which will be unpacked to the directory specified by user. It is called the Miniconda installation directory. When you install a package into an environment conda will modify an environment directory and a cache directory, which by default are placed into the Miniconda installation directory. Also, conda can create and modify $HOME/.condarc and $HOME/.conda. This file and directory are located outside of the installation directory.
Conda doesn't modify other external files or directories unless the user explicitly specifies that in configuration files, conda command options or any other way.
Note, that the tools that you install with conda can modify external files. This behavior is unrelated to the Miniconda installation approach.
Normally, environments are created within the Miniconda installation directory. Although it's possible to make an environment on a separate zFS, at the moment we don't recommend this approach, because it might lead to the further issues with updates.
Note, that you will need a Miniconda installation on each LPAR where you want to use environments.
Please, let us know if you need further assistance.
------------------------------
Tatiana Balaburkina
Rocket Software
------------------------------
Thoughts/comments/suggestions welcome.
1. conda create -n git226
2. conda activate git226
3. conda install git
4. conda deactivate git226
5. env | sort > before
6. conda activate git226
7. env | sort > after
8. conda deactivate git226
9. diff before after > diff
10. edit diff
- ignore > CONDA .... records
- ignore all records that don't have a > or <
- ignore PS2 and PWD
- ignore SHELL and SHLVL
- capture delta for remaining
- create exports for delta
export LIBPATH=/u/lbdyck/miniconda/envs/git226/lib:$LIBPATH
export MANPATH=/u/lbdyck/miniconda/envs/git226/share/man:/u/lbdyck/miniconda/envs/git226/man:$MANPATH
export PATH=/u/lbdyck/miniconda/envs/git226/bin:$PATH
- adjust the exports for the new path (e.g. /tools/git/)
export LIBPATH=/tools/git/lib:$LIBPATH
export MANPATH=/tools/git/share/man:/tools/git/man:$MANPATH
export PATH=/tools/git/bin:$PATH
11. Create a new zfs (e.g. git226.zfs)
12. mount it at /tools/git/
13. copy (cp) from /u/lbdyck/miniconda/envs/git226 to /tools/git
cp -R /u/lbdyck/miniconda/envs/git226/ /tools/git/
14. Add new exports to /etc/profile
I was hoping to find a document, or file, that explained the customizations required but was unable to do so and the activate script was more than I wanted to review - thus the use of the env command.
------------------------------
LionelDyck <><
https://github.com/zigi
https://github.com/lbdyck
------------------------------
+1- Rocketeer
Thank you for your document - I've developed an approach that seems to be working very well for me now.
Thoughts/comments/suggestions welcome.
1. conda create -n git226
2. conda activate git226
3. conda install git
4. conda deactivate git226
5. env | sort > before
6. conda activate git226
7. env | sort > after
8. conda deactivate git226
9. diff before after > diff
10. edit diff
- ignore > CONDA .... records
- ignore all records that don't have a > or <
- ignore PS2 and PWD
- ignore SHELL and SHLVL
- capture delta for remaining
- create exports for delta
export LIBPATH=/u/lbdyck/miniconda/envs/git226/lib:$LIBPATH
export MANPATH=/u/lbdyck/miniconda/envs/git226/share/man:/u/lbdyck/miniconda/envs/git226/man:$MANPATH
export PATH=/u/lbdyck/miniconda/envs/git226/bin:$PATH
- adjust the exports for the new path (e.g. /tools/git/)
export LIBPATH=/tools/git/lib:$LIBPATH
export MANPATH=/tools/git/share/man:/tools/git/man:$MANPATH
export PATH=/tools/git/bin:$PATH
11. Create a new zfs (e.g. git226.zfs)
12. mount it at /tools/git/
13. copy (cp) from /u/lbdyck/miniconda/envs/git226 to /tools/git
cp -R /u/lbdyck/miniconda/envs/git226/ /tools/git/
14. Add new exports to /etc/profile
I was hoping to find a document, or file, that explained the customizations required but was unable to do so and the activate script was more than I wanted to review - thus the use of the env command.
------------------------------
LionelDyck <><
https://github.com/zigi
https://github.com/lbdyck
------------------------------
Thoughts/comments/suggestions welcome.
1. conda create -n git226
2. conda activate git226
3. conda install git
4. conda deactivate git226
5. env | sort > before
6. conda activate git226
7. env | sort > after
8. conda deactivate git226
9. diff before after > diff
10. edit diff
- ignore > CONDA .... records
- ignore all records that don't have a > or <
- ignore PS2 and PWD
- ignore SHELL and SHLVL
- capture delta for remaining
- create exports for delta
export LIBPATH=/u/lbdyck/miniconda/envs/git226/lib:$LIBPATH
export MANPATH=/u/lbdyck/miniconda/envs/git226/share/man:/u/lbdyck/miniconda/envs/git226/man:$MANPATH
export PATH=/u/lbdyck/miniconda/envs/git226/bin:$PATH
- adjust the exports for the new path (e.g. /tools/git/)
export LIBPATH=/tools/git/lib:$LIBPATH
export MANPATH=/tools/git/share/man:/tools/git/man:$MANPATH
export PATH=/tools/git/bin:$PATH
11. Create a new zfs (e.g. git226.zfs)
12. mount it at /tools/git/
13. copy (cp) from /u/lbdyck/miniconda/envs/git226 to /tools/git
cp -R /u/lbdyck/miniconda/envs/git226/ /tools/git/
14. Add new exports to /etc/profile
I was hoping to find a document, or file, that explained the customizations required but was unable to do so and the activate script was more than I wanted to review - thus the use of the env command.
------------------------------
LionelDyck <><
https://github.com/zigi
https://github.com/lbdyck
------------------------------
My two cents regarding your approach.
1. Steps 1-4 can be optimized a bit:
conda create -n git226 git2. It's quite common for programs on Unix to be tied to their install location. For example, an executable file can contain absolute paths inside that will still refer to its original install location even if you move it to a different place. When you copy /u/lbdyck/miniconda/envs/git226 to /tools/git, some files might still refer to /u/lbdyck/miniconda/envs/git226, and will not work without that directory.
Conda solves this problem by replacing paths within files during 'conda install'.
Some tools, including Git, also provide a way to circumvent this issue by specifying paths in environment variables like GIT_EXEC_PATH. Other tools don't have such a workaround.
Regards,
Vladimir
------------------------------
Vladimir Ein
Rocket Software
------------------------------
Recent badge winners
Sign up
Already have an account? Login
Welcome to the Rocket Forum!
Please log in or register:
Employee Login | Registration Member Login | RegistrationEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.
77 4th Avenue
Waltham, MA 02451 USA
