z/OS Tools & Language

Back to discussions
Expand all | Collapse all

Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

  • 1.  Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 12-09-2020 02:11
    My first glance at MiniConda. It appears to be customized on a single system installation concept. 

    Historically we have installed the OpenSource tools easily into each products own ZFS dataset mounted on the maintenance path. Propagation to all the other systems occurred via DFDSS backup of said ZFS dataset to be restored and renamed as required on each target system.

    In simple terms can anyone document how this install/maintain/propagate process could be performed using MiniConda in Z/OS.

    I have started reading the User Manuals @ https://docs.conda.io and my initial impression is that it is similar in concept to APT in DEBIAN LINUX.

    Additionally what firewall rules would I need to get this working?

    Regards
    Bruce

    ------------------------------
    Bruce Hewson
    Citibank
    ------------------------------


  • 2.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 12-09-2020 14:38
    Hello Bruce,

    Not sure if all this qualifies for 'simple terms'...

    As long as installation paths (i.e. mount points) on different systems are the same, you can use your old approach with Miniconda. You can install Miniconda into a separate zFS, then copy that zFS to other LPARs, and mount the copies there. Whenever you need to update the tools, you would do 'conda update' on the master system and then re-deploy the zFS data set to other LPARs again. Simple as that.

    However, certain things need to be thought over before implementing it. In particular, you need to decide whether zFS copies are read-only or if they allow write access. This affects the location of environments created by conda users on your systems - if anyone would want to use conda itself and not just the tools shipped with it.

    1. If write access is allowed, a new environment is by default placed into the 'envs' sub-directory within Miniconda installation, becomes 'globally visible', and can be used or modified by other users.

    The zFS data set must allow extra space for new environments. Also, new environments will be destroyed (and users upset) each time you update zFS data sets.

    2. If Miniconda's directory is read-only, an attempt to create a new environment will place it into the user's home directory under '$HOME/.conda/envs'. Any user can still have personal environments tuned to their wishes; this also allows you to pre-define some 'canned' environments within Miniconda (on the master system) and be sure they remain unmodified on the target LPARs.

    The downside is that configuration (e.g. tailoring files in the 'etc' directory) can only be done on the master system.

    There are other options, e.g. you can tell conda where to create environments by setting the 'envs_dirs' parameter in the .condarc file.


    Another important thing to keep in mind is that tools shipped with Miniconda require an environment to be activated for them to work properly. The 'conda activate' command sets up the shell variables these tools need - mostly PATH and LIBPATH, but many tools have extra requirements. This means, users on all LPARs will need a 'conda activate' command in their profile. This command replaces a bunch of 'export' statements that you needed earlier when you were installing the tools separately.


    Regarding firewall setup, I would imagine you need to allow HTTP/HTTPS access to the conda server if you plan to install packages from the server, and that's it. Conda also supports so-called 'file channels' where you place packages into the local filesystem and therefore can disable internet access at all.

    Unfortunately I'm not familiar with Debian's 'apt' and can't comment on this one, except that package managers in general look somewhat similar.

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Software
    ------------------------------

    Original Message


  • 3.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 12-09-2020 21:48
    Hello Vladimir,

    still trying to gain an understanding on this whole concept.

    I got into this when I cam to download the latest Python 3.7

    Last time it was simple, download the file, install to a specified directory, on which was mounted a ZFS dataset named as Python.V370, update our global /etc/profile, and then we could ship those ZFS datasets out to other Z/OS systems, and IPL. Then everything was completed.  For 100+ systems.

    As Rocket now requires us to use MiniConda to do this process, I am required to massage our existing process, with the complexities built into Miniconda, that hide what I used to do myself, and somehow end up with a simple, foolproof, set of guidelines for the installation and maintenance of the Open Source Tools and Languages supplied by Rocket.

    Nothing I have read so far make it easy for me to make that conversion.

    Sorry about that.

    Hoping the experts at Rocket can assist.

    Regards
    Bruce

    ------------------------------
    Bruce Hewson
    Citibank
    ------------------------------

    Original Message


  • 4.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 12-10-2020 09:50
    Edited by Dave Andrews 12-10-2020 11:06
    While not an answer to all your questions (to which we will follow up in a subsequent post), this 20 minute video may help as an introduction to using conda on z/OS.


    ------------------------------
    Peter Fandel
    Rocket Software
    ------------------------------

    Original Message


  • 5.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 12-15-2020 09:44
    Hello,

    Next step is to make holes in our firewalls.  That is in progress.  But how can I check if my company has a Support Contract so that I can get an ID to access the Rocket Secured Server?

    Once I get to actually install one of the Open Source Languages or Tools via MiniConda, I should be able to look much more closely at what is required for product propagation to all our sites.

    Regards
    Bruce

    ------------------------------
    Bruce Hewson
    Citibank
    ------------------------------

    Original Message


  • 6.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 12 days ago
    I'm trying to wrap my head around this new conda process. I only want to deploy a select package or packages (e.g. git) without all the conda stuff.  How can I do that?  I don't see a way to install a package (with co/pre-requisites) to a specific directory (filesystem) but that would be create if there were one.

    ------------------------------
    Lionel Dyck
    lbdyck
    ------------------------------

    Original Message


  • 7.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 12 days ago
    Hello Lionel,

    'conda create -p /path/to/env' is probably what you're looking for. With this command, you can create an environment in whatever location you need, e.g. on a separate filesystem. Please note, however, that deploying it separately from conda means one would need to manually set up environment variables the same way conda does.

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Software
    ------------------------------

    Original Message


  • 8.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 11 days ago
    I tried the create -p /path and tried to install git to it but i was told it was already installed in a previous environment as I had already installed it but not to that path.

    After installing to a specific path where/how do i find the environment setup?  I trust it is in a file somewhere?

    ------------------------------
    Lionel Dyck
    lbdyck
    ------------------------------

    Original Message


  • 9.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 10 days ago
    Lionel,
    Could you post here exact commands and outputs? It would help us to assist you.

    ------------------------------
    Tatiana Balaburkina
    Rocket Software
    ------------------------------

    Original Message


  • 10.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 10 days ago
    As there is no command history that I can find I am unable to do this.  I will try to recreate.

    ------------------------------
    Lionel Dyck
    lbdyck
    ------------------------------

    Original Message


  • 11.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 12 days ago
    Hi Lionel,

    The old download and install system involved about 40 steps to install Git.  Once you have z/OS Miniconda set up on at least one system, these 40 or so steps are replaced with a single command.  The full advantages of using conda are explained and demonstrated in this video. 

    Rocket has moved to conda both for ease of use as per above and also for security reasons explained in the above-referenced video.

    Many of our customers will separate admin from developer role with respect to download and install whereby admin will install z/OS Miniconda and use it to set up environments which can then be deployed to developer systems obviating the need for developers to learn any conda commands.  Others may install z/OS Miniconda and then allow and/or expect in-house developers to issue conda commands to download what they need.  Still others may expect each developer to set up their own z/OS Miniconda as well as use it for obtaining and deploying open source ports.  Any way you slice it however, conda will be required at some point in the process.

    Another video demonstrates how to install and set up z/OS Miniconda here.

    Regards,
    -Peter

    ------------------------------
    Peter Fandel
    Rocket Software
    ------------------------------

    Original Message


  • 12.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 10 days ago
    I agree Lionel.   I can't tell you how pleased I was when I found out I had to learn yet another way to install a product on z/OS.   Remember the days of SMP/E for IBM products and IEBCOPY for ISVs.  Things were so much simpler back then.    :-)

    ------------------------------
    Gary Freestone
    IBM Australia
    ------------------------------

    Original Message


  • 13.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 10 days ago
    Using the conda create -p .... I was able to create a unique directory for git 2.26 and have it installed. 

    Now - where do I find the information on how to setup my environment to use it? Does miniconda create a file somewhere that I can then copy all/part to place into my target systems /etc/profile ?

    ------------------------------
    Lionel Dyck
    lbdyck
    ------------------------------

    Original Message


  • 14.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 10 days ago
    Gary - agreed.  The old way was much easier to use (imho) as it was granular and easy to document for change management and auditors. With conda it's a black box that doesn't provide the audit history that change management or auditors seek. Conda also hides the environment setup which is something the auditors look at. When I worked at a US government site providing a copy of the /etc/profile and /etc/rc was key to the annual audit process.

    ------------------------------
    Lionel Dyck
    lbdyck
    ------------------------------

    Original Message


  • 15.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 10 days ago
    Hi Lionel,

    Rocket ported tools (incl. conda) are open sourced. Although you can always learn the code, you are absolutely don't need to! If you want to see, which environment variables were modified, you can run the following commands and compare outputs:

    > env | sort #prints out current environment settings

    > conda activate <env_name> #activates an environment and adjusts environment settings

    > env | sort #prints out new environment settings

     You are even not obliged to modify general config files (like /etc/profile). You can run "./<installation_path>/etc/profile.d/conda.sh" command manually, when you want to work with conda in your current session.

    In terms of history, I find conda even much more convenient that the old way. Just one command "conda list" shows you all the installed packages with their versions and build numbers. So you don't need to guess where you should gather information which builds you are using.



    ------------------------------
    Peter Fandel
    Rocket Software
    ------------------------------

    Original Message


  • 16.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 10 days ago
    True to a point - but something that is non-standard and thus known only to the select few for history/etc.

    On the environment I did find them but in the path/conda/activate.d/xxx.sh where there is one for git and one for perl and one for .....

    Using miniconda makes it difficult if I just want to roll out bash or perl compared to the previous installation process. And when installing to a specific path (create -p xxx) there is more than just the requested package - there is conda related directories which I presume can be ignored when deploying beyond the conda environment).

    Miniconda does look to be very nice but it isn't z/OS and makes working with existing change management and audit controls challenging as well as the stated challenges of trying to deploy only a specific package.

    Thank you for your help - at this point I think I've got what I need to proceed and I'm sure as I learn more about miniconda it will become more intuitive and the doc I will have to write up for others, including auditors, will make things less ivory towerish.

    ------------------------------
    Lionel Dyck
    lbdyck
    ------------------------------

    Original Message


  • 17.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 9 days ago
    Hi Lionel,

    We at Rocket are of the opinion that open source will rapidly gain adoption on z/OS and rather than Bash here and Perl there, mainframe users will be deploying much larger numbers of open source tools and making them a critical part of their tool chains and application development infrastructure. 

    We will continue to do our best to support you through this transition.

    Best Regards,
    -Peter

    ------------------------------
    Peter Fandel
    Rocket Software
    ------------------------------

    Original Message


  • 18.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 9 days ago
    Peter - you are correct that many open source tools will become standard but until they are deployed as part of the standard z/OS deployment they will not be pervasive. Witness zowe - IBM realized that it needed to provide a distribution using IBM channels for it to gain a broader audience and level of confidence by the customers. Many smaller shops don't have the sysprog bandwidth to deal with multiple vendors and multiple install methods - granted miniconda helps but it is yet another install tool (yait in open source parlence :) ).

    ------------------------------
    Lionel Dyck
    lbdyck
    ------------------------------

    Original Message


  • 19.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 9 days ago
    Hi Lionel,

    Let's try to get it working for you. You've probably progressed a bit during this conversation, so I'm not sure how's everything going at the moment. Could you please tell us in more detail (if possible) what you're trying to achieve and what are the issues you're currently experiencing?

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Software
    ------------------------------

    Original Message


  • 20.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 9 days ago

    I have the target directory (filesystem) with git and all it's components.  I found the xxx.sh to configure the profile and now need to find the time to collect each of those files into a consolidated file and then add it to my /etc/profile (initially for testing to my ~/.profile.  I hope to get to that this weekend.

     

    Thank you for all your help.

     

    Btw. The xxxx.sh files use a variable for the conda directory and it would be easier in this case if the variable were resolved when creating the xxxx.sh files.

     

    I'll update this weekend or early next week on my progress.

     

    Lionel B. Dyck <sdg><

    Website: https://www.lbdsoftware.com

     

    "Worry more about your character than your reputation.  Character is what you are, reputation merely what others think you are." - John Wooden

     




    Original Message


  • 21.  RE: Z/OS Conda - architectural concepts in a multi-site multi-system environment with a single maintenance build system.

    Posted 9 days ago
    One question about combining those .sh files -

    Do I understand correctly that the reason for this is that for audit / change management, you can only report /etc/profile and not some scripts 'external' to it?

    Won't it work if you just 'source' them from your /etc/profile like this below? Also please keep in mind that PATH, LIBPATH and MANPATH are modified by conda itself and not by those activation scripts.

    export CONDA_PREFIX=/path/to/your/env
    export PATH=$CONDA_PREFIX/bin:$PATH
    export LIBPATH=$CONDA_PREFIX/lib:$LIBPATH
    export MANPATH=$CONDA_PREFIX/share/man:$CONDA_PREFIX/man:$MANPATH

    for i in $CONDA_PREFIX/etc/conda/activate.d/*.sh; do . $i; done

    ------------------------------
    Vladimir Ein
    Rocket Software
    ------------------------------

    Original Message