Open-source Languages & Tools for z/OS

 View Only
Expand all | Collapse all

Miniconda/Conda

  • 1.  Miniconda/Conda

    Posted 10-21-2021 14:15
    Hello,
    I tried to create  a new conda environment separate from the base. It failed as we don't have a support contract (and don't have access to the secure channel).
    I activated the base environment to see if I could get a bit further.
    bash-4.3$ conda activate base
    (base) bash-4.3$
    (base) bash-4.3$ conda info --envs
    # conda environments:
    #
    base * /global/rocket/miniconda

    (base) bash-4.3$ conda install -c https://conda.anaconda.org/zoss-appdev git
    Solving environment: failed
    CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/zoss-appdev/noarch/repodata.json>

    Our z/OS LPAR doesn't have access to the internet.

    How can I install the the package via conda without internet connectivity from z/OS?
    Can I download the package to z/OS (internet to workstation to z/OS) and run conda install?

    thanks

    ------------------------------
    Gary Grossi
    IT Director, Z Systems
    Alight Solutions LLC
    Lincolnshire IL United States
    ------------------------------


  • 2.  RE: Miniconda/Conda

    ROCKETEER
    Posted 10-22-2021 07:27

    Hi Gary,

    Take a look at Vladimir's reply in this thread:
    https://community.rocketsoftware.com/forums/forum-home/digestviewer/view-question?ContributedContentKey=59f5f08a-7f9b-4f6b-8481-a7f3e546dd28&CommunityKey=1e694975-142d-4f2d-9b52-0e37e225db41&tab=digestviewer#8b26efc1-308c-4e8e-b762-3c969fbb93d4

    To summarize, 2 options can be used
    1. Customers on a support contract get a ready-for-use installer that sets up a local channel for them. This described in miniconda UG in  "Deploying a custom channel for Rocket Open AppDev for Z" section
    2. You can download git from https://anaconda.org/zoss-appdev, upload it in binary mode to the LPAR and use 'conda install git-xxxx.tar.bz2' command, but in this case, you should manage dependencies manually, in your case these are bash, curl, expat, libtag, openssl, perl, zlib, so you need to download and install each one too.


    Thanks,
    Alexander



    ------------------------------
    Alexander Klochkov
    QA
    Rocket Internal - All Brands
    Vilnius Lithuania
    ------------------------------



  • 3.  RE: Miniconda/Conda

    Posted 11-04-2021 11:26
    As we don't have mainframe internet access and currently don't have commercial support, I would like to create a local file channel used to perform conda installs.
    How can I do this?
    thanks

    ------------------------------
    Gary Grossi
    IT Director, Z Systems
    Alight Solutions LLC
    Lincolnshire IL United States
    ------------------------------



  • 4.  RE: Miniconda/Conda

    ROCKETEER
    Posted 11-05-2021 10:25
    Edited by Sergey Rezepin 11-05-2021 10:26
    Hi Gary,

    The file channel can be created manually:
    1) Download the required packages from https://anaconda.org/zoss-appdev and copy them on an USS filesystem.
    2) Create the 'channel' directory with 2 subdirectories by platform: 'zos-z' and 'noarch'.
    3) Sort downloaded packages to the subdirectories by platforms.
    4) Run 'conda index' command for both directories: 'conda index channel/zos-z channel/noarch'
    5) Optional: add your local file channel to .condarc file.
    There is a detailed guide with the same steps:
    https://docs.conda.io/projects/conda/en/4.6.0/user-guide/tasks/create-custom-channels.html

    The other way is using an installer for the local file channel. It helps to simplify the process of installation and deployment of the local file channel and contains the latest builds of packages. It is available for customers with Rocket Open AppDev for Z support on Rocket Community portal.

    Thanks,
    Sergey

    ------------------------------
    Sergey Rezepin
    Rocket Software
    ------------------------------



  • 5.  RE: Miniconda/Conda

    Posted 11-07-2021 16:47

    Hello,
    Followed these steps:

    Created directories
    /global/rocket/channel/noarch
    /global/rocket/channel/zos-z

    Downloaded these files to the zos-z directory
    bash-4.3.48-2.tar.bz2  curl-7.73.0-0.tar.bz2  expat-2.2.9-4.tar.bz2
    git-2.26.2-pl524_78.tar.bz2  gzip-1.9-4.tar.bz2  libtag-1.7-0.tar.bz2
    openssl-1.1.1h-12.tar.bz2  perl-5.24.4-4.tar.bz2  sed-4.2.2-12.tar.bz2
    unzip-6.0-12.tar.bz2  vim-8.0.0022-0.tar.bz2  zlib-1.2.11-4.tar.bz2

    Ran conda index on both directories
    These files were added to the zos-z directory 
    repodata.json  repodata.json.bz2

    Tried to run 
    cd /hewitt/rocket/channel
    conda search *git* --channel file://hewitt/rocket/channel/ --override-channels

    Received the error:

    OSError: [Errno 47] errno=47 EDC5047I An invalid file name was specified as a function parameter. (errno2=0xC00B028E), last_op=0, error 90308706 x5620062, abend 1378 x562 reason 62, all'
    `$ /global/rocket/miniconda/bin/conda search *git* --channel file://global/rocket/channel/ --override-channels`

    Then ran:
    cd /hewitt/rocket/channel/zos-z
    conda search *git* --channel file://hewitt/rocket/channel/ --override-channels
    Loading channels: done
    # Name Version Build Channel
    git 2.26.2 pl524_78 channel

    This receives an error (not wildcarding git)
    conda search git --channel file://hewitt/rocket/channel/ --override-channels


    Question
    What is the syntax of the conda install command and what directory will the software be installed?

    Something like this?

    cd /hewitt/rocket/channel/zos-z
    conda install *git* --channel file://hewitt/rocket/channel/ --override-channels


    Thanks,
    Gary



    ------------------------------
    Gary Grossi
    IT Director, Z Systems
    Alight Solutions LLC
    Lincolnshire IL United States
    ------------------------------



  • 6.  RE: Miniconda/Conda

    ROCKETEER
    Posted 11-08-2021 01:24
    Edited by Vladimir Ein 11-08-2021 01:25
    Hello Gary,

    The command would look like this
    conda install git --channel file:///hewitt/rocket/channel/

    It's a bit different from yours:

    1. No need to 'cd' into the channel directory. As long as you specify a correct path, conda will find it no matter what your current directory is.
    2. No asterisks around the package name. No need for a pattern here. Moreover, depending on your directory contents, a pattern might lead to somewhat unexpected results (because the shell will try to substitute it with file names). Since you want to install Git, just type git and that's it.
    3. Three slashes after the 'file' prefix. This is because 'file://' specifies the protocol (just like 'http://' does), and the rest is the path to your channel directory. You need a full path, so starting it with a slash gives the third slash there. This is important - the OSError message your were getting with 'conda search' was most probably caused by having just two slashes in the --channel parameter.
    4. While --override-channels shouldn't break anything, I've removed it because whatever channel you specify on the command line, it takes precedence over the channels in .condarc, and Git will be installed from that local channel anyway. The --override-channels key is useful when you want to prevent conda from searching .condarc channels altogether.


    Also, 'conda search git' should work fine without asterists if you specify the correct path to your local channel (that is, having three slashes).

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Internal - All Brands
    ------------------------------



  • 7.  RE: Miniconda/Conda

    Posted 11-08-2021 14:52
    Hello Vladimir,
    Thanks for the quick response.

    You are right, I was missing the file:/// for the channel.

    When I run the conda search without --override-channels, it looks like it's trying to reach the internet.

    Here is the .condarc file:
    (base) bash-4.3$ cat /hewitt/rocket/miniconda/.condarc
    add_pip_as_python_dependency: False
    channel_alias: https://condaserver.rocketsoftware.com/api/repo
    channels:
    - zoss-appdev

    questions
    1) what is the syntax of the local file channel and where in the .condarc file should it be placed so it's searched first?
    2) is this the correct syntax for adding proxies when we need to reach the internet?

    # Proxy settings: http://[username]:[password]@[server]:[port]
    proxy_servers:
    http: pass@corp.com:8080">http://user:pass@corp.com:8080
    https: pass@corp.com:8080">https://user:pass@corp.com:8080

    3)  where does the software get installed when running conda install git? Is it in the file channel directory?
    4) is there a . script that includes all of the env variables for all of the dependent packages?


    Thanks,
    Gary





    ------------------------------
    Gary Grossi
    IT Director, Z Systems
    Alight Solutions LLC
    Lincolnshire IL United States
    ------------------------------



  • 8.  RE: Miniconda/Conda

    ROCKETEER
    Posted 11-11-2021 14:04
    Hello Gary,

    0) It's trying to reach the internet in order to search the zoss-dev channel configured in your .condarc. If you remove that channel (or comment it out), conda will probably stop going to internet.

    1) You can find a description of the syntax here:
    https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html#config-channels

    You need to put your path first in the 'channels' list for it to be searched first. Just use the same value you placed in the --channel parameter on conda install.

    2) Your proxy sample seems to be somewhat chewed up. It might be that the forum attempted to format it. It's better to wrap such text into a 'preformatted' section; right now it's difficult to tell whether it's right or wrong.

    But anyway, the page about .condarc I mentioned above (in #1) also provides a sample of proxy configuration you can probably use.

    3) It gets installed into the active environment, which is usually a subdirectory in the miniconda/envs directory. It is not the file channel directory. When you do 'conda info --envs' it shows the location for each environment. The 'conda install' command also shows you the environment location before it asks for confirmation.

    4) Conda does not maintain such a script; it configures the environment variables dynamically on 'conda activate' depending on what packages have been installed. In addition to that, some packages may perform additional initialization using activation scripts. You can find more details on this here:

    https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#activating-an-environment

    https://docs.conda.io/projects/conda-build/en/latest/resources/activate-scripts.html

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Internal - All Brands
    ------------------------------



  • 9.  RE: Miniconda/Conda

    Posted 11-12-2021 18:15
    Hello Vladimir,
    After updating the .condarc file with the local file channel, I was able to run search as expected.

    (base) bash-4.3$ conda search git
    Loading channels: done
    # Name Version Build Channel
    git 2.26.2 pl524_78 channel

    Also, successfully ran the install.

    conda install git
    ...
    Downloading and Extracting Packages
    perl-5.24.4 | 16.8 MB | ########## | 100%
    openssl-1.1.1h | 9.7 MB | ########## | 100%
    curl-7.73.0 | 7.9 MB | ########## | 100%
    libtag-1.7 | 39 KB | ########## | 100%
    git-2.26.2 | 65.2 MB | ########## | 100%
    expat-2.2.9 | 201 KB | ########## | 100%
    Preparing transaction: done
    Verifying transaction: done
    Executing transaction: done
    (base) bash-4.3$


    (base) bash-4.3$ git --version
    git version 2.26.2-78


    Running into some encoding issues compared to 2.14.4.

    Example 1  (notice the placement of the git files being first)

    (base) bash-4.3$ cat .gitattributes
    # git files (which must be ASCII or UTF-8)
    .gitattributes zos-working-tree-encoding=iso8859-1
    .gitignore zos-working-tree-encoding=iso8859-1

    # default for text files
    * zos-working-tree-encoding=ibm-1047

    # binary files
    *.jpg binary
    *.png binary
    *.gif binary
    *.zip binary

    (base) bash-4.3$ git add .gitattributes
    fatal: can't add ".gitattributes": file tag (ISO8859-1) does not match its attributes (IBM-1047)


    Example 2  (changed the placement of the git files to second)

    (base) bash-4.3$ cat .gitattributes
    # default for text files
    * zos-working-tree-encoding=ibm-1047

    # git files (which must be ASCII or UTF-8)
    .gitattributes zos-working-tree-encoding=iso8859-1
    .gitignore zos-working-tree-encoding=iso8859-1

    # binary files
    *.jpg binary
    *.png binary
    *.gif binary
    *.zip binary
    #

    The git add .gitattributes works fine.


    Example 3
    Using the same .gitattributes file as in Example 2
    echo hello > file0

    (base) bash-4.3$ ls -T file0
    t ISO8859-1 T=on file0

    fatal: can't add "file0": file tag (ISO8859-1) does not match its attributes (IBM-1047)

    If I touch a file and use vi to update, or just create a new file using vi, the git add works.

    What am I missing?
    Thanks, Gary


    ------------------------------
    Gary Grossi
    IT Director, Z Systems
    Alight Solutions LLC
    Lincolnshire IL US
    ------------------------------



  • 10.  RE: Miniconda/Conda

    ROCKETEER
    Posted 11-14-2021 12:18
    Hello Gary,

    Per official documentation for .gitattributes, when more than one pattern in .gitattributes matches a file, the later match takes precedence:

    When more than one pattern matches the path, a later line overrides an earlier line. This overriding is done per attribute.

    Then, when you add a file using 'git add', Git on z/OS requires the file tag to match the encoding set in zos-working-tree-encoding, if that attribute is specified for the file. This is done on purpose in order to prevent accidental commit of data in an undesired encoding. In the example 1, your .gitattributes  was getting IBM-1047 in zos-working-tree-encoding because of the all-match (*) pattern (but .gitattributes must be in ISO8859-1 anyway, or Git won't be able to read it). As for example 3, bash creates files in ISO8859-1 by default, which again differs from what was specified in zos-working-tree-encoding. You can learn more about file tags in the description of the chtag command.

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    ------------------------------



  • 11.  RE: Miniconda/Conda

    Posted 11-14-2021 15:18
    Hello Vladimir,
    Thanks for the quick response.

    It looks like bash 4.3.46 uses ibm-1047 when creating files compared to 4.3.48 using iso8859-1.

    [@02858@SYSA]$ git --version
    git version 2.14.4_zos_b09

    [@02858@SYSA]$ bash --version
    GNU bash, version 4.3.46(51)-release (i370-ibm-openedition)

    [@02858@SYSA]$ echo hello > G
    [@02858@SYSA]$ ls -T G
    t IBM-1047 T=on G

    --

    (base) bash-4.3$ git --version
    git version 2.26.2-78

    (base) bash-4.3$ bash --version
    GNU bash, version 4.3.48(2)-release (i370-ibm-openedition)

    (base) bash-4.3$ echo hello > GG
    (base) bash-4.3$ ls -T GG
    t ISO8859-1 T=on GG

    Question:
    How do I change the default behavior of bash so that it creates ibm-1047 encoded files?

    I am familiar with chtag  but most/all or our users are not.  In my opinion, it's easier and consistent to to use ibm-1047 as a default and let git do the conversion when/if needed. It also seems to correlate better with the rules in the .gitattributes file.


    cat .gitattributes
    # the pattern matching is done with the latest match in the file

    # default for text files
    * zos-working-tree-encoding=ibm-1047

    # git files (which must be ASCII or UTF-8)
    .gitattributes zos-working-tree-encoding=iso8859-1
    .gitignore zos-working-tree-encoding=iso8859-1

    # binary files
    *.jpg binary
    *.png binary
    *.gif binary
    *.zip binary
    #

    Thanks, Gary

    ------------------------------
    Gary Grossi
    IT Director, Z Systems
    Alight Solutions LLC
    Lincolnshire IL US
    ------------------------------



  • 12.  RE: Miniconda/Conda

    ROCKETEER
    Posted 11-17-2021 13:21
    Hello Gary,

    Encoding may differ depending on where you redirect your output. If you're writing to a file that already exists and is tagged, bash 4.3.48 will use the encoding of that file (as specified by its tag) instead of ISO8859-1.

    $ echo "123" > brand-new-file.txt
    $ touch pre-allocated-file.txt && chtag -tc 1047 pre-allocated-file.txt
    $ echo "123" > pre-allocated-file.txt

    $ ls -T
    t ISO8859-1 T=on brand-new-file.txt
    t IBM-1047 T=on pre-allocated-file.txt

    $ od -h brand-new-file.txt
    0000000000 31 32 33 0A
    0000000004

    $ od -h pre-allocated-file.txt
    0000000000 F1 F2 F3 15
    0000000004

    The od command shows that brand-new-file.txt is in ISO8859-1, whereas the tagged pre-existing file remains in IBM-1047.

    You can change the default output encoding using the environment variable _ENCODING_FILE_NEW like this:

    $ _ENCODE_FILE_NEW=IBM-1047 echo "123" > ebcdic-file.txt

    - or -

    $ export _ENCODE_FILE_NEW=IBM-1047
    $ echo "123" > ebcdic-file.txt

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    ------------------------------



  • 13.  RE: Miniconda/Conda

    Posted 11-17-2021 17:51
    Hello Vladimir,
    Thanks, the env variable solves the issue.

    Here is an example sequence of needing to run chtag:

    export _ENCODE_FILE_NEW=IBM-1047

    # file will not have a code page and be untagged
    touch file1
    - untagged T=off file1

    vi file1 and update
    m IBM-1047 T=off file1

    # vim has an issue with the file until it's tagged (chtag -t file1)
    vim file1


    If vim is used to update a newly touched file, the code set and tag are set properly. vi doesn't seem to do that.
    Is there way around that without chtag?

    Thanks again,
    Gary

    ------------------------------
    Gary Grossi
    IT Director, Z Systems
    Alight Solutions LLC
    Lincolnshire IL US
    ------------------------------



  • 14.  RE: Miniconda/Conda

    ROCKETEER
    Posted 11-26-2021 06:54
    Edited by Vladimir Ein 11-26-2021 07:00
    Hello Gary,

    Unfortunately that's the way vi works. It seems to be opening files in binary mode, which, combined with AUTOTAG flag in _CEE_RUNOPTS, produces files tagged as 'mixed contents'. Mixed-tag files are treated by most tools just like binary files; their contents is not converted from one encoding to another, which is reasonable for files containing binary data. Generally I'd recommend to avoid the 'mixed' tags as much as possible.

    You can deal with this multiple ways, none of them being very convenient though.

    1. Issue chtag right after touch to set the tag to the desired encoding. Tools that open the file will then honor and keep the tag.

    2. Issue chtag after vi, like you did.

    3. Disable autotagging for vi, by calling it like this:

    _CEE_RUNOPTS='FILETAG(AUTOCVT,NOAUTOTAG) POSIX(ON)' vi file1

    You can also define an alias for it.

    alias vi="_CEE_RUNOPTS='FILETAG(AUTOCVT,NOAUTOTAG) POSIX(ON)' vi"

    This way vi will keep the file untagged.

    I'm sure there are other workarounds, these just were first to come to my mind.

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Software Engineer
    Rocket Software
    ------------------------------



  • 15.  RE: Miniconda/Conda

    Posted 11-28-2021 15:39
    Hello Vladimir,
    I like the alias approach with disabling autotagging.

     Do you think IBM would respond to a customer enhancement request to have vi behave like vim (default to IBM-1047 and T=on) ?

    Thanks for the response...much appreciated.
    Gary

    ------------------------------
    Gary Grossi
    IT Director, Z Systems
    Alight Solutions LLC
    Lincolnshire IL US
    ------------------------------



  • 16.  RE: Miniconda/Conda

    ROCKETEER
    Posted 12-02-2021 05:39
    Hello Gary,

    I don't know what IBM's policy is regarding updates to these built-in USS tools (vi and others). The only way to find out is probably to try and open an enhancement request.

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Software Engineer
    Rocket Software
    ------------------------------