Miniconda/Conda | Rocket Forum

Hello,
I tried to create a new conda environment separate from the base. It failed as we don't have a support contract (and don't have access to the secure channel).
I activated the base environment to see if I could get a bit further.
bash-4.3$ conda activate base
(base) bash-4.3$
(base) bash-4.3$ conda info --envs
# conda environments:
#
base * /global/rocket/miniconda

(base) bash-4.3$ conda install -c https://conda.anaconda.org/zoss-appdev git
Solving environment: failed
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/zoss-appdev/noarch/repodata.json>

Our z/OS LPAR doesn't have access to the internet.

How can I install the the package via conda without internet connectivity from z/OS?
Can I download the package to z/OS (internet to workstation to z/OS) and run conda install?

thanks

------------------------------
Gary Grossi
IT Director, Z Systems
Alight Solutions LLC
Lincolnshire IL United States
------------------------------

Hi Gary,

Take a look at Vladimir's reply in this thread:
https://community.rocketsoftware.com/forums/forum-home/digestviewer/view-question?ContributedContentKey=59f5f08a-7f9b-4f6b-8481-a7f3e546dd28&CommunityKey=1e694975-142d-4f2d-9b52-0e37e225db41&tab=digestviewer#8b26efc1-308c-4e8e-b762-3c969fbb93d4

To summarize, 2 options can be used
1. Customers on a support contract get a ready-for-use installer that sets up a local channel for them. This described in miniconda UG in "Deploying a custom channel for Rocket Open AppDev for Z" section
2. You can download git from https://anaconda.org/zoss-appdev, upload it in binary mode to the LPAR and use 'conda install git-xxxx.tar.bz2' command, but in this case, you should manage dependencies manually, in your case these are bash, curl, expat, libtag, openssl, perl, zlib, so you need to download and install each one too.

Thanks,
Alexander

------------------------------
Alexander Klochkov
QA
Rocket Internal - All Brands
Vilnius Lithuania
------------------------------

Hi Gary,

Take a look at Vladimir's reply in this thread:
https://community.rocketsoftware.com/forums/forum-home/digestviewer/view-question?ContributedContentKey=59f5f08a-7f9b-4f6b-8481-a7f3e546dd28&CommunityKey=1e694975-142d-4f2d-9b52-0e37e225db41&tab=digestviewer#8b26efc1-308c-4e8e-b762-3c969fbb93d4

To summarize, 2 options can be used
1. Customers on a support contract get a ready-for-use installer that sets up a local channel for them. This described in miniconda UG in "Deploying a custom channel for Rocket Open AppDev for Z" section
2. You can download git from https://anaconda.org/zoss-appdev, upload it in binary mode to the LPAR and use 'conda install git-xxxx.tar.bz2' command, but in this case, you should manage dependencies manually, in your case these are bash, curl, expat, libtag, openssl, perl, zlib, so you need to download and install each one too.

Thanks,
Alexander

------------------------------
Alexander Klochkov
QA
Rocket Internal - All Brands
Vilnius Lithuania
------------------------------

As we don't have mainframe internet access and currently don't have commercial support, I would like to create a local file channel used to perform conda installs.
How can I do this?
thanks

------------------------------
Gary Grossi
IT Director, Z Systems
Alight Solutions LLC
Lincolnshire IL United States
------------------------------

Hi Gary,

The file channel can be created manually:
1) Download the required packages from https://anaconda.org/zoss-appdev and copy them on an USS filesystem.
2) Create the 'channel' directory with 2 subdirectories by platform: 'zos-z' and 'noarch'.
3) Sort downloaded packages to the subdirectories by platforms.
4) Run 'conda index' command for both directories: 'conda index channel/zos-z channel/noarch'
5) Optional: add your local file channel to .condarc file.
There is a detailed guide with the same steps:
https://docs.conda.io/projects/conda/en/4.6.0/user-guide/tasks/create-custom-channels.html

The other way is using an installer for the local file channel. It helps to simplify the process of installation and deployment of the local file channel and contains the latest builds of packages. It is available for customers with Rocket Open AppDev for Z support on Rocket Community portal.

Thanks,
Sergey

------------------------------
Sergey Rezepin
Rocket Software
------------------------------

Hello,
Followed these steps:

Created directories
/global/rocket/channel/noarch
/global/rocket/channel/zos-z

Downloaded these files to the zos-z directory
bash-4.3.48-2.tar.bz2 curl-7.73.0-0.tar.bz2 expat-2.2.9-4.tar.bz2
git-2.26.2-pl524_78.tar.bz2 gzip-1.9-4.tar.bz2 libtag-1.7-0.tar.bz2
openssl-1.1.1h-12.tar.bz2 perl-5.24.4-4.tar.bz2 sed-4.2.2-12.tar.bz2
unzip-6.0-12.tar.bz2 vim-8.0.0022-0.tar.bz2 zlib-1.2.11-4.tar.bz2

Ran conda index on both directories
These files were added to the zos-z directory
repodata.json repodata.json.bz2

Tried to run
cd /hewitt/rocket/channel
conda search *git* --channel file://hewitt/rocket/channel/ --override-channels

Received the error:
OSError: [Errno 47] errno=47 EDC5047I An invalid file name was specified as a function parameter. (errno2=0xC00B028E), last_op=0, error 90308706 x5620062, abend 1378 x562 reason 62, all'
`$ /global/rocket/miniconda/bin/conda search *git* --channel file://global/rocket/channel/ --override-channels`

Then ran:
cd /hewitt/rocket/channel/zos-z
conda search *git* --channel file://hewitt/rocket/channel/ --override-channels
Loading channels: done
# Name Version Build Channel
git 2.26.2 pl524_78 channel

This receives an error (not wildcarding git)
conda search git --channel file://hewitt/rocket/channel/ --override-channels

Question
What is the syntax of the conda install command and what directory will the software be installed?

Something like this?
cd /hewitt/rocket/channel/zos-z
conda install *git* --channel file://hewitt/rocket/channel/ --override-channels

Thanks,
Gary

------------------------------
Gary Grossi
IT Director, Z Systems
Alight Solutions LLC
Lincolnshire IL United States
------------------------------

Hello,
Followed these steps:

Created directories
/global/rocket/channel/noarch
/global/rocket/channel/zos-z

Downloaded these files to the zos-z directory
bash-4.3.48-2.tar.bz2 curl-7.73.0-0.tar.bz2 expat-2.2.9-4.tar.bz2
git-2.26.2-pl524_78.tar.bz2 gzip-1.9-4.tar.bz2 libtag-1.7-0.tar.bz2
openssl-1.1.1h-12.tar.bz2 perl-5.24.4-4.tar.bz2 sed-4.2.2-12.tar.bz2
unzip-6.0-12.tar.bz2 vim-8.0.0022-0.tar.bz2 zlib-1.2.11-4.tar.bz2

Ran conda index on both directories
These files were added to the zos-z directory
repodata.json repodata.json.bz2

Tried to run
cd /hewitt/rocket/channel
conda search *git* --channel file://hewitt/rocket/channel/ --override-channels

Received the error:
OSError: [Errno 47] errno=47 EDC5047I An invalid file name was specified as a function parameter. (errno2=0xC00B028E), last_op=0, error 90308706 x5620062, abend 1378 x562 reason 62, all'
`$ /global/rocket/miniconda/bin/conda search *git* --channel file://global/rocket/channel/ --override-channels`

Then ran:
cd /hewitt/rocket/channel/zos-z
conda search *git* --channel file://hewitt/rocket/channel/ --override-channels
Loading channels: done
# Name Version Build Channel
git 2.26.2 pl524_78 channel

This receives an error (not wildcarding git)
conda search git --channel file://hewitt/rocket/channel/ --override-channels

Question
What is the syntax of the conda install command and what directory will the software be installed?

Something like this?
cd /hewitt/rocket/channel/zos-z
conda install *git* --channel file://hewitt/rocket/channel/ --override-channels

Thanks,
Gary

------------------------------
Gary Grossi
IT Director, Z Systems
Alight Solutions LLC
Lincolnshire IL United States
------------------------------

Hello Gary,

The command would look like this

conda install git --channel file:///hewitt/rocket/channel/

It's a bit different from yours:

No need to 'cd' into the channel directory. As long as you specify a correct path, conda will find it no matter what your current directory is.
No asterisks around the package name. No need for a pattern here. Moreover, depending on your directory contents, a pattern might lead to somewhat unexpected results (because the shell will try to substitute it with file names). Since you want to install Git, just type git and that's it.
Three slashes after the 'file' prefix. This is because 'file://' specifies the protocol (just like 'http://' does), and the rest is the path to your channel directory. You need a full path, so starting it with a slash gives the third slash there. This is important - the OSError message your were getting with 'conda search' was most probably caused by having just two slashes in the --channel parameter.
While --override-channels shouldn't break anything, I've removed it because whatever channel you specify on the command line, it takes precedence over the channels in .condarc, and Git will be installed from that local channel anyway. The --override-channels key is useful when you want to prevent conda from searching .condarc channels altogether.

Also, 'conda search git' should work fine without asterists if you specify the correct path to your local channel (that is, having three slashes).

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Internal - All Brands
------------------------------

Hello Gary,

The command would look like this

conda install git --channel file:///hewitt/rocket/channel/

It's a bit different from yours:

No need to 'cd' into the channel directory. As long as you specify a correct path, conda will find it no matter what your current directory is.
No asterisks around the package name. No need for a pattern here. Moreover, depending on your directory contents, a pattern might lead to somewhat unexpected results (because the shell will try to substitute it with file names). Since you want to install Git, just type git and that's it.
Three slashes after the 'file' prefix. This is because 'file://' specifies the protocol (just like 'http://' does), and the rest is the path to your channel directory. You need a full path, so starting it with a slash gives the third slash there. This is important - the OSError message your were getting with 'conda search' was most probably caused by having just two slashes in the --channel parameter.
While --override-channels shouldn't break anything, I've removed it because whatever channel you specify on the command line, it takes precedence over the channels in .condarc, and Git will be installed from that local channel anyway. The --override-channels key is useful when you want to prevent conda from searching .condarc channels altogether.

Also, 'conda search git' should work fine without asterists if you specify the correct path to your local channel (that is, having three slashes).

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Internal - All Brands
------------------------------

Hello Vladimir,
Thanks for the quick response.

You are right, I was missing the file:/// for the channel.

When I run the conda search without --override-channels, it looks like it's trying to reach the internet.

Here is the .condarc file:
(base) bash-4.3$ cat /hewitt/rocket/miniconda/.condarc
add_pip_as_python_dependency: False
channel_alias: https://condaserver.rocketsoftware.com/api/repo
channels:
- zoss-appdev

questions
1) what is the syntax of the local file channel and where in the .condarc file should it be placed so it's searched first?
2) is this the correct syntax for adding proxies when we need to reach the internet?

# Proxy settings: http://[username]:[password]@[server]:[port]
proxy_servers:
http: http://user:pass@corp.com:8080
https: https://user:pass@corp.com:8080

3) where does the software get installed when running conda install git? Is it in the file channel directory?
4) is there a . script that includes all of the env variables for all of the dependent packages?

Thanks,
Gary

------------------------------
Gary Grossi
IT Director, Z Systems
Alight Solutions LLC
Lincolnshire IL United States
------------------------------

Hello Gary,

0) It's trying to reach the internet in order to search the zoss-dev channel configured in your .condarc. If you remove that channel (or comment it out), conda will probably stop going to internet.

1) You can find a description of the syntax here:
https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html#config-channels

You need to put your path first in the 'channels' list for it to be searched first. Just use the same value you placed in the --channel parameter on conda install.

2) Your proxy sample seems to be somewhat chewed up. It might be that the forum attempted to format it. It's better to wrap such text into a 'preformatted' section; right now it's difficult to tell whether it's right or wrong.

But anyway, the page about .condarc I mentioned above (in #1) also provides a sample of proxy configuration you can probably use.

3) It gets installed into the active environment, which is usually a subdirectory in the miniconda/envs directory. It is not the file channel directory. When you do 'conda info --envs' it shows the location for each environment. The 'conda install' command also shows you the environment location before it asks for confirmation.

4) Conda does not maintain such a script; it configures the environment variables dynamically on 'conda activate' depending on what packages have been installed. In addition to that, some packages may perform additional initialization using activation scripts. You can find more details on this here:

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#activating-an-environment

https://docs.conda.io/projects/conda-build/en/latest/resources/activate-scripts.html

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Internal - All Brands
------------------------------

Hello Vladimir,
After updating the .condarc file with the local file channel, I was able to run search as expected.

(base) bash-4.3$ conda search git
Loading channels: done
# Name Version Build Channel
git 2.26.2 pl524_78 channel

Also, successfully ran the install.

conda install git
...
Downloading and Extracting Packages
perl-5.24.4 | 16.8 MB | ########## | 100%
openssl-1.1.1h | 9.7 MB | ########## | 100%
curl-7.73.0 | 7.9 MB | ########## | 100%
libtag-1.7 | 39 KB | ########## | 100%
git-2.26.2 | 65.2 MB | ########## | 100%
expat-2.2.9 | 201 KB | ########## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(base) bash-4.3$

(base) bash-4.3$ git --version
git version 2.26.2-78

Running into some encoding issues compared to 2.14.4.

Example 1 (notice the placement of the git files being first)

(base) bash-4.3$ cat .gitattributes
# git files (which must be ASCII or UTF-8)
.gitattributes zos-working-tree-encoding=iso8859-1
.gitignore zos-working-tree-encoding=iso8859-1

# default for text files
* zos-working-tree-encoding=ibm-1047

# binary files
*.jpg binary
*.png binary
*.gif binary
*.zip binary

(base) bash-4.3$ git add .gitattributes
fatal: can't add ".gitattributes": file tag (ISO8859-1) does not match its attributes (IBM-1047)

Example 2 (changed the placement of the git files to second)

(base) bash-4.3$ cat .gitattributes
# default for text files
* zos-working-tree-encoding=ibm-1047

# git files (which must be ASCII or UTF-8)
.gitattributes zos-working-tree-encoding=iso8859-1
.gitignore zos-working-tree-encoding=iso8859-1

# binary files
*.jpg binary
*.png binary
*.gif binary
*.zip binary
#

The git add .gitattributes works fine.

Example 3
Using the same .gitattributes file as in Example 2
echo hello > file0

(base) bash-4.3$ ls -T file0
t ISO8859-1 T=on file0

fatal: can't add "file0": file tag (ISO8859-1) does not match its attributes (IBM-1047)

If I touch a file and use vi to update, or just create a new file using vi, the git add works.

What am I missing?
Thanks, Gary

------------------------------
Gary Grossi
IT Director, Z Systems
Alight Solutions LLC
Lincolnshire IL US
------------------------------

Hello Gary,

Per official documentation for .gitattributes, when more than one pattern in .gitattributes matches a file, the later match takes precedence:

When more than one pattern matches the path, a later line overrides an earlier line. This overriding is done per attribute.

Then, when you add a file using 'git add', Git on z/OS requires the file tag to match the encoding set in zos-working-tree-encoding, if that attribute is specified for the file. This is done on purpose in order to prevent accidental commit of data in an undesired encoding. In the example 1, your .gitattributes was getting IBM-1047 in zos-working-tree-encoding because of the all-match (*) pattern (but .gitattributes must be in ISO8859-1 anyway, or Git won't be able to read it). As for example 3, bash creates files in ISO8859-1 by default, which again differs from what was specified in zos-working-tree-encoding. You can learn more about file tags in the description of the chtag command.

Regards,
Vladimir

------------------------------
Vladimir Ein
------------------------------

Hello Gary,

Per official documentation for .gitattributes, when more than one pattern in .gitattributes matches a file, the later match takes precedence:

When more than one pattern matches the path, a later line overrides an earlier line. This overriding is done per attribute.

Then, when you add a file using 'git add', Git on z/OS requires the file tag to match the encoding set in zos-working-tree-encoding, if that attribute is specified for the file. This is done on purpose in order to prevent accidental commit of data in an undesired encoding. In the example 1, your .gitattributes was getting IBM-1047 in zos-working-tree-encoding because of the all-match (*) pattern (but .gitattributes must be in ISO8859-1 anyway, or Git won't be able to read it). As for example 3, bash creates files in ISO8859-1 by default, which again differs from what was specified in zos-working-tree-encoding. You can learn more about file tags in the description of the chtag command.

Regards,
Vladimir

------------------------------
Vladimir Ein
------------------------------

Hello Vladimir,
Thanks for the quick response.

It looks like bash 4.3.46 uses ibm-1047 when creating files compared to 4.3.48 using iso8859-1.

[@02858@SYSA]$ git --version
git version 2.14.4_zos_b09

[@02858@SYSA]$ bash --version
GNU bash, version 4.3.46(51)-release (i370-ibm-openedition)

[@02858@SYSA]$ echo hello > G
[@02858@SYSA]$ ls -T G
t IBM-1047 T=on G

--

(base) bash-4.3$ git --version
git version 2.26.2-78

(base) bash-4.3$ bash --version
GNU bash, version 4.3.48(2)-release (i370-ibm-openedition)

(base) bash-4.3$ echo hello > GG
(base) bash-4.3$ ls -T GG
t ISO8859-1 T=on GG

Question:
How do I change the default behavior of bash so that it creates ibm-1047 encoded files?

I am familiar with chtag but most/all or our users are not. In my opinion, it's easier and consistent to to use ibm-1047 as a default and let git do the conversion when/if needed. It also seems to correlate better with the rules in the .gitattributes file.

cat .gitattributes
# the pattern matching is done with the latest match in the file

# default for text files
* zos-working-tree-encoding=ibm-1047

# git files (which must be ASCII or UTF-8)
.gitattributes zos-working-tree-encoding=iso8859-1
.gitignore zos-working-tree-encoding=iso8859-1

# binary files
*.jpg binary
*.png binary
*.gif binary
*.zip binary
#

Thanks, Gary

------------------------------
Gary Grossi
IT Director, Z Systems
Alight Solutions LLC
Lincolnshire IL US
------------------------------

Hello Gary,

Encoding may differ depending on where you redirect your output. If you're writing to a file that already exists and is tagged, bash 4.3.48 will use the encoding of that file (as specified by its tag) instead of ISO8859-1.

$ echo "123" > brand-new-file.txt
$ touch pre-allocated-file.txt && chtag -tc 1047 pre-allocated-file.txt
$ echo "123" > pre-allocated-file.txt

$ ls -T
t ISO8859-1 T=on brand-new-file.txt
t IBM-1047 T=on pre-allocated-file.txt

$ od -h brand-new-file.txt
0000000000 31 32 33 0A
0000000004

$ od -h pre-allocated-file.txt
0000000000 F1 F2 F3 15
0000000004

The od command shows that brand-new-file.txt is in ISO8859-1, whereas the tagged pre-existing file remains in IBM-1047.

You can change the default output encoding using the environment variable _ENCODING_FILE_NEW like this:

$ _ENCODE_FILE_NEW=IBM-1047 echo "123" > ebcdic-file.txt

- or -

$ export _ENCODE_FILE_NEW=IBM-1047
$ echo "123" > ebcdic-file.txt

Regards,
Vladimir

------------------------------
Vladimir Ein
------------------------------

Hello Gary,

Encoding may differ depending on where you redirect your output. If you're writing to a file that already exists and is tagged, bash 4.3.48 will use the encoding of that file (as specified by its tag) instead of ISO8859-1.

$ echo "123" > brand-new-file.txt
$ touch pre-allocated-file.txt && chtag -tc 1047 pre-allocated-file.txt
$ echo "123" > pre-allocated-file.txt

$ ls -T
t ISO8859-1 T=on brand-new-file.txt
t IBM-1047 T=on pre-allocated-file.txt

$ od -h brand-new-file.txt
0000000000 31 32 33 0A
0000000004

$ od -h pre-allocated-file.txt
0000000000 F1 F2 F3 15
0000000004

The od command shows that brand-new-file.txt is in ISO8859-1, whereas the tagged pre-existing file remains in IBM-1047.

You can change the default output encoding using the environment variable _ENCODING_FILE_NEW like this:

$ _ENCODE_FILE_NEW=IBM-1047 echo "123" > ebcdic-file.txt

- or -

$ export _ENCODE_FILE_NEW=IBM-1047
$ echo "123" > ebcdic-file.txt

Regards,
Vladimir

------------------------------
Vladimir Ein
------------------------------

Hello Vladimir,
Thanks, the env variable solves the issue.

Here is an example sequence of needing to run chtag:

export _ENCODE_FILE_NEW=IBM-1047

# file will not have a code page and be untagged
touch file1
- untagged T=off file1

vi file1 and update
m IBM-1047 T=off file1

# vim has an issue with the file until it's tagged (chtag -t file1)
vim file1

If vim is used to update a newly touched file, the code set and tag are set properly. vi doesn't seem to do that.
Is there way around that without chtag?

Thanks again,
Gary

------------------------------
Gary Grossi
IT Director, Z Systems
Alight Solutions LLC
Lincolnshire IL US
------------------------------

Hello Gary,

Unfortunately that's the way vi works. It seems to be opening files in binary mode, which, combined with AUTOTAG flag in _CEE_RUNOPTS, produces files tagged as 'mixed contents'. Mixed-tag files are treated by most tools just like binary files; their contents is not converted from one encoding to another, which is reasonable for files containing binary data. Generally I'd recommend to avoid the 'mixed' tags as much as possible.

You can deal with this multiple ways, none of them being very convenient though.

1. Issue chtag right after touch to set the tag to the desired encoding. Tools that open the file will then honor and keep the tag.

2. Issue chtag after vi, like you did.

3. Disable autotagging for vi, by calling it like this:

_CEE_RUNOPTS='FILETAG(AUTOCVT,NOAUTOTAG) POSIX(ON)' vi file1

You can also define an alias for it.

alias vi="_CEE_RUNOPTS='FILETAG(AUTOCVT,NOAUTOTAG) POSIX(ON)' vi"

This way vi will keep the file untagged.

I'm sure there are other workarounds, these just were first to come to my mind.

Regards,
Vladimir

------------------------------
Vladimir Ein
Software Engineer
Rocket Software
------------------------------

Hello Gary,

Unfortunately that's the way vi works. It seems to be opening files in binary mode, which, combined with AUTOTAG flag in _CEE_RUNOPTS, produces files tagged as 'mixed contents'. Mixed-tag files are treated by most tools just like binary files; their contents is not converted from one encoding to another, which is reasonable for files containing binary data. Generally I'd recommend to avoid the 'mixed' tags as much as possible.

You can deal with this multiple ways, none of them being very convenient though.

1. Issue chtag right after touch to set the tag to the desired encoding. Tools that open the file will then honor and keep the tag.

2. Issue chtag after vi, like you did.

3. Disable autotagging for vi, by calling it like this:

_CEE_RUNOPTS='FILETAG(AUTOCVT,NOAUTOTAG) POSIX(ON)' vi file1

You can also define an alias for it.

alias vi="_CEE_RUNOPTS='FILETAG(AUTOCVT,NOAUTOTAG) POSIX(ON)' vi"

This way vi will keep the file untagged.

I'm sure there are other workarounds, these just were first to come to my mind.

Regards,
Vladimir

------------------------------
Vladimir Ein
Software Engineer
Rocket Software
------------------------------

Hello Vladimir,
I like the alias approach with disabling autotagging.

Do you think IBM would respond to a customer enhancement request to have vi behave like vim (default to IBM-1047 and T=on) ?

Thanks for the response...much appreciated.
Gary

------------------------------
Gary Grossi
IT Director, Z Systems
Alight Solutions LLC
Lincolnshire IL US
------------------------------

Hello Gary,

I don't know what IBM's policy is regarding updates to these built-in USS tools (vi and others). The only way to find out is probably to try and open an enhancement request.

Regards,
Vladimir

------------------------------
Vladimir Ein
Software Engineer
Rocket Software
------------------------------

Sign up

Please log in or register:

Welcome to the Rocket Forum!

Please log in or register:

Scanning file for viruses.

This file cannot be downloaded