Still trying to get an install --- Weeks of frustration is building

Tom Longfellow
Author
Participating Frequently
15 replies
Forum|Forum|3 years ago
January 20, 2022

I am working with z/OS 2.4

I was using the manifest file in order to get all dependencies resolved in one pass.

You have given me some ideas
A) a newer manifest file. I think there is a newer one from Rocket
B) individual or small group installs of key products only.

Unfortunately, I have no clue which package is generating the bash character (unknown command) problem.

I have a vague recollection from earlier threads that if the 'wrong' level of python was in your PATH, bad things could happen.  That is something else that will eat up my working days.    What should be at most a two day project has been a burden to me since I started it over 3 months ago

------------------------------
Tom Longfellow
Senior Systems Programmer
Maryland Judicial Information Systems (JIS)
Annapolis MD US
------------------------------

Another day with no working results.
Ended up reinstalling from scratch. miniconda install worked (eventually --- but I am getting better at this)
Environment create says it works.
conda activate of the new environment worked.

conda install -c https://conda.anaconda.org/zoss-appdev zip unzip

Fails
with
bash: $'\\025': command not found

Again - no clue as to what part of conda is producing the bash error or how I could find it or correct it.
Another damned brick wall

------------------------------
Tom Longfellow
Senior Systems Programmer
Maryland Judicial Information Systems (JIS)
Annapolis MD US
------------------------------

+1

Vladimir Ein
Rocketeer
110 replies
Forum|Forum|3 years ago
January 20, 2022

Another day with no working results.
Ended up reinstalling from scratch. miniconda install worked (eventually --- but I am getting better at this)
Environment create says it works.
conda activate of the new environment worked.

conda install -c https://conda.anaconda.org/zoss-appdev zip unzip

Fails
with
bash: $'\\025': command not found

Again - no clue as to what part of conda is producing the bash error or how I could find it or correct it.
Another damned brick wall

------------------------------
Tom Longfellow
Senior Systems Programmer
Maryland Judicial Information Systems (JIS)
Annapolis MD US
------------------------------

Hello Tom,

Looking at your environment variables in one of the attached files, I can tell you're definitely missing some of the important ones.

First, there's no such thing as _BPX_AUTOCVT - the correct name is _BPXK_AUTOCVT (yes, both BPX and BPXK prefixes exist on z/OS, and are used for different parts of USS; they're not interchangeable of course).

Then, conda requires more than just _BPXK_AUTOCVT... The PDF doc provided with Miniconda (zOS Miniconda Documentation 1.2.1.pdf) explicitly states that:

Each end user who needs access to Miniconda should have following settings in a Bash startup file:

export _BPXK_AUTOCVT=ON
export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
. <miniconda_installation_path>/etc/profile.d/conda.sh

So you need the _CEE_RUNOPTS, too. Without these two variables, most of our Ported Tools aren't going to work properly.

--
Regards,
Vladimir

------------------------------
Vladimir Ein
Software Engineer
Rocket Software
------------------------------

Adam Britt
Participating Frequently
9 replies
Forum|Forum|3 years ago
January 21, 2022

Hello Tom,

Looking at your environment variables in one of the attached files, I can tell you're definitely missing some of the important ones.

First, there's no such thing as _BPX_AUTOCVT - the correct name is _BPXK_AUTOCVT (yes, both BPX and BPXK prefixes exist on z/OS, and are used for different parts of USS; they're not interchangeable of course).

Then, conda requires more than just _BPXK_AUTOCVT... The PDF doc provided with Miniconda (zOS Miniconda Documentation 1.2.1.pdf) explicitly states that:

Each end user who needs access to Miniconda should have following settings in a Bash startup file:

export _BPXK_AUTOCVT=ON
export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
. <miniconda_installation_path>/etc/profile.d/conda.sh

So you need the _CEE_RUNOPTS, too. Without these two variables, most of our Ported Tools aren't going to work properly.

--
Regards,
Vladimir

------------------------------
Vladimir Ein
Software Engineer
Rocket Software
------------------------------

I have noticed that bash on the mainframe outputs non-displayable characters in oct. \\025 in oct is hex 15, which in EBCDIC 037/1047 is the new line character. I think bash wants input in American ascii (or maybe iso8859-1). So sending an EBCDIC newline to bash results in bash interpreting it as a command, which can't be found on the path, which gives the error.

Setting up auto-conversion, as Vladimir suggests, will probably fix the problem.

------------------------------
Adam Martin Britt
IT-Architect
BEC
Randers SV DK
------------------------------

Tom Longfellow
Author
Participating Frequently
15 replies
Forum|Forum|3 years ago
January 24, 2022

I have noticed that bash on the mainframe outputs non-displayable characters in oct. \\025 in oct is hex 15, which in EBCDIC 037/1047 is the new line character. I think bash wants input in American ascii (or maybe iso8859-1). So sending an EBCDIC newline to bash results in bash interpreting it as a command, which can't be found on the path, which gives the error.

Setting up auto-conversion, as Vladimir suggests, will probably fix the problem.

------------------------------
Adam Martin Britt
IT-Architect
BEC
Randers SV DK
------------------------------

Much closer but still no go for installing the entire bundle. This time, after the correct AUTOCVT was specified, we get

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

Traceback (most recent call last):

File "/usr/local/Rocket/miniconda/lib/python3.7/site-packages/conda/exceptions.py", line 819, in __call__

return func(*args, **kwargs)

File "/usr/local/Rocket/miniconda/lib/python3.7/site-packages/conda/cli/main.py", line 78, in _main

exit_code = do_call(args, p)

File "/usr/local/Rocket/miniconda/lib/python3.7/site-packages/conda/cli/conda_argparse.py", line 77, in do_call

exit_code = getattr(module, func_name)(args, parser)

File "/usr/local/Rocket/miniconda/lib/python3.7/site-packages/conda/cli/main_install.py", line 11, in execute

install(args, parser, 'install')

File "/usr/local/Rocket/miniconda/lib/python3.7/site-packages/conda/cli/install.py", line 182, in install

specs.extend(common.specs_from_url(fpath, json=context.json))

File "/usr/local/Rocket/miniconda/lib/python3.7/site-packages/conda/cli/common.py", line 128, in specs_from_url

for line in open(path):

File "/usr/local/Rocket/miniconda/lib/python3.7/codecs.py", line 323, in decode

(result, consumed) = self._buffer_decode(data, self.errors, final)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 5: invalid start byte

`$ /usr/local/Rocket/miniconda/bin/conda install -c https://conda.anaconda.org/zoss-appdev --file /u/tech999/appdev_manifest_1.2.1.txt`

Any ideas??

I do see that UTF-8 has raised its ugly head. ISO8859 must not be good enough. chtag -tc 859 was used during miniconda installation

The entire install output is attached

------------------------------
Tom Longfellow
Senior Systems Programmer
Maryland Judicial Information Systems (JIS)
Annapolis MD US
------------------------------

Tom Longfellow
Author
Participating Frequently
15 replies
Forum|Forum|3 years ago
January 24, 2022

Hello Tom,

Looking at your environment variables in one of the attached files, I can tell you're definitely missing some of the important ones.

First, there's no such thing as _BPX_AUTOCVT - the correct name is _BPXK_AUTOCVT (yes, both BPX and BPXK prefixes exist on z/OS, and are used for different parts of USS; they're not interchangeable of course).

Then, conda requires more than just _BPXK_AUTOCVT... The PDF doc provided with Miniconda (zOS Miniconda Documentation 1.2.1.pdf) explicitly states that:

Each end user who needs access to Miniconda should have following settings in a Bash startup file:

export _BPXK_AUTOCVT=ON
export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
. <miniconda_installation_path>/etc/profile.d/conda.sh

So you need the _CEE_RUNOPTS, too. Without these two variables, most of our Ported Tools aren't going to work properly.

--
Regards,
Vladimir

------------------------------
Vladimir Ein
Software Engineer
Rocket Software
------------------------------

Finally --- SUCCESS ---

The change to the BPXK AUTOCVT variable took me further. BUT a recommendation to others. Avoid installing the manifest files as downloaded. Something in there triggered Utf-8 character conversions when trying to install 'everything'

I pared back to the ones I needed and the install completed. Next will be some testing.

Thanks to everyone for their help.

------------------------------
Tom Longfellow
Senior Systems Programmer
Maryland Judicial Information Systems (JIS)
Annapolis MD US
------------------------------

+1

Vladimir Ein
Rocketeer
110 replies
Forum|Forum|3 years ago
January 25, 2022

Finally --- SUCCESS ---

The change to the BPXK AUTOCVT variable took me further. BUT a recommendation to others. Avoid installing the manifest files as downloaded. Something in there triggered Utf-8 character conversions when trying to install 'everything'

I pared back to the ones I needed and the install completed. Next will be some testing.

Thanks to everyone for their help.

------------------------------
Tom Longfellow
Senior Systems Programmer
Maryland Judicial Information Systems (JIS)
Annapolis MD US
------------------------------

Hello Tom,

First of all I would like to clarify a couple of things.

The manifest files are treated by conda as UTF-8 files. However, our manifests only contain single-byte UTF-8 characters - these characters form a subset that is actually called ASCII, and this set of characters is identical in both UTF-8 and ISO8859-1 (and many other encodings). So, in a certain sense, our manifest files are valid in both UTF-8 and ISO8859-1. There's nothing in there that 'triggers UTF-8 character conversions', because UTF-8 conversion (or, rather, decoding) for manifest files happens always.

That said, as long as conda reads manifest files as valid ASCII text, there will be no UTF-8 validation issues - valid ASCII text is always valid in UTF-8, too. For this to happen on USS:

(1) the file must be tagged according to the encoding of its contents, and
(2) the encoding itself must be eligible for auto-conversion - which limits us to ISO8859-1 and IBM-1047.

Looking at your output with the UTF-8 error, I can guess that (sorry, only a guess):

The manifest file was copied to mainframe over FTP in text mode, which effectively converts the file from ISO8859-1 to IBM-1047. Binary mode is preferable since it retains the file's original encoding.
You then changed the file tag to the code page 859, which is an alias for ISO8859-15, not ISO8859-1. Not sure where 859 came from; the Miniconda doc mentions 819, which is an alias for ISO8859-1. Now, ISO8859-15 is not eligible for automatic conversion, and so the manifest file was read 'as is', in its current IBM-1047 encoding. In this encoding, it's not a valid UTF-8, neither it is a valid ASCII, - hence the UnicodeDecodeError.

Now let's get back to your last message. I would like to reiterate what Tatiana already said in the other thread: we do test our products on a live system ('QA' in her message stands for 'quality assurance'), and this includes the installation process. The manifest files published on the Rocket Community portal can and should be used 'as is'. Please verify your assumptions before giving other users bad advice like that.

Regards,
Vladimir

------------------------------
Vladimir Ein
Software Engineer
Rocket Software
------------------------------

Tom Longfellow
Author
Participating Frequently
15 replies
Forum|Forum|3 years ago
January 25, 2022

Hello Tom,

First of all I would like to clarify a couple of things.

The manifest files are treated by conda as UTF-8 files. However, our manifests only contain single-byte UTF-8 characters - these characters form a subset that is actually called ASCII, and this set of characters is identical in both UTF-8 and ISO8859-1 (and many other encodings). So, in a certain sense, our manifest files are valid in both UTF-8 and ISO8859-1. There's nothing in there that 'triggers UTF-8 character conversions', because UTF-8 conversion (or, rather, decoding) for manifest files happens always.

That said, as long as conda reads manifest files as valid ASCII text, there will be no UTF-8 validation issues - valid ASCII text is always valid in UTF-8, too. For this to happen on USS:

(1) the file must be tagged according to the encoding of its contents, and
(2) the encoding itself must be eligible for auto-conversion - which limits us to ISO8859-1 and IBM-1047.

Looking at your output with the UTF-8 error, I can guess that (sorry, only a guess):

The manifest file was copied to mainframe over FTP in text mode, which effectively converts the file from ISO8859-1 to IBM-1047. Binary mode is preferable since it retains the file's original encoding.
You then changed the file tag to the code page 859, which is an alias for ISO8859-15, not ISO8859-1. Not sure where 859 came from; the Miniconda doc mentions 819, which is an alias for ISO8859-1. Now, ISO8859-15 is not eligible for automatic conversion, and so the manifest file was read 'as is', in its current IBM-1047 encoding. In this encoding, it's not a valid UTF-8, neither it is a valid ASCII, - hence the UnicodeDecodeError.

Now let's get back to your last message. I would like to reiterate what Tatiana already said in the other thread: we do test our products on a live system ('QA' in her message stands for 'quality assurance'), and this includes the installation process. The manifest files published on the Rocket Community portal can and should be used 'as is'. Please verify your assumptions before giving other users bad advice like that.

Regards,
Vladimir

------------------------------
Vladimir Ein
Software Engineer
Rocket Software
------------------------------

Thanks for the education. I was aware that character conversions are an unavoidable thing in the Linux/Unix/PC/EBCDIC world.
So much so, that I have developed bad habits over the years.
I usually use the PC ISO8859 value between mainframes and 'server boxes'. These bad habits have been at the root of all my problems so far. I used BPX (and not BPXK) out of the habit of working with CEE and USS on the mainframe. I used -tc 859 because the echo of ISO8859 was in my head. Also, my FTP client on my PC is geared to treat *.txt as an ISO8859 PC file that needs to be translated to 037/1047 on the mainframe. Overrides are of course possible during the FTP process but it requires monitoring and special ftp client commands.

So, the final result has been a long process, but knowledge has been created for future installs.

Thanks again for helping me through this interesting world of cross system development and Open Source

------------------------------
Tom Longfellow
Senior Systems Programmer
Maryland Judicial Information Systems (JIS)
Annapolis MD US
------------------------------

Sign up

Please log in or register:

Welcome to the Rocket Forum!

Please log in or register:

Scanning file for viruses.

This file cannot be downloaded