Open-source Languages & Tools for z/OS

ROCKETEER Tatiana Balaburkina posted 11-02-2022 06:40

Hi Gary,

1) Yes, it works as design. Please, check documentation: 

Miniconda is initially installed with a base environment containing a few packages needed for its operation. However it is not recommended to install programs in the base environment. Therefore, it is recommended that you create separate environments to keep programs isolated from each other.

<When you update your Miniconda version> Previously existing environments are not effected by the update. The update only makes changes to the packages in the base environment.

2) Due to License agreement with IBM Python 3.9.5 in a form of conda package is available only for customers on support for Rocket Open AppDev for Z. Unless you are a Rocket customer, you can't install Python 3.9.5 into other environments. 
3) You can try to install boto3 from an open Anaconda channel (e.g., conda-forge) or using pip. Since it is a pure python package, I don't anticipate any installation issues. However, even pure python packages might require z-specific changes and we can't guarantee that packages installed from resources other than Rocket channels are operational.
Pip is a part of Python package and will be updated with it later.

Gary Grossi posted 11-02-2022 09:21

Hello Tatiana,

1)  Thanks for the confirmation regarding the base environment with the update process. I did install bash 5.1 into the base environment after the upgrade.
It's interesting for "end users" they need to run "exec <miniconda_installation_path>/bin/bash" and ". <miniconda_installation_path>/etc/profile.d/conda.sh"
along with a "conda activate  <new env>" .  Isn't all of this from the "base" env? 

Also, I noticed when the base env had bash 4.3 installed and when activating to a new env with bash 5.1 installed, my shell was 4.3.
Would I need to run bash 5 afterwards from the new env?

2)  I understand IBM's Python (now at 3.10)  can be downloaded and pax installed without support. Wouldn't this be OK for Rocket to make it available as a package on zoss-appdev? Is there a way to "copy" a package from one conda env to another?  
It looks like I need pip3 to install the boto3 package and it's not available in my <new env> with Python 3.7.

From base, would I try running conda install  --name <new env> boto3?

Thanks,
Gary

Gary Grossi posted 11-02-2022 15:24

Attached is a file with the latest detailed update.
High-level steps:
1)
(GG) RC=(0) [SYSA] bash-5.1$ pip3 install boto3[crt]
Running setup.py install for awscrt ... error
ERROR: Command errored out with exit status 1:

2)
Afterwards, now receiving this warning...doesn't seem to have an impact. Occurs in all envs.

(base) RC=(0) [SYSA] bash-5.1$ conda list
/newco/rocket/miniconda/lib/python3.9/site-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({0}) or chardet ({1}) doesn't match a supported "
# packages in environment at /newco/rocket/miniconda:

3)
Being in the GG env, I was able to install boto3 (without crt) but was it installed in the base environment. pip3 was used from base.(/newco/rocket/miniconda/bin/pip3)

(GG) RC=(0) [SYSA] bash-5.1$ pip3 install boto3
Collecting boto3
Using cached boto3-1.26.0-py3-none-any.whl (132 kB)
Requirement already satisfied: botocore<1.30.0,>=1.29.0 in /newco/rocket/miniconda/lib/python3.9/site-packages (from boto3) (1.29.0)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /newco/rocket/miniconda/lib/python3.9/site-packages (from boto3) (1.0.1)
Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /newco/rocket/miniconda/lib/python3.9/site-packages (from boto3) (0.6.0)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /newco/rocket/miniconda/lib/python3.9/site-packages (from botocore<1.30.0,>=1.29.0->boto3) (1.26.12)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /newco/rocket/miniconda/lib/python3.9/site-packages (from botocore<1.30.0,>=1.29.0->boto3) (2.7.3)
Requirement already satisfied: six>=1.5 in /newco/rocket/miniconda/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.30.0,>=1.29.0->boto3) (1.16.0)
Installing collected packages: boto3
Successfully installed boto3-1.26.0
WARNING: You are using pip version 21.3.1; however, version 22.3 is available.
You should consider upgrading via the '/newco/rocket/miniconda/bin/python3.9 -m pip install --upgrade pip' command.

4)
(base) RC=(0) [SYSA] bash-5.1$ python3
Python 3.9.5 (heads/pyz_dev-3.9:7cc8dd352f, Nov 2 2021, 05:50:11) on zos
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> quit()


I plan to re-install miniconda to start fresh.

Thanks,
Gary

ROCKETEER Tatiana Balaburkina posted 11-03-2022 09:25

Hi Gary,
1) There are two ways to configure your environment. Both are described in the documentation. You should prefer one of them depending what is your login shell in RACF.
You can have Miniconda installed into your environment but you might want to control its usage. That's why you need to activate an environment.

Peter Fandel posted 11-03-2022 10:34

Hi Gary,

Unfortunately IBM does not permit us to make their Python available for download from anaconda.org.  Rocket would be ok with it, but not IBM.  If you were to get on support with Rocket, then your use case would be possible because we can make it available on Rocket's private conda channel server, and also you would have access to the SMP/E install option.  You would also be more secure in that you would have access to the latest security fixes.

Best regards,
-Peter

Gary Grossi posted 11-03-2022 10:40

Hi Tatiana,
OK, I understand, We use /bin/sh  for each user in their RACF OMVS segment. For us, the approach (at least for now) is to have each user add a .bashrc file to their home directory and after logging in, run an alias we call "bash5" that will run the bash 5.1 shell (<miniconda_installation_path>/bin/bash).  This will drive .bashrc and activate  an environment for them. For now, users who need Python 3.9.5 will need to use it in the base environment until it's available as a conda package for installation in a separate environment.

Using Python 3.9.5 in the base environment, I was able to get the python modules installed that support "boto3" for use with AWS. 
This seemed to work:
python3.9 -m pip install --upgrade pip
pip3 install requests==2.28.1
pip3 install urllib3==1.26.12
pip3 install boto3==1.26.1

Note I tried installing boto3[crt] but it still fails. If you're interested in that, please let me know.

I will let you know how the testing proceeds.

Thanks,
Gary

Gary Grossi posted 11-03-2022 11:12

Hi Peter,
Thanks for the information...much appreciated. Curious, does IBM embrace or support the conda environment?  Is conda geared more for developers and testing versus running in production for interactive and batch users? I'm trying to understand/assess the strategy around all of it. We are doing a POC and trying hard to be successful while using the unsupported tools. You're right, we know the required/compliant advantages of commercial support but while evaluating the public software stack is somewhat difficult for a POC. We should connect on a call and try to better understand how we can perform our evaluation.

Thanks again,
Gary 

Peter Fandel posted 11-03-2022 11:57

IBM does not explicitly embrace conda across the board but uses it "here and there".  Three years ago when Rocket made the decision to adopt conda as a delivery and deployment mechanism it was in response to many complaints from our user community relating to the old system that was ok for downloading and installing one or two tools but burdensome when multiple tools were wanted or in the case of many dependencies (e.g. because Git has four dependencies, installing Git required five download and some 40-50 steps).  At that time IBM had embraced conda for their Open Data Analytics for Z product as well as making use of it on the power platform.  Conda is great for some users (multi-platform developers tend to like it fine) but it is another system to be learned and in particular for those not comfortable with Unix it is an "uphill climb".  For this reason we introduced SMP/E as an option earlier this year.

Gary Grossi posted 11-04-2022 09:26

After running the following updates in the base environment, we were able to successfully run a python script to list AWS S3 buckets.
Updates:
python3.9 -m pip install --upgrade pip
pip3 install requests==2.28.1
pip3 install urllib3==1.26.12
pip3 install boto3==1.26.1

I will start a new thread for getting the AWS Common Runtime...boto3[crt]  module installed.

Thanks,
Gary