Open-source Languages & Tools for z/OS

 View Only
Expand all | Collapse all

Python installation on z/OS filesystem

  • 1.  Python installation on z/OS filesystem

    Posted 04-26-2017 15:48

    I ran into problems downloading the .Tar file to the file system i created on z/OS. I created a file
    system with cyl(5000 150) When i TAR python-2017-04-12-py27.tar it run a long time and eventually
    runs out of space. I cant believe this file is actually this big. How big should my filesystem be?
    I followed your directions on installation of Rocket z/OS open source tools product. I appreciate your assistance.



  • 2.  RE: Python installation on z/OS filesystem

    Posted 04-27-2017 14:50

    This release of Python is, in fact, quite large. The size of download itself (a compressed tar file) is 485,749,868 bytes (call it half a gigabyte…) and the uncompressed, restored file tree’s size is 2,458,780K (call it 2.5GB). Here’s the output of my uncompress/restore:

    bash-2.03$ time gzip -d -c python-2017-04-12-py27.tar.gz  | tar xoUXf -
    real  2m30.392s
    user  0m24.329s
    sys 0m8.110s
    bash-2.03$ du -sk python-2017-04-12-py27
       2458780 python-2017-04-12-py27
    bash-2.03$
    

    The installation instructions are (somewhat unfortunately) contained in a README file in the restored file tree. I’m including them here for completeness.


    This is a release of Python 2.7.13 and Python 3.6.1 for z/OS. There are separate downloads for python 2 and 3, but they were both built in the same process, with the same sources.

    This release of Python contain the packages required for NumPy, SciPy, Pandas, and Jupyter, as well as some add-ons for Jupyter, and several other packages as well.

    The conda packages are not currently usable.

    JupyterHub is not available for Python 2.7.

    StatsModels has not yet been built for Python 3.6.1.

    All other Python packages is this release are built for both 2.7.13 and 3.6.1.

    • This release was built on a z/OS 2.2 system.
    • Everything is built 64 bit and in ASCII mode.
    • This distribution was created on “2017-04-12-12-30”.

    Note! If you did not use xoUXf options in your tar command then installation will fail and you must start over.

    Note! Environment variable _BPXK_AUTOCVT must be set to ON

    Note! If you use tcsh, the export commands below must be edited by replacing every “export” with “setenv”, and replacing every “=” with " ", in your file ~/.tcshrc.

    DOWNLOAD AND UNPACK INSTRUCTIONS

    Choose whether you want to install only Python 2.7.13, only Python 3.6.1, or both. Extra space savings arise if you download the combined release, and your users will be able to choose whether to use an environment that contains python2, python3, or both. The extra savings results because the install procedure uses hard links between files when possible.

    After being unpacked, this release will occupy about 2.4 GB, if you chose either Python 2 or Python 3, or 3.6 GB if you chose both.

    Create a directory for the install, and cd into it.

    Unpack using “gzip -d -c downloadfile.tar.gz | tar xoUXf -”.

    PREREQUISITES

    Before proceeding with installation instructions below you should install and set up the following:

    • bash 4.3 - Used by the utilities for the gzip and xz packages.
    • perl 5.24 - Used by two openssl scripts.

    INSTALLATION INSTRUCTIONS

    The following lines must be edited and executed. You can either execute them in order from the command line or paste them into a script and execute the script.

    export BASH_PREFIX=<the_directory_containing_bin_bash>
    export PERL_PREFIX=<the_directory_containing_bin_perl>
    INSTALL=<your_install_directory>
    export RELEASE_NAME=python-2017-04-12
    export RELEASE_TYPE=py27 # or py36 or both, to match the download you chose
    export RELEASE_DIR=$INSTALL/${RELEASE_NAME}-${RELEASE_TYPE}
    export PKGS_BASE=$RELEASE_DIR/pkgs
    cd $RELEASE_DIR; ls -ld python27 python36 both
    # based on the results of the ls, run one or more of these commands
    cd $RELEASE_DIR/python27; bin/install_all_packages
    cd $RELEASE_DIR/python36; bin/install_all_packages
    cd $RELEASE_DIR/both; bin/install_all_packages
    

    USAGE

    Every user of this distribution should put the following lines at the end of your shell init file:

    If you use sh, put these lines at the end of your ~/.profile:
    If you use bash, put these lines at the end of your ~/.bashrc:

    export _BPXK_AUTOCVT=ON
    INSTALL=<your_install_directory>
    export RELEASE_NAME=python-2017-04-12
    export RELEASE_TYPE=py27 # or py36 or both, to match the download
    export RELEASE_DIR=$INSTALL/${RELEASE_NAME}-${RELEASE_TYPE}
    export PYTHON_ENV=python27 # pr python36 or both, to match your desired environment
    
    export PYTHON_HOME=$RELEASE_DIR/$PYTHON_ENV
    export PATH=$PYTHON_HOME/bin:$PATH
    export LIBPATH=$PYTHON_HOME/lib:$LIBPATH
    export FFI_LIB=$PYTHON_HOME/lib/ffi
    export TERMINFO=$PYTHON_HOME/share/terminfo
    export PKG_CONFIG_PATH=$PYTHON_HOME/lib/pkgconfig:$PYTHON_HOME/share/pkgconfig
    export CURL_CA_BUNDLE=$PYTHON_HOME/etc/ssl/cacert.pem
    

    If you use sh, re-run your init file with:

    . ~/.profile
    

    If you use bash, re-run your init file with:

    . ~/.bashrc
    

    Note: There is a problem with this TERMINFO if you use /bin/vi. You will need to choose one of these approaches:

    • Comment out the line above that sets TERMINFO, and start python with: env TERMINFO=$PYTHON_HOME/share/terminfo python
    • Keep the line above that sets TERMINFO, start vi with: env TERMINFO= vi

    Note: There is code in Python (and elsewhere) that uses /dev/urandom. This code will not work unless the ICSF service is started. If it has not been started, you will get results similar to this:

    head -c40 /dev/urandom
    head: /dev/urandom: EDC5157I An internal error has occurred.


  • 3.  RE: Python installation on z/OS filesystem

    Posted 05-12-2017 01:12

    Hi, Am having trouble with the install. Hoping someone can help.

    I get to the part where you enter bin/install_all_packages
    At this point it returns lines like:
    installing 0 files from /…py36/pkgs/bzip2-1.0.6-1 to /…py36/python36
    pax: ÂÑ>ÂÍ>:Ñø–ÂÑ>Â:Ä/È-ÂÑ>Â:Ä_ø-ÂÑ>Â:ÀÑÃÃ-ÂÑ>Â:ÁÅÊÁø-ÂÑ>Â:ÃÅÊÁø-ÂÑ>Â:ÅÊÁø-ÂÑ>Â

    I saw the notes that say “Note! If you did not use xoUXf options in your tar command then installation will fail and you must start over.”, I had actually not used those options, so I started again and used them. Same result.

    Saw the “Note! Environment variable _BPXK_AUTOCVT must be set to ON”, so I set that, and tried again. Same result.

    I did some digging and see that there is also the “install_package” which does a single package. Do you have the syntax for issuing this on its own, so that I can test my process on a single package rather than the complete list?

    Thanks



  • 4.  RE: Python installation on z/OS filesystem

    Posted 03-20-2019 12:34

    Hey there. I’m very new to installing this kind of product so please be gentle
    I was having problems with trying to do the gzip and tar in one command using “gzip -d -c downloadfile.tar.gz | tar xoUXf -”. I was able to gzip the file but now I’m not sure of what the tar command should look like. Currently, I have the python-2017-04-12-py27.tar file in /u/ROCKET/python/py27. When all is said and done, that is where I want the product to live.
    Any guidance would be greatly appreciated!
    Thanks,
    Debbi



  • 5.  RE: Python installation on z/OS filesystem

    Posted 05-18-2017 21:56

    I got past my problem above. The trick is the _BPXK_AUTOCVT parameter.

    In your notes it says “Note! Environment variable _BPXK_AUTOCVT must be set to ON”. Well you are dealing with people who may not know the difference between “EXPORT _BPXK_AUTOCVT=ON” and “_BPXK_AUTOCVT=ON”. Both set the variable, but the scope is different. For example you say “INSTALL=<your_install_directory>” but your other lines say “EXPORT RELEASE…”. As a guide to noobs like me, you should make your installation documentation as idiot proof as possible.

    On the other hand, I have learnt a lot about USS and shell scripts. I also found a program to convert between ASCII and EBCDIC which is “dd conv=ebcdic if={input file} of={output file}” IBM supplies it with the standard distribution.

    Good luck to other installers of Python 3.6.1. Can’t wait to start using it.



  • 6.  RE: Python installation on z/OS filesystem

    Posted 05-22-2017 05:33

    Hi,

    Thank you for feedback. We will try to make our guides more understandable.
    Also you can use iconv to convert between different code set.



  • 7.  RE: Python installation on z/OS filesystem

    Posted 06-02-2017 00:14

    Hi

    I have made some progress with my installation.

    First I got error message: Fatal Python error: failed to get random numbers to initialize Python

    I did a lot of work to start ICSF (we do not have the hardware installed, but it seems that just running the STC is sufficient).

    When I start again I got the following:
    ModuleNotFoundError: No module named 'encodings’
    initstdio: failed to import module encodings.utf_8
    Fatal Python error: Py_Initialize: can’t initialize sys standard streams

    Current thread 0x174ce00000000000 (most recent call first):
    äáá----á-èÇÁ-ËÑÅ>/%-ëñå âêè-Ï/Ë-ÊÁÄÁÑÎÁÀ–[1] + Done(131) python 33555310 Abort
    /usr/lpp/python/python-2017-04-12-py36/python36/bin/python

    I did some work on the rubbish characters and assuming they were ASCII the EBCDIC equivalent is “CEE____E_The_signal_SIGABRT_was_received”

    Any ideas anyone?



  • 8.  RE: Python installation on z/OS filesystem

    Posted 06-05-2017 04:53

    Hi,

    Could you give more details about this issue? When does this error occur:

    Fatal Python error: failed to get random numbers to initialize Python

    What is next?
    How do you get this error? Do you run “python” (interactive mode) or “python your_file”:

    ModuleNotFoundError: No module named 'encodings’
    initstdio: failed to import module encodings.utf_8
    Fatal Python error: Py_Initialize: can’t initialize sys standard streams



  • 9.  RE: Python installation on z/OS filesystem

    Posted 06-06-2017 02:11

    Hi Tatyana,

    Glad to share with you some more information…

    I get into the OMVS shell and this conversation follows
    # echo $PATH
    /usr/lpp/python/python-2017-04-12-py36/python36/bin:/usr/lpp/java/current/bin:/bin
    # head -c8 /dev/urandom
    head: /dev/urandom: EDC5157I An internal error has occurred.
    # python
    Fatal Python error: failed to get random numbers to initialize Python
    äáá----á-èÇÁ-ËÑÅ>/%-ëñå âêè-Ï/Ë-ÊÁÄÁÑÎÁÀ–[1] + Done(131) python
    33554739Abort /usr/lpp/python/python-2017-04-12-py36/python36/bin/python
    So at this point I issue “S CSF” on the master console and get message "CSFM126I CRYPTOGRAPHY - FULL CPU-BASED SERVICES ARE AVAILABLE. "

    Back into OMVS and
    # head -c8 /dev/urandom
    Ê -¸-pó,
    # python
    ModuleNotFoundError: No module named 'encodings’
    initstdio: failed to import module encodings.utf_8
    Fatal Python error: Py_Initialize: can’t initialize sys standard streams

    Current thread 0x174cca0000000000 (most recent call first):
    äáá----á-èÇÁ-ËÑÅ>/%-ëñå âêè-Ï/Ë-ÊÁÄÁÑÎÁÀ–[1] + Done(131) python
    50331957Abort /usr/lpp/python/python-2017-04-12-py36/python36/bin/python

    I have the following settings in my environment …
    # env
    FFI_LIB=/usr/lpp/python/python-2017-04-12-py36/python36/lib/ffi
    PYTHON_ENV=python36
    MAIL=/usr/mail/DXC00
    _BPX_TERMPATH=OMVS
    PATH=/usr/lpp/python/python-2017-04-12-py36/python36/bin:/usr/lpp/java/current/bin:/bin
    _BPXK_AUTOCVT=ON
    SHELL=/bin/sh
    CURL_CA_BUNDLE=/usr/lpp/python/python-2017-04-12-py36/python36/etc/ssl/cacert.pem
    PYTHON_HOME=/usr/lpp/python/python-2017-04-12-py36/python36
    RELEASE_NAME=python-2017-04-12
    COLUMNS=80
    PYTHONHOME=/usr/lpp/python/bin
    _=/bin/env
    CLASSPATH=.:/usr/lpp/smp/classes:/usr/lpp/java/current:
    LOGNAME=DXC00
    STEPLIB=none
    LANG=C
    PKG_CONFIG_PATH=/usr/lpp/python/python-2017-04-12-py36/python36/lib/pkgconfig:/usr/lpp/python/python-2017-04-12-py36/python36/share/pkgconfig
    RELEASE_DIR=/usr/lpp/python/python-2017-04-12-py36
    LIBPATH=/usr/lpp/python/python-2017-04-12-py36/python36/lib:/lib:/usr/lib:.
    RELEASE_TYPE=py36
    _OPTIONS=
    TERM=dumb
    HOME=/
    PYTHONPATH=/usr/lpp/python/lib/python2.7
    LINES=20
    JAVA_HOME=/usr/lpp/java/current
    TZ=UTC-10 UTC-11,M10.1.0,M4.1.0
    MANPATH=/usr/man/%L:/usr/lpp/python/man/
    NLSPATH=/usr/lib/nls/msg/%L/%N
    TERMINFO=/usr/lpp/python/python-2017-04-12-py36/python36/share/terminfo

    When I did this print for you I saw that my PYTHONPATH, MANPATH and PYTHONHOME were still pointing to python 2.7, I hadn’t touched them because the new release came with PYTHON_HOME which was a different variable.

    I tried removing the PYTHONPATH, MANPATH and PYTHONHOME and now it looks a lot healthier…
    # python
    Python 3.6.1 (default, Apr 7 2017, 13:59:34) [C] on zos
    Type “help”, “copyright”, “credits” or “license” for more information.
    >>>

    So it looks like I have pretty much solved my own problem.
    If I could offer a suggestion or two:

    1. a note on running ICSF to get around the /dev/urandom would be nice
    2. a mention of the difference between PYTHONHOME and PYTHON_HOME environment variables would be nice
    3. a sample “hello World” type program with instructions on how to run it would be nice.
    4. The error messages print in ASCII not EBCDIC (the CEEnnnn which appear as äáá----á-èÇÁ-ËÑÅ>/%-ëñå âêè-Ï/Ë-ÊÁÄÁÑÎÁÀ–) Looks like something isn’t setting the ASCII/EBCDIC flag properly.

    Cheers



  • 10.  RE: Python installation on z/OS filesystem

    Posted 06-13-2017 09:30

    Thank you for your detailed description of your problems, and for the suggested improvements. Python for z/OS is currently under very active development, and your feedback is very helpful.



  • 11.  RE: Python installation on z/OS filesystem

    Posted 03-20-2019 15:29

    Ok I have managed to get the combined gzip and tar to work…yay! Now a couple of question on the list of commands for the script…
    export BASH_PREFIX=<the_directory_containing_bin_bash> Got it…
    export PERL_PREFIX=<the_directory_containing_bin_perl> Got it…
    INSTALL=<your_install_directory>
    Here is where I have a question. I have /u/ROCKET/python created and I’m wanting to install both 27 and 36. So, should the /u/ROCKET/python be my install dir? Will I end up with a python27 and python36 dirs under that?
    export RELEASE_NAME=python-2017-04-12
    export RELEASE_TYPE=py27 # or py36 or both, to match the download you chose
    Then here…do I make the value here actually “both” in order to get both releases or something like py27, run the script, then py36 and run the script a 2nd time?
    export RELEASE_DIR=INSTALL/{RELEASE_NAME}-${RELEASE_TYPE}
    export PKGS_BASE=$RELEASE_DIR/pkgs
    cd $RELEASE_DIR; ls -ld python27 python36 both

    based on the results of the ls, run one or more of these commands

    cd $RELEASE_DIR/python27; bin/install_all_packages
    cd $RELEASE_DIR/python36; bin/install_all_packages
    cd $RELEASE_DIR/both; bin/install_all_packages

    Again, any guidance is appreciated!
    Deb



  • 12.  RE: Python installation on z/OS filesystem

    Posted 03-29-2019 11:40

    Hi Deb,
    Please, read this topic: