RocketCE for Power

 View Only
  • 1.  Releasing RocketCE Packages with OpenCE v1.8.0

    ROCKETEER
    Posted 01-12-2023 00:39
    Edited by Rajesh Nukala 02-10-2023 03:44

    As part of this Rocket provides the AI/ML conda packages and GPU Operator for Power machines.

    The RocketCE conda packages are build and supported for Power 9 and Power 10 with ppc64le architecture. These packages are installed into a conda environment so after installation is complete, the frameworks are ready for use. Presently these packages are available in RocketCE conda channel.
    This version of packages are available for Python 3.8, 3.9 and 3.10 and for CUDA 11.2 and 11.4.
    Note: Access and availability to this data is subject to the community and can change at any time.

    We are releasing RocketCE packages based on OpenCE v1.8.0.

    To know more about RocketCE and the various packages released with OpenCE1.8.0 please use the appropriate links below.



    ------------------------------
    Rajesh Nukala
    Rocket Internal - All Brands
    ------------------------------



  • 2.  RE: Releasing RocketCE Packages with OpenCE v1.8.0

    Posted 01-12-2023 09:21
    Hello Rajesh, thanks for releasing Open-CE publicly - this definitely helps :)

    When testing and comparing against our internal IBM builds I have found a few inconsistencies with the rocketce implementation/build of open-ce.

    1.) When installing e.g. numpy via rocketce channel it seems to be missing a link (or the package at all) to libgfortran:

    (base) mambauser@a86b90e1762b:/tmp$ micromamba install -n base -c "rocketce/label/rocketce-1.8.0" -c main python=3.9 numpy
    [...]
    >>> import numpy as np
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/__init__.py", line 143, in <module>
    from . import lib
    File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/lib/__init__.py", line 25, in <module>
    from .index_tricks import *
    File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/lib/index_tricks.py", line 12, in <module>
    import numpy.matrixlib as matrixlib
    File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/matrixlib/__init__.py", line 4, in <module>
    from .defmatrix import *
    File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py", line 11, in <module>
    from numpy.linalg import matrix_power
    File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/linalg/__init__.py", line 73, in <module>
    from . import linalg
    File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/linalg/linalg.py", line 34, in <module>
    from numpy.linalg import _umath_linalg
    ImportError: libgfortran.so.5: cannot open shared object file: No such file or directory

    This can be fixed by installing libgfortan manually, but to avoid struggle it would be great have it linked automatically. When installing numpy via e.g. the IBM internal channel it works without issues:

    (base) mambauser@a86b90e1762b:/tmp$ micromamba install -n base -c http://ausgsa.ibm.com/gsa/ausgsa/projects/o/open-ce/conda/Open-CE-r1.8/1.8.0/opence-p10/ -c main python=3.9 numpy
    [...]
    >>> import numpy as np
    >>> np.__version__
    '1.23.4'

    2.) If I want to install tensorflow and pytorch using open-ce 1.8.0 rocketce channel this doesn't work:

    (base) mambauser@59d26b4271c4:/tmp$ micromamba install -n base -c "rocketce/label/rocketce-1.8.0" -c main python=3.9 tensorflow pytorch
    [...]
    error libmamba Could not solve for environment specs
    Encountered problems while solving:
    - nothing provides _tensorflow_select 2.0.* needed by tensorflow-2.10.1-cuda11.4_py38_1

    The environment can't be solved, aborting the operation

    critical libmamba Could not solve for environment specs

    However if I use our internal channel there is no solving issue and it works without problems.

    Would be great if you could look into that!

    Thanks!

    ------------------------------
    Marvin Giessing
    AI Systems & Red Hat OpenShift Architect
    Rocket Forum Shared Account
    ------------------------------



  • 3.  RE: Releasing RocketCE Packages with OpenCE v1.8.0

    ROCKETEER
    Posted 01-13-2023 10:43
    Hi Marvin,

    Thanks for reaching us on the issues. 
    For the issue 1, could you please provide the installation logs and also the list of installed packages in your environment by running the below commands 
    • micromamba install -n base -c "rocketce/label/rocketce-1.8.0" -c main python=3.9 numpy -v
    • micomamba list

    Thanks & Regards,
    Rajesh


    ------------------------------
    Rajesh Nukala
    Rocket Internal - All Brands
    ------------------------------



  • 4.  RE: Releasing RocketCE Packages with OpenCE v1.8.0

    Posted 01-13-2023 11:17
    Edited by Marvin Giessing 01-13-2023 11:17

    Sure thing, here you go!

    (base) mambauser@54e9ba86f764:/tmp$ micromamba list

    List of packages in environment: "/opt/conda"

      Name                   Version     Build              Channel                      

    ───────────────────────────────────────────────────────────────────────────────────────

      _libgcc_mutex          0.1         main               main                         

      _openmp_mutex          5.1         1_gnu              main                         

      blas                   1.0         openblas           main                         

      ca-certificates        2022.10.11  h6ffa863_0         main                         

      certifi                2022.12.7   py39h6ffa863_0     main                         

      ld_impl_linux-ppc64le  2.38        hec883e6_1         main                         

      libffi                 3.4.2       h4a02239_6         main                         

      libgcc-ng              11.2.0      h1234567_1         main                         

      libgomp                11.2.0      h1234567_1         main                         

      libopenblas            0.3.21      openmp_h1234567_1  rocketce/label/rocketce-1.8.0

      libstdcxx-ng           11.2.0      h1234567_1         main                         

      llvm-openmp            14.0.6      h33aec20_0         rocketce/label/rocketce-1.8.0

      ncurses                6.3         hf118e41_3         main                         

      nomkl                  3.0         0                  main                         

      numpy                  1.23.4      py39h6ac58bb_0     rocketce/label/rocketce-1.8.0

      numpy-base             1.23.4      py39h2d094e7_0     rocketce/label/rocketce-1.8.0

      openblas               0.3.21      openmp_h1234567_1  rocketce/label/rocketce-1.8.0

      openblas-devel         0.3.21      openmp_h1234567_1  rocketce/label/rocketce-1.8.0

      openssl                1.1.1s      h140841e_0         main                         

      pip                    22.3.1      py39h6ffa863_0     main                         

      python                 3.9.15      hf689259_2         main                         

      readline               8.2         hf118e41_0         main                         

      setuptools             65.6.3      py39h6ffa863_0     main                         

      sqlite                 3.40.1      hefe40ec_0         main                         

      tk                     8.6.12      h7e00dab_0         main                         

      tzdata                 2022g       h04d1e81_0         main                         

      wheel                  0.37.1      pyhd3eb1b0_0       main                         

      xz                     5.2.8       hf118e41_0         main                         

      zlib                   1.2.13      h69ee22e_0         rocketce/label/rocketce-1.8.0



    ------------------------------
    Marvin Giessing
    AI Systems & Red Hat OpenShift Architect
    Rocket Forum Shared Account
    ------------------------------



  • 5.  RE: Releasing RocketCE Packages with OpenCE v1.8.0

    ROCKETEER
    Posted 01-24-2023 11:38
    Thank you Marvin for providing the logs. Below are my comments on the issues reported
     
    Issue 1 - Fixed this issue. Please let me know if you still see any issues.
    Issue 2 - Use the command by replacing the label name "rocketce/label/rocketce-1.8.0" with channel name "rocketce"
    • micromamba install -n base -c rocketce -c main python=3.9 tensorflow pytorch


    ------------------------------
    Rajesh Nukala
    Rocket Internal - All Brands
    ------------------------------



  • 6.  RE: Releasing RocketCE Packages with OpenCE v1.8.0

    Posted 01-25-2023 06:40
    Thanks a lot Rajesh!

    Issue 1: Installation is fixed - thanks! h
    However the build itself doesn't link OpenBLAS compared to the default numpy provided by conda/conda-forge and is therefore kinda quite slow:

    (rocketce) mambauser@7115e2ac3367:/tmp$ python -m timeit 'import numpy as np; N=1000; A=np.random.rand(N,N); B=np.random.rand(N,N); np.matmul(A,B)'

    1 loop, best of 5: 1.83 sec per loop

    (defaults) mambauser@7115e2ac3367:/tmp$ python -m timeit 'import numpy as np; N=1000; A=np.random.rand(N,N); B=np.random.rand(N,N); np.matmul(A,B)'

    1 loop, best of 5: 154 msec per loop

    I'm in discussion with opence developers to have that fixed.

    Issue 2: Thanks a lot that helps!



    ------------------------------
    Marvin Giessing
    AI Systems & Red Hat OpenShift Architect
    Rocket Forum Shared Account
    ------------------------------



  • 7.  RE: Releasing RocketCE Packages with OpenCE v1.8.0

    ROCKETEER
    Posted 02-06-2023 03:05
    Thanks Marvin for the confirmation and also for reaching us on the numpy performance issue.

    Now the fix for numpy matmul performance issue is available is in rocketce. Please let us know if you face any further issues.

    [suyog@rain6p1 numpy]$ conda list | grep numpy
    # packages in environment at /home/suyog/miniconda3/envs/testnumpy:
    numpy                     1.23.4           py38h6ac58bb_0    rocketce
    numpy-base                1.23.4           py38h4c1029e_0    rocketce
    (testnumpy) [suyog@rain6p1 numpy]$ python -m timeit 'import numpy as np; N=1000; A=np.random.rand(N,N); B=np.random.rand(N,N); np.matmul(A,B)'
    1 loop, best of 5: 20 msec per loop


    ------------------------------
    Rajesh Nukala
    Rocket Internal - All Brands
    ------------------------------