Thanks Marvin for the confirmation and also for reaching us on the numpy performance issue.
Now the fix for numpy matmul performance issue is available is in rocketce. Please let us know if you face any further issues.
Original Message:
Sent: 01-25-2023 06:40
From: Marvin Giessing
Subject: Releasing RocketCE Packages with OpenCE v1.8.0
Thanks a lot Rajesh!
Issue 1: Installation is fixed - thanks! h
However the build itself doesn't link OpenBLAS compared to the default numpy provided by conda/conda-forge and is therefore kinda quite slow:
(rocketce) mambauser@7115e2ac3367:/tmp$ python -m timeit 'import numpy as np; N=1000; A=np.random.rand(N,N); B=np.random.rand(N,N); np.matmul(A,B)'
1 loop, best of 5: 1.83 sec per loop
(defaults) mambauser@7115e2ac3367:/tmp$ python -m timeit 'import numpy as np; N=1000; A=np.random.rand(N,N); B=np.random.rand(N,N); np.matmul(A,B)'
1 loop, best of 5: 154 msec per loop
I'm in discussion with opence developers to have that fixed.
Issue 2: Thanks a lot that helps!
------------------------------
Marvin Giessing
AI Systems & Red Hat OpenShift Architect
Rocket Forum Shared Account
Original Message:
Sent: 01-24-2023 11:37
From: Rajesh Nukala
Subject: Releasing RocketCE Packages with OpenCE v1.8.0
Thank you Marvin for providing the logs. Below are my comments on the issues reported
Issue 1 - Fixed this issue. Please let me know if you still see any issues.
Issue 2 - Use the command by replacing the label name "rocketce/label/rocketce-1.8.0" with channel name "rocketce"- micromamba install -n base -c rocketce -c main python=3.9 tensorflow pytorch
------------------------------
Rajesh Nukala
Rocket Internal - All Brands
Original Message:
Sent: 01-13-2023 11:16
From: Marvin Giessing
Subject: Releasing RocketCE Packages with OpenCE v1.8.0
Sure thing, here you go!
(base) mambauser@54e9ba86f764:/tmp$ micromamba list
List of packages in environment: "/opt/conda"
Name Version Build Channel
───────────────────────────────────────────────────────────────────────────────────────
_libgcc_mutex 0.1 main main
_openmp_mutex 5.1 1_gnu main
blas 1.0 openblas main
ca-certificates 2022.10.11 h6ffa863_0 main
certifi 2022.12.7 py39h6ffa863_0 main
ld_impl_linux-ppc64le 2.38 hec883e6_1 main
libffi 3.4.2 h4a02239_6 main
libgcc-ng 11.2.0 h1234567_1 main
libgomp 11.2.0 h1234567_1 main
libopenblas 0.3.21 openmp_h1234567_1 rocketce/label/rocketce-1.8.0
libstdcxx-ng 11.2.0 h1234567_1 main
llvm-openmp 14.0.6 h33aec20_0 rocketce/label/rocketce-1.8.0
ncurses 6.3 hf118e41_3 main
nomkl 3.0 0 main
numpy 1.23.4 py39h6ac58bb_0 rocketce/label/rocketce-1.8.0
numpy-base 1.23.4 py39h2d094e7_0 rocketce/label/rocketce-1.8.0
openblas 0.3.21 openmp_h1234567_1 rocketce/label/rocketce-1.8.0
openblas-devel 0.3.21 openmp_h1234567_1 rocketce/label/rocketce-1.8.0
openssl 1.1.1s h140841e_0 main
pip 22.3.1 py39h6ffa863_0 main
python 3.9.15 hf689259_2 main
readline 8.2 hf118e41_0 main
setuptools 65.6.3 py39h6ffa863_0 main
sqlite 3.40.1 hefe40ec_0 main
tk 8.6.12 h7e00dab_0 main
tzdata 2022g h04d1e81_0 main
wheel 0.37.1 pyhd3eb1b0_0 main
xz 5.2.8 hf118e41_0 main
zlib 1.2.13 h69ee22e_0 rocketce/label/rocketce-1.8.0
------------------------------
Marvin Giessing
AI Systems & Red Hat OpenShift Architect
Rocket Forum Shared Account
Original Message:
Sent: 01-13-2023 10:43
From: Rajesh Nukala
Subject: Releasing RocketCE Packages with OpenCE v1.8.0
Hi Marvin,
Thanks for reaching us on the issues.
For the issue 1, could you please provide the installation logs and also the list of installed packages in your environment by running the below commands
- micromamba install -n base -c "rocketce/label/rocketce-1.8.0" -c main python=3.9 numpy -v
- micomamba list
Thanks & Regards,
Rajesh
------------------------------
Rajesh Nukala
Rocket Internal - All Brands
Original Message:
Sent: 01-12-2023 07:58
From: Marvin Giessing
Subject: Releasing RocketCE Packages with OpenCE v1.8.0
Hello Rajesh, thanks for releasing Open-CE publicly - this definitely helps :)
When testing and comparing against our internal IBM builds I have found a few inconsistencies with the rocketce implementation/build of open-ce.
1.) When installing e.g. numpy via rocketce channel it seems to be missing a link (or the package at all) to libgfortran:
(base) mambauser@a86b90e1762b:/tmp$ micromamba install -n base -c "rocketce/label/rocketce-1.8.0" -c main python=3.9 numpy
[...]
>>> import numpy as np
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/__init__.py", line 143, in <module>
from . import lib
File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/lib/__init__.py", line 25, in <module>
from .index_tricks import *
File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/lib/index_tricks.py", line 12, in <module>
import numpy.matrixlib as matrixlib
File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/matrixlib/__init__.py", line 4, in <module>
from .defmatrix import *
File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py", line 11, in <module>
from numpy.linalg import matrix_power
File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/linalg/__init__.py", line 73, in <module>
from . import linalg
File "/root/miniconda3/envs/rocket1.8/lib/python3.9/site-packages/numpy/linalg/linalg.py", line 34, in <module>
from numpy.linalg import _umath_linalg
ImportError: libgfortran.so.5: cannot open shared object file: No such file or directory
This can be fixed by installing libgfortan manually, but to avoid struggle it would be great have it linked automatically. When installing numpy via e.g. the IBM internal channel it works without issues:
(base) mambauser@a86b90e1762b:/tmp$ micromamba install -n base -c http://ausgsa.ibm.com/gsa/ausgsa/projects/o/open-ce/conda/Open-CE-r1.8/1.8.0/opence-p10/ -c main python=3.9 numpy
[...]
>>> import numpy as np
>>> np.__version__
'1.23.4'
2.) If I want to install tensorflow and pytorch using open-ce 1.8.0 rocketce channel this doesn't work:
(base) mambauser@59d26b4271c4:/tmp$ micromamba install -n base -c "rocketce/label/rocketce-1.8.0" -c main python=3.9 tensorflow pytorch
[...]
error libmamba Could not solve for environment specs
Encountered problems while solving:
- nothing provides _tensorflow_select 2.0.* needed by tensorflow-2.10.1-cuda11.4_py38_1
The environment can't be solved, aborting the operation
critical libmamba Could not solve for environment specs
However if I use our internal channel there is no solving issue and it works without problems.
Would be great if you could look into that!
Thanks!
------------------------------
Marvin Giessing
AI Systems & Red Hat OpenShift Architect
Rocket Forum Shared Account
Original Message:
Sent: 01-12-2023 00:38
From: Rajesh Nukala
Subject: Releasing RocketCE Packages with OpenCE v1.8.0
As part of this Rocket provides the AI/ML conda packages and GPU Operator for Power machines.
The RocketCE conda packages are build and supported for Power 9 and Power 10 with ppc64le architecture. These packages are installed into a conda environment so after installation is complete, the frameworks are ready for use. Presently these packages are available in RocketCE conda channel.
This version of packages are available for Python 3.8, 3.9 and 3.10 and for CUDA 11.2 and 11.4.
Note: Access and availability to this data is subject to the community and can change at any time.
We are releasing RocketCE packages based on OpenCE v1.8.0.
To know more about RocketCE and the various packages released with OpenCE1.8.0 please use the appropriate links below.
------------------------------
Rajesh Nukala
Rocket Internal - All Brands
------------------------------