Python installation path and permissions

Hi all

I'm trying to run some Python programs that requiere some third party libraries and I'm having some problems installing them as a regular user.

This is the folder structure:

[root@~]# ls -lh /usr/uv/python/
drwxr-xr-x 2 root root 4,0K jul 17 13:21 bin
drwxr-xr-x 3 root root   22 jun 15  2021 include
drwxr-xr-x 4 root root  106 jun 15  2021 lib
drwxr-xr-x 3 root root   16 jun 15  2021 share
drwxr-xr-x 2 root root   29 jun 15  2021 syslib


[root@~]# ls -lh /usr/uv/python/lib/python3.9/ | grep site-pac
drwxr-xr-x 18 root root 4,0K jul 17 13:21 site-packages

On another server there are some differences, and the owner is the uvdb user, not root.

[root@~]# ls -lh /usr/uv/python/
drwxr-xr-x 2 uvdb root 4.0K May 16 13:14 bin
drwxr-xr-x 3 uvdb root   24 Oct 18  2023 include
drwxr-xr-x 4 uvdb root  114 Oct 18  2023 lib
drwxr-xr-x 3 uvdb root   17 Oct 18  2023 share
drwxr-xr-x 2 uvdb root   68 Oct 23  2023 syslib

When I try to install some package it's going to be installed in the user space, so it's not available for all the users:

SH-4.2$ /usr/uv/python/bin/python3 -m pip install slugify
Defaulting to user installation because normal site-packages is not writeable

This can be circumvented installing packages as root, but this seems as a hack brute force solution.

I tried changing owners of the site-packages folder, so now it's owned by a group to which all my users belong, and the packages then installed for all users. The problem is that some packages need to write to the bin folder, the one that is owned by uvdb, and I'm hesitant to change ownership there and break something in the Python integration.

Is there a way to solve this issues via configuration or installation?

You will likely need to set the PYTHONUSERBASE environment variable to a location where non-root users can write.

Then use the "--user" command line option on the python -m pip command.

If you need interactive UV shells and phantom processes to run your python logic that uses modules stored in the PYTHONUSERBASE directory then the environment variable needs to be set within the custom_env script inside the UV installation's bin directory. This gets used each time a new uv process is launched.

The alternative is to use the ENV PYTHONUSERBASE={path} command at TCL. Not sure if a shell launched out of the UV command line inherits those settings so worth testing if that option is to be used.

Hi Gregor

I'm a bit lost.

I guess PYTHONUSERBASE  is a variable that is set in the .pyconfig file, right?.

Edit: I guess not...

>PYTHON
Invalid config parameter, PYTHONUSERBASE=/u2/usuarios/quiter/.local/lib/python3.11/site-packages/

Can I install the third party modules in the PYPROGS folder?

About adding the user flag to pip: I want the packages to be usable for all users, not only one of them.

And, also, about interactive shells: for the moment this is not an issue. The python modules are going to be called only from BASIC via PyCallFunction.

------------------------------
Héctor Cortiguera
Quiter Servicios Informaticos SL
------------------------------

Hi Gregor

I'm a bit lost.

I guess PYTHONUSERBASE  is a variable that is set in the .pyconfig file, right?.

Edit: I guess not...

>PYTHON
Invalid config parameter, PYTHONUSERBASE=/u2/usuarios/quiter/.local/lib/python3.11/site-packages/

Can I install the third party modules in the PYPROGS folder?

About adding the user flag to pip: I want the packages to be usable for all users, not only one of them.

And, also, about interactive shells: for the moment this is not an issue. The python modules are going to be called only from BASIC via PyCallFunction.

------------------------------
Héctor Cortiguera
Quiter Servicios Informaticos SL
------------------------------

Hi @Héctor Cortiguera

PYTHONUSERBASE is an environment variable that is part of and recognised by the python interpreter.

It allows for the installation and use of python packages from a location other than the python installation directory.

This is helpful as typically the python installation directory it owned by user root (on Linux) so is not writable by normal users.

Once that environment variable is set then adding the "--user" option to the pip command will result in package management being directed to the path defined.

For your scenario you need all UV processes to establish the same value for the PYTHONUSERBASE environment variable so they all point to the same package installations.

For interactive sessions (as I mentioned) this can be achieved via the custom_env file in the bin directory of the universe installation path.

For non-interactive sessions you can either deal with setting this variable within the UOLOGIN (for uvcs connections) or ODBCLOGIN (for odbc connections) subroutines, or implement the logic within your own code.

Regardless of which subroutine you place it , you should simply need to execute the "ENV SET PYTHONUSERBASE={path}" command to establish the variable. From then on the uv python interpreter should reference packages installed into your PYTHONUSERBASE directory.

Hi Gregor

Thanks for the tip.

Using $PYTHONUSERBASE now I can install third party libraries in a custom path.

But this path is not being added to the Python interpreter path.

How can I force Python to add PYTHONUSERBASE to its path?

Hi @Héctor Cortiguera

Great to hear you can now install the modules were you want them.

The PYTHONUSERBASE environment variable should be recognised by the python interpreter when it starts.

You can test this via the following TCL command:

SH -c 'echo "PYTHONUSERBASE=${PYTHONUSERBASE}";/usr/uv/python/bin/python3 -m site'

(Note that my example is from a RHEL8 UV 11.3.5 install. The "/usr/uv" install directory for you installation might differ. The "echo" command might need to be different on Windows installs).

The output from the TCL command should show you the current setting of the environment variable as well as how the interpreter is using it.

Once you know the interpreter is able to detect and include the PYTHONUSERBASE setting your next challenge is to get this variable set within your execution environment.

As I mentioned, interactive shells can be catered for by using the custom_env file in the bin directory of the UV installation (i.e. on RHEL8 that is /usr/uv/bin/custom_env).

If you are wanting to use this via an API connection to the UniRPC "uvcs" or "defcs" services then you have two options:

1. Implement code in the UOLOGIN subroutine that UV runs for each new AIP connection (there are knowledge base articles on this routine)
2. Implement code in your own BASIC programs to set the environment variable.

The other option is to not rely on the PYTHONUSERBASE being recognised by the python interpreter, and instead configure UV python to add the directory you assigned to the PYTHONUSERBASE variable to it's list of search folders.

You do this by creating a ".pth" file (read about Path configuration files in the Python "site" module documentation) within the lib/pythonX.Y/site-packages folder of your UV python installation (on my RHEL8 server that is /usr/uv/python/lib/python3.11/site-packages/).
You might need admin/root access to do this but it is a one-time change.
So, if you create the file custom.pth within that directory and set it's contents to be the path to your PYTHONUSERBASE directory then each time the python interpreter runs it will include that directory in it's search path for modules.
After setting this file up, re-run the above TCL command to review the python module search path and ensure it includes your custom directory.
With this setup you still need to use the PYTHONUSERBASE environment variable and the --user option on the pip command when you install your application-specific modules.

Hi, @Gregor Scott

I'm taking this issue again.

I want to follow the .pth file option, so I created my own custom.pth

$ cat /usr/uv/python/lib/python3.11/site-packages/pylibs.pth
/u2/quiter/GEN4GL/PYLIBS

Now my PYTHONUSERBASE environment variable is empty, and Python is adding the content of the .pth file to the path.

>SH -c 'echo "PYTHONUSERBASE=${PYTHONUSERBASE}";/usr/uv/python/bin/python3 -m site'
PYTHONUSERBASE=
sys.path = [
    '/home/quiter',
    '/usr/uv/python/lib/python311.zip',
    '/usr/uv/python/lib/python3.11',
    '/usr/uv/python/lib/python3.11/lib-dynload',
    '/home/quiter/.local/lib/python3.11/site-packages',
    '/usr/uv/python/lib/python3.11/site-packages',
    '/u2/quiter/GEN4GL/PYLIBS',
    '/usr/uv/bin',
    '/usr/uv/XDEMO/PP',
    '/u2/quiter/GEN4GL/PYPROGS',
]
USER_BASE: '/home/quiter/.local' (exists)
USER_SITE: '/home/quiter/.local/lib/python3.11/site-packages' (exists)
ENABLE_USER_SITE: True
>

My problem is that the USER_BASE value is set to '/home/quiter/.local', and not the path I put into the .pth file.

So when I try to install a module with pip it goes into that folder, and not the one I want.

Is there a way to fix this? Do I need to set both PYTHONUSERBASE and the .pth file?

------------------------------
Héctor Cortiguera
Quiter Servicios Informaticos SL
------------------------------

Elaborating on this:

I have installed a library in the path indicated by the custom.pth file:

$ ls -lh /u2/quiter/GEN4GL/PYLIBS/lib/python3.11/site-packages/
total 0
drwxrwxr-x 2 quiter quiter 143 may 15 15:26 python_slugify-8.0.4.dist-info
drwxrwxr-x 3 quiter quiter 137 may 15 15:26 slugify
drwxrwxr-x 3 quiter quiter  60 may 15 15:23 text_unidecode
drwxrwxr-x 2 quiter quiter 150 may 15 15:23 text_unidecode-1.3.dist-info

I haven't set the $PYTHONUSERBASE environment variable, but my path is in the sys.path of Python

$ echo "PYTHONUSERBASE=${PYTHONUSERBASE}";/usr/uv/python/bin/python3 -m site
PYTHONUSERBASE=
sys.path = [
    '/home/quiter',
    '/usr/uv/python/lib/python311.zip',
    '/usr/uv/python/lib/python3.11',
    '/usr/uv/python/lib/python3.11/lib-dynload',
    '/home/quiter/.local/lib/python3.11/site-packages',
    '/usr/uv/python/lib/python3.11/site-packages',
    '/u2/quiter/GEN4GL/PYLIBS',
    '/usr/uv/bin',
    '/usr/uv/XDEMO/PP',
    '/u2/quiter/GEN4GL/PYPROGS',
]
USER_BASE: '/home/quiter/.local' (exists)
USER_SITE: '/home/quiter/.local/lib/python3.11/site-packages' (exists)

But I can't import the module 

$ /uv/python/bin/python3
Python 3.11.0 (main, Nov 16 2022, 04:47:09) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import slugify
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'slugify'
>>>

So: what's the point of the .pth file?