UOPY - Uploading a program

If the file is very large you might want to consider looking into using SequentialFile and update in blocks

The process hand for anything after 128K.... is that considered a big file? 

sorry, no idea on limitations, was just offering an alternative

Maybe adapt this code:

To this:

------------------------------
Mike Young
Senior PS Techical Consultant
Aptean
Atlanta GA US
------------------------------

Alejandro,

Does this help at all - it mentions a Python 128K limitation

https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072/54517228#54517228

Regards

JayJay

------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

Hi,

It's been a long time, but I need to revive this. I still have this issue. @John Jenkins, thanks for the suggestion, but the fix is something that the CSV library has available. I tried looking for something in the UOPY documentation, but I don't see any parameter we can pass to override any max-size variable. 

@Mike Young I tried your code suggestion; this is what I got:

Traceback (most recent call last):
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 179, in receive
    return self._socket.recv(buf_size)
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/test/unidata.py", line 53, in pushFile
    udfile.write(fname,lines)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_file.py", line 345, in write
    resp_code = self._call_server(in_packet, out_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_uniobject.py", line 29, in _call_server
    self._session.rpc_call(in_packet, out_packet, in_exec)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_session.py", line 200, in rpc_call
    self._rpc_connection.call(in_packet, out_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 72, in call
    self._read_packet(in_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 123, in _read_packet
    in_packet.receive(self.rpc_socket)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 252, in receive
    self.header_buffer = rpc_socket.receive(_PACKET_HEADER_SIZE)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 182, in receive
    raise UOError(ErrorCodes.UOE_RPC_TIMEOUT) from e
uopy._uoerror.UOError: Error [81015] : The connection has timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/test/unidata.py", line 91, in <module>
    compileFile(dname, fname)
  File "/Users/test/unidata.py", line 66, in compileFile
    pushFile(dname,fname)
  File "/Users/test/unidata.py", line 54, in pushFile
    except UOError as ue:
NameError: name 'UOError' is not defined

The connection is timing out. Super weird, it is actually not throwing a UOError. And remember, If I try a file of 127Kb it goes without a problem... 

If anyone has any other ideas, please let me know.

Thanks in advance

Alejandro

Traceback (most recent call last):
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 179, in receive
    return self._socket.recv(buf_size)
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/test/unidata.py", line 53, in pushFile
    udfile.write(fname,lines)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_file.py", line 345, in write
    resp_code = self._call_server(in_packet, out_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_uniobject.py", line 29, in _call_server
    self._session.rpc_call(in_packet, out_packet, in_exec)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_session.py", line 200, in rpc_call
    self._rpc_connection.call(in_packet, out_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 72, in call
    self._read_packet(in_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 123, in _read_packet
    in_packet.receive(self.rpc_socket)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 252, in receive
    self.header_buffer = rpc_socket.receive(_PACKET_HEADER_SIZE)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 182, in receive
    raise UOError(ErrorCodes.UOE_RPC_TIMEOUT) from e
uopy._uoerror.UOError: Error [81015] : The connection has timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/test/unidata.py", line 91, in <module>
    compileFile(dname, fname)
  File "/Users/test/unidata.py", line 66, in compileFile
    pushFile(dname,fname)
  File "/Users/test/unidata.py", line 54, in pushFile
    except UOError as ue:
NameError: name 'UOError' is not defined

Alejandro,

My python code and UniVerse are on the same server.

I created a big file, and keep expanding.  At this time I am up to 16185840 bytes and it still works.

Note that it is fast in my environment where there is no network delay between the client and the server.  

Is your UniVerse on the same system as your python code?

Have you tried increasing the timeout on the uopy connection?

Hi Mike,

Thanks for your insights. My Python runs on my local machine, but my database Unidata is on a remote server. Not sure if the problem is the network; I try a file of 127KB, and goes up in a second, as soon as I put 128KB, It just hangs. How can I increase more time to the uopy connection? 

Also, not sure if this has any impact, but I'm pushing a program into BP. 

Again, thank you for keeping the conversation

Regards

Alejandro,

From the python side, you can just add it in the connection parameters, either in the uopy.ini file or when you make the connection:

def makeconnection(user, password):
    config = {
             'user': user,
             'password': password,
             'service': 'uvcs',
             'host': 'localhost',
             'account': 'C:\\\\U2\\\\UV\\\\XDEMO',
             'pooling': True,
             'min_pool_size': 2,
             'max_pool_size': 5,
             'timeout': 300,
             'encoding': 'GB18030'
         }
    thissession = uopy.connect(**config)
    return thissession

Thank you,

I tried your code, but sadly It still hangs. 

Also, I'm using uopy 1.2.0. I updated to 1.3.0, and I'm unable to establish a session. I keep looking at the documentation, but everything seems to be the same. I keep getting:

Error [81009] : The RPC failed

And in the logs I see this message:

2023-11-16 17:00:16,609 : 91525 : MainThread : _unirpc : __init__ : 169 ：ERROR : module 'socket' has no attribute 'TCP_KEEPIDLE'

I was hoping that maybe the new version would help with my uploading issue, but no idea why I'm not even able to connect :S As soon as I switch to 1.2.0 it connects with no issues and we are just having problems with uploading big programs.

------------------------------
Alejandro Garcia
Software Engineer Manager
F W Davison and Company Inc. DBA PrismHR
MA US
------------------------------

Alejandro,

I am not having the problem with Python 3.9.12.  I have found some references online about issues with the socket libraries in older Python releases.  

What version of Python are you on?

Hi, it's been some time, and sadly this is still an issue :S 

I found some more information, and currently, these are 2 different issues. The connection problem with newer versions of Python seems like a problem with the specific version of Python that you use to run the application; I was able to find one that actually works. For that, I will open another thread. 

To keep the discussion of this thread, uploading a program bigger than 128KB, the process still hangs. Now, this time, I was able to get something from the logs, and this is the only message that shows:

2024-11-14 18:45:22,158 : 24089 : MainThread : _uniobject : _call_server : 35 ：ERROR : unpack requires a buffer of 24 bytes

Does this help to troubleshoot the issue? It looks like the error is coming from uniobject. Do you have any ideas? Is this something that needs to be configured on the server? This buffer? or is it something that get configured in the connection?

Any help is greatly appreciated.

Alejandro,

I found this specific reference which points to a possible cause https://python.readthedocs.io/en/stable/library/struct.html - does this help?

Regards

JJ

Hi John,

Thank you, I think is good, maybe this reference is something that is use on the UniRPCPacket which is part of the implementation of the write function of UOPY.

def write(self, record_id, record, lock_flag=0):
        """Write a record into the file.

        Args:
            record_id (any): the record id - can be str, bytes, or DynArray.
            record (any): the record to be written - can be DynArray, str, bytes.
            lock_flag (int, optional): 0 (default), LOCK_RETAIN, LOCK_WAIT, or LOCK_RETAIN + LOCK_WAIT

        Returns:
            None

        Raises:
            UOError

        """
        _logger.debug("Enter", record_id, record, lock_flag)

        self._check_opened()
        with self._lock:
            in_packet = UniRPCPacket()
            out_packet = UniRPCPacket()

            out_packet.write(0, FuncCodes.EIC_WRITE)
            out_packet.write(1, self._handle)
            out_packet.write(2, _map_write_lock_flag(lock_flag))
            out_packet.write(3, self._session.encode(record_id))
            out_packet.write(4, self._session.encode(record))

            resp_code = self._call_server(in_packet, out_packet)

            if resp_code == 0:
                self._status = in_packet.read(1)
            else:
                raise UOError(code=resp_code)

        _logger.debug("Exit")

So, maybe somebody from Rocket can review it and see if we are pointing to the right place.

At the same time, I did more testing. I found that if I try to write a program whose size is bigger than 131820 bytes, that is when I get the error. 131821 I get the error, 131820 is fine. I Attached the Python program (testWrite.py), which is the one that I used for the test, and 2 "programs" (it is just a bunch of PRINTS). The 127KB gets uploaded into the BP folder without any issue, and the 128KB program fails. 

Here are the output:

128KB

/usr/bin/python3 ../testWrite.py BP 128KB
before connection
udcs
defcs
after connection:
UD
Connection established
Script Name is ../testWrite.py
All parameters ['../testWrite.py', 'BP', '128KB']
All parameters ['../testWrite.py', 'BP', '128KB']
done reading
uopy.File done
Traceback (most recent call last):
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_uniobject.py", line 29, in _call_server
    self._session.rpc_call(in_packet, out_packet, in_exec)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_session.py", line 200, in rpc_call
    self._rpc_connection.call(in_packet, out_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 72, in call
    self._read_packet(in_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 123, in _read_packet
    in_packet.receive(self.rpc_socket)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 253, in receive
    self._unpack_header()
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 399, in _unpack_header
    self.header_fields = unpack(_PACKET_MSG_HDR_FMT, self.header_buffer)
struct.error: unpack requires a buffer of 24 bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/test/testWrite.py", line 57, in <module>
  File "/Users/test/testWrite.py", line 20, in pushFile
    print("file uploaded")
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_file.py", line 345, in write
    resp_code = self._call_server(in_packet, out_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_uniobject.py", line 36, in _call_server
    raise UOError(ErrorCodes.UOE_UNKNOWN_ERROR) from e
uopy._uoerror.UOError: Error [-1] : Unexpected Error

127KB

/usr/bin/python3 ../testWrite.py BP 127KB 
before connection
udcs
defcs
after connection:
UD
Connection established
Script Name is ../testWrite.py
All parameters ['../testWrite.py', 'BP', '127KB']
All parameters ['../testWrite.py', 'BP', '127KB']
done reading
uopy.File done
udfile Write done
file uploaded

I'm using UOPY 1.3.0. If anyone can also replicate this test. I would like to know if the issue is with my machine or my server or if this is happening to someone else; it could be a restriction from UOPY.

Thanks in advance

def write(self, record_id, record, lock_flag=0):
        """Write a record into the file.

        Args:
            record_id (any): the record id - can be str, bytes, or DynArray.
            record (any): the record to be written - can be DynArray, str, bytes.
            lock_flag (int, optional): 0 (default), LOCK_RETAIN, LOCK_WAIT, or LOCK_RETAIN + LOCK_WAIT

        Returns:
            None

        Raises:
            UOError

        """
        _logger.debug("Enter", record_id, record, lock_flag)

        self._check_opened()
        with self._lock:
            in_packet = UniRPCPacket()
            out_packet = UniRPCPacket()

            out_packet.write(0, FuncCodes.EIC_WRITE)
            out_packet.write(1, self._handle)
            out_packet.write(2, _map_write_lock_flag(lock_flag))
            out_packet.write(3, self._session.encode(record_id))
            out_packet.write(4, self._session.encode(record))

            resp_code = self._call_server(in_packet, out_packet)

            if resp_code == 0:
                self._status = in_packet.read(1)
            else:
                raise UOError(code=resp_code)

        _logger.debug("Exit")

/usr/bin/python3 ../testWrite.py BP 128KB
before connection
udcs
defcs
after connection:
UD
Connection established
Script Name is ../testWrite.py
All parameters ['../testWrite.py', 'BP', '128KB']
All parameters ['../testWrite.py', 'BP', '128KB']
done reading
uopy.File done
Traceback (most recent call last):
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_uniobject.py", line 29, in _call_server
    self._session.rpc_call(in_packet, out_packet, in_exec)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_session.py", line 200, in rpc_call
    self._rpc_connection.call(in_packet, out_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 72, in call
    self._read_packet(in_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 123, in _read_packet
    in_packet.receive(self.rpc_socket)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 253, in receive
    self._unpack_header()
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 399, in _unpack_header
    self.header_fields = unpack(_PACKET_MSG_HDR_FMT, self.header_buffer)
struct.error: unpack requires a buffer of 24 bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/test/testWrite.py", line 57, in <module>
  File "/Users/test/testWrite.py", line 20, in pushFile
    print("file uploaded")
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_file.py", line 345, in write
    resp_code = self._call_server(in_packet, out_packet)
  File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_uniobject.py", line 36, in _call_server
    raise UOError(ErrorCodes.UOE_UNKNOWN_ERROR) from e
uopy._uoerror.UOError: Error [-1] : Unexpected Error

/usr/bin/python3 ../testWrite.py BP 127KB 
before connection
udcs
defcs
after connection:
UD
Connection established
Script Name is ../testWrite.py
All parameters ['../testWrite.py', 'BP', '127KB']
All parameters ['../testWrite.py', 'BP', '127KB']
done reading
uopy.File done
udfile Write done
file uploaded

Alejandro,

I tested your program from Windows and Linux clients and was successful in each.

[root@qasdevtsmvdw02 testWrite]# /usr/ud82/python/bin/python3 /usr/ud82/XDEMO/te
stWrite/testWrite/testWrite.py BP 127KB
before connection
udcs
defcs
after connection:
UD
Connection established
Script Name is /usr/ud82/XDEMO/testWrite/testWrite/testWrite.py
All parameters ['/usr/ud82/XDEMO/testWrite/testWrite/testWrite.py', 'BP', '127KB
']
done reading
uopy.File done
udfile Write done
file uploaded
[root@qasdevtsmvdw02 testWrite]# /usr/ud82/python/bin/python3 /usr/ud82/XDEMO/te
stWrite/testWrite/testWrite.py BP 128KB
before connection
udcs
defcs
after connection:
UD
Connection established
Script Name is /usr/ud82/XDEMO/testWrite/testWrite/testWrite.py
All parameters ['/usr/ud82/XDEMO/testWrite/testWrite/testWrite.py', 'BP', '128KB
']
done reading
uopy.File done
udfile Write done
file uploaded

:AE BP 127KB
Top of "127KB" in "BP", 1,199 lines, 131,820 characters.
*---: Q
Quit "127KB" in file "BP" unchanged.
:AE BP 128KB
Top of "128KB" in "BP", 1,199 lines, 131,821 characters.
*---: Q
Quit "128KB" in file "BP" unchanged.
:

This was using uopy 1.3.0 connecting to Unidata 8.2.4 (hosted on linux) and Python 3.9.12.

Windows 2010 client and centOS 7 client. 

Seems an issue in your environment. What is the client and server OS? Sorry if answered prior.

Regards,

Dave

[root@qasdevtsmvdw02 testWrite]# /usr/ud82/python/bin/python3 /usr/ud82/XDEMO/te
stWrite/testWrite/testWrite.py BP 127KB
before connection
udcs
defcs
after connection:
UD
Connection established
Script Name is /usr/ud82/XDEMO/testWrite/testWrite/testWrite.py
All parameters ['/usr/ud82/XDEMO/testWrite/testWrite/testWrite.py', 'BP', '127KB
']
done reading
uopy.File done
udfile Write done
file uploaded
[root@qasdevtsmvdw02 testWrite]# /usr/ud82/python/bin/python3 /usr/ud82/XDEMO/te
stWrite/testWrite/testWrite.py BP 128KB
before connection
udcs
defcs
after connection:
UD
Connection established
Script Name is /usr/ud82/XDEMO/testWrite/testWrite/testWrite.py
All parameters ['/usr/ud82/XDEMO/testWrite/testWrite/testWrite.py', 'BP', '128KB
']
done reading
uopy.File done
udfile Write done
file uploaded

:AE BP 127KB
Top of "127KB" in "BP", 1,199 lines, 131,820 characters.
*---: Q
Quit "127KB" in file "BP" unchanged.
:AE BP 128KB
Top of "128KB" in "BP", 1,199 lines, 131,821 characters.
*---: Q
Quit "128KB" in file "BP" unchanged.
:

Hi Dave,

Thank you for running the test. I ran the script from Mac OS (M1) to Unidata 8.2.4 (hosted on Windows) and Python 3.9.6.

I must say that I also have a problem with Python binaries. The UOPY library is only able to make a successful connection with this binary:

Python 3.9.6 (default, Oct  4 2024, 08:01:31) 
[Clang 16.0.0 (clang-1600.0.26.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

I have tried other ones, and I get an RPC error. I am opening another thread for that, but it looks like it is related to the CLANG library. Only the Python compiled with CLANG 16 seems to be working fine with UOPY. I switched to using Anaconda to better control the different Python environments, and that's when the problem started. Well, even before that, in an update of a Python version.

This one works:

/usr/bin/python3

Python 3.9.6 (default, Oct  4 2024, 08:01:31)

[Clang 16.0.0 (clang-1600.0.26.4)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>>

This one from Anaconda doesn't work

Python 3.9.20 (main, Oct  3 2024, 02:24:59)

[Clang 14.0.6 ] :: Anaconda, Inc. on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>>

Actually, no version from Anaconda seems to work. The only difference that I can see is the Clang. Anaconda looks like it is using version 14. The Python that works is one that I installed via Homebrew, and that one uses Clang 16. I'm not sure if that is the issue, but at this point, I don't know what else could be. People in Windows are seen to be using a Python binary that is compiled with GCC instead of CLANG.

Also, I have been trying the Rocket Explorer plugin, but there is no progress. I can connect to the server and open the account, but if I try to open the File to list the programs, it doesn't work. 

Do you have any ideas or workarounds? It has become very hard to work with Unidata if the code is remote.

Thanks in advance