Rocket U2 | UniVerse & UniData

 View Only
Expand all | Collapse all

UOPY - Uploading a program

  • 1.  UOPY - Uploading a program

    Posted 06-20-2023 16:07

    Hello, it is me again fighting UOPY. 

    I need to push a big file, and after multiple testing, I found that I could not push a file bigger than 128K. The process hangs there forever.

    Is this a Unidata limitation or a UOPY limitation? Is there a way to change this limit?

    Here is my code:

    def pushFile(dname,fname):
      f = open(dname+'/'+fname,'rb')
      lines = f.read()
      f.close()
      print('done reading')
      # print(lines)
      udfile = uopy.File(dname,session=_session)
      udfile.write(fname,lines)
      udfile.close()
      print("file uploaded")

    Any help is greatly appreciated.



    ------------------------------
    Alejandro Garcia
    Software Engineer Manager
    F W Davison and Company Inc. DBA PrismHR
    MA US
    ------------------------------


  • 2.  RE: UOPY - Uploading a program

    PARTNER
    Posted 06-21-2023 07:26

    If the file is very large you might want to consider looking into using SequentialFile and update in blocks



    ------------------------------
    Mike Young
    Senior PS Techical Consultant
    Aptean
    Atlanta GA US
    ------------------------------



  • 3.  RE: UOPY - Uploading a program

    Posted 06-21-2023 09:32

    The process hand for anything after 128K.... is that considered a big file? 



    ------------------------------
    Alejandro Garcia
    Software Engineer Manager
    F W Davison and Company Inc. DBA PrismHR
    MA US
    ------------------------------



  • 4.  RE: UOPY - Uploading a program

    PARTNER
    Posted 06-21-2023 09:44
    Edited by Mike Young 06-21-2023 09:45

    sorry, no idea on limitations, was just offering an alternative

    Maybe adapt this code:

      udfile = uopy.File(dname,session=_session)
      udfile.write(fname,lines)
      udfile.close()
      print("file uploaded")

    To this:

      try:
         udfile = uopy.File(dname,session=_session)
        udfile.write(fname,lines)
      except UOError as ue:
        print(f"Error writing {fname} returned code {ue.code} - {ue}")
        return
      print("file uploaded")



    ------------------------------
    Mike Young
    Senior PS Techical Consultant
    Aptean
    Atlanta GA US
    ------------------------------



  • 5.  RE: UOPY - Uploading a program

    Posted 06-27-2023 16:33
    Edited by John Jenkins 06-27-2023 17:36

    Alejandro,

    Does this help at all - it mentions a Python 128K limitation

    https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072/54517228#54517228

    Regards

    JayJay



    ------------------------------
    John Jenkins
    Thame, Oxfordshire
    ------------------------------



  • 6.  RE: UOPY - Uploading a program

    Posted 11-08-2023 13:08

    Hi,

    It's been a long time, but I need to revive this. I still have this issue. @John Jenkins, thanks for the suggestion, but the fix is something that the CSV library has available. I tried looking for something in the UOPY documentation, but I don't see any parameter we can pass to override any max-size variable. 

    @Mike Young I tried your code suggestion; this is what I got:

    Traceback (most recent call last):
      File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 179, in receive
        return self._socket.recv(buf_size)
    socket.timeout: timed out
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/Users/test/unidata.py", line 53, in pushFile
        udfile.write(fname,lines)
      File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_file.py", line 345, in write
        resp_code = self._call_server(in_packet, out_packet)
      File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_uniobject.py", line 29, in _call_server
        self._session.rpc_call(in_packet, out_packet, in_exec)
      File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_session.py", line 200, in rpc_call
        self._rpc_connection.call(in_packet, out_packet)
      File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 72, in call
        self._read_packet(in_packet)
      File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 123, in _read_packet
        in_packet.receive(self.rpc_socket)
      File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 252, in receive
        self.header_buffer = rpc_socket.receive(_PACKET_HEADER_SIZE)
      File "/Users/test/Library/Python/3.9/lib/python/site-packages/uopy/_unirpc.py", line 182, in receive
        raise UOError(ErrorCodes.UOE_RPC_TIMEOUT) from e
    uopy._uoerror.UOError: Error [81015] : The connection has timed out
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/test/unidata.py", line 91, in <module>
        compileFile(dname, fname)
      File "/Users/test/unidata.py", line 66, in compileFile
        pushFile(dname,fname)
      File "/Users/test/unidata.py", line 54, in pushFile
        except UOError as ue:
    NameError: name 'UOError' is not defined

    The connection is timing out. Super weird, it is actually not throwing a UOError. And remember, If I try a file of 127Kb it goes without a problem... 

    If anyone has any other ideas, please let me know.

    Thanks in advance

    Alejandro



    ------------------------------
    Alejandro Garcia
    Software Engineer Manager
    F W Davison and Company Inc. DBA PrismHR
    MA US
    ------------------------------



  • 7.  RE: UOPY - Uploading a program

    ROCKETEER
    Posted 11-15-2023 11:10

    Alejandro,

    My python code and UniVerse are on the same server.

    I created a big file, and keep expanding.  At this time I am up to 16185840 bytes and it still works.

    Note that it is fast in my environment where there is no network delay between the client and the server.  

    Is your UniVerse on the same system as your python code?

    Have you tried increasing the timeout on the uopy connection?



    ------------------------------
    Mike Rajkowski
    MultiValue Product Evangelist
    Rocket Internal - All Brands
    US
    ------------------------------



  • 8.  RE: UOPY - Uploading a program

    Posted 11-16-2023 16:07

    Hi Mike,

    Thanks for your insights. My Python runs on my local machine, but my database Unidata is on a remote server. Not sure if the problem is the network; I try a file of 127KB, and goes up in a second, as soon as I put 128KB, It just hangs. How can I increase more time to the uopy connection? 

    Also, not sure if this has any impact, but I'm pushing a program into BP. 

    Again, thank you for keeping the conversation

    Regards



    ------------------------------
    Alejandro Garcia
    Software Engineer Manager
    F W Davison and Company Inc. DBA PrismHR
    MA US
    ------------------------------



  • 9.  RE: UOPY - Uploading a program

    ROCKETEER
    Posted 11-16-2023 16:54

    Alejandro,

    From the python side, you can just add it in the connection parameters, either in the uopy.ini file or when you make the connection:

    def makeconnection(user, password):
        config = {
                 'user': user,
                 'password': password,
                 'service': 'uvcs',
                 'host': 'localhost',
                 'account': 'C:\\U2\\UV\\XDEMO',
                 'pooling': True,
                 'min_pool_size': 2,
                 'max_pool_size': 5,
                 'timeout': 300,
                 'encoding': 'GB18030'
             }
        thissession = uopy.connect(**config)
        return thissession



    ------------------------------
    Mike Rajkowski
    MultiValue Product Evangelist
    Rocket Internal - All Brands
    US
    ------------------------------



  • 10.  RE: UOPY - Uploading a program

    Posted 11-16-2023 17:06
    Edited by Alejandro Garcia 11-16-2023 17:07

    Thank you,

    I tried your code, but sadly It still hangs. 

    Also, I'm using uopy 1.2.0. I updated to 1.3.0, and I'm unable to establish a session. I keep looking at the documentation, but everything seems to be the same. I keep getting:

    Error [81009] : The RPC failed

    And in the logs I see this message:

    2023-11-16 17:00:16,609 : 91525 : MainThread : _unirpc : __init__ : 169 :ERROR : module 'socket' has no attribute 'TCP_KEEPIDLE'

    I was hoping that maybe the new version would help with my uploading issue, but no idea why I'm not even able to connect :S As soon as I switch to 1.2.0 it connects with no issues and we are just having problems with uploading big programs.



    ------------------------------
    Alejandro Garcia
    Software Engineer Manager
    F W Davison and Company Inc. DBA PrismHR
    MA US
    ------------------------------



  • 11.  RE: UOPY - Uploading a program

    ROCKETEER
    Posted 11-17-2023 09:14

    Alejandro,

    I am not having the problem with Python 3.9.12.  I have found some references online about issues with the socket libraries in older Python releases.  

    What version of Python are you on?



    ------------------------------
    Mike Rajkowski
    MultiValue Product Evangelist
    Rocket Internal - All Brands
    US
    ------------------------------