Rocket U2 | UniVerse & UniData

 View Only
  • 1.  Alternative to DynamicArrays in Python

    Posted 14 days ago

    Hi all

    I've been doing some experiments accessing UniVerse data via Python for some time and I had my own library before the official Rocket uopy utilities were available to me.

    One thing I implemented is my own custom representation of a UniVerse record as a Python dictionary and I was wondering if it would be useful to anyone. Maybe I can add it to the Python github.

    Anyway, I'll explain the logic in my implementation and why I still use it in some places instead of the "natural" DynArrays.

    My implementation stems from the structure of a lot of the data we use here, as a lot of records are sparsely populated (I use to call my own records "sparse records" to differentiate them from the DynArray)

    Example: A register with 90 values, which only 4 contain real data. One of them is a multivalue that also contains several "empty" fields.

    1. "Value1"
    2. "Value2"
    3. ""
    4. ""
    ...
    60. "a|b|c|d||||||||m|n"
    ...
    90. "Value90"

    Using a DynArray results in a mostly empty list, with the footprint that this involves, maily using thousands of huge sparsely populated records.

    It will look like this

    ['Value1', 'Value2', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', ['a', 'b', 'c', 'd', '', '', '', '', '', '', '', '', 'm', 'n', '', '', '', '', '', '', '', '', '', '', ''], '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', 'Value90']

    In my solution I'm using a defaultdict. This is a variation of the usual Python dictionary that returns a default value when a key is not present.

    class UniverseDict(defaultdict):

    def __init__(self):
    super().__init__(lambda: None)


    In this case the value is None, as sometimes I need to tell if a value is not present versus if a value is just an empty string. But it can return an empty string if that is more convenient.

    The previous example would look like this in my sparse dictionary:

    {
        0: 90,
        1: {
    			0: 1,
    			1: 'Value1'
        },
        2: {
            0: 1,
            1: 'Value2'
        },
        60: {
            0: 14,
            1: 'a',
            2: 'b',
            3: 'c',
            4: 'd',
            13: 'm',
            14: 'n'
        },
        90: {
            0: 1,
            1: 'Value90'
        }
    }


    The keys are integers that are equivalent to the indexes used in a DynArray. record[1] = 'Value1', record[90] = 'Value90', etc
    The key 0 is special as its value is the number of values in the register or in a value. In this case, record[0] is 90, as there are 90 values. record[60][0] is 14 as there are 14 values. The default value for this entry is 1, to represent single-valued entries (such as 'Value1' or 'Value2').

    My trouble arises when using this dictionaries along with uopy. I have made a custom class derived from uopy.File so I can create my own records. It's just a copy of uopy.File with a call to my constructor instead of a call to the constructor of DynArray.

    class SparseFile(uopy.File):
    
        def __init__(self, name, dict_flag=0, session=None):
            super().__init__(name, dict_flag, session)
    
        def read_uv_register(self, record_id, lock_flag=0):
            _logger.debug("Enter", record_id, lock_flag)
    
            self._check_opened()
            with self._lock:
                # some code
                if resp_code != 0:
                    # other code
                else:
                    record = UVRegister(record_id, datos=in_packet.read(2), session=self._session)
    
            _logger.debug("Exit", record)
            return record


    This is a house-made hack and I was wondering if something like this could be integrated into uopy. Maybe not the SparseRecord class per se but allowing uopy.File to have another parameter in the read_uv_register method that allows to pass a factory method or object so that custom representations of UniVerse records can be built on the fly.

    Also, if this approach may be useful to somebody I would be glad to add the whole code to the github repository, as it's quite lengthy.



    ------------------------------
    Hector Cortiguera
    Rocket Forum Shared Account
    ------------------------------


  • 2.  RE: Alternative to DynamicArrays in Python

    ROCKETEER
    Posted 9 days ago
    Hector,

    Are you determining what fields have data on the server?
    Or are you passing a list of fields you want to returned?

    Both u2py an uopy have methods to read/write named fields ( i.e u2py.File.readnamedfields and uopy.File.read_named_fields  )

    This allows you to only bring back the fields requested for the id ( or ids in uopy )

    Note that the methods also deal with the OCONV/ICONV issues when reading/writing, yet this does not return it in a Python Dict item.

    I have some examples around u2py, and described the solution in another thread in the forum.  below is a snipit from my reply in that thread, or you can view the entire thread here

    "I have a few examples of functions I wrote to do the reads and writes, yet none with extensive lock handling at this time.

    Note that the topic of an ORM has come up in the past, I have tinkered with writing my own from time to time.  I have a few functions at the Rocket Github site, that I use as a poor man's ORM, I tend to use it more than my more elaborate attempts ( which in most cases uses these functions ).

    In the u2support.py module, there is u2PyReadDict and u2PyWriteDict, that allows you to use a list of U2 Dictionary Items, when reading from the U2 file, it returns a Python Dictionary, that you can use/update and write back.

    i.e.
    python> import u2support
    python> u2dictitems = [ "FIRST_NAME", "LAST_NAME", "CITY", "STATE", "ZIP" ]
    python> pydict = u2support.u2ReadPyDict( "MEMBERS", "0110", u2dictitems )

    Welcome to the XDEMO Account
    Version: 3.1.5

    python> pydict
    {'FIRST_NAME': 'Anita', 'LAST_NAME': 'Johnson', 'CITY': 'New Glasgow', 'STATE': 'Nebraska', 'ZIP': '01870'}
    python>

    Note this sample code is not that extensive, and does not handle locking at this time."

    --- 
    Also note that the above example is built on u2py, but same can be done with the uopy module.


    ------------------------------
    Mike Rajkowski
    support
    Rocket Internal - All Brands
    DENVER CO US
    ------------------------------



  • 3.  RE: Alternative to DynamicArrays in Python

    Posted 9 days ago

    Hi, Mike

    You raise some interesting points I haven't considered.

    First of all: no, I'm not reading fields separately. This just reads a whole register, it's just another representation for the result given by the read_uv_register method in uopy.File. All my processing is client-side and is just another way to represent the data.

    I'm not working with dictionaries at the moment as most of the work done with these tools I'm making is index-based but is something that comes up from time to time and it's worth considering. Thanks for your examples.



    ------------------------------
    Hector Cortiguera
    Rocket Forum Shared Account
    ------------------------------



  • 4.  RE: Alternative to DynamicArrays in Python

    Posted 7 days ago
    Following the hack I did rewriting the uopy.File read_uv_register method I have created a more flexible way to do this:
    class LambdaFile(uopy.File):
    
        def read_uv_register(self, record_id, lock_flag=0, register_factory=None):
    
            self._check_opened()
            with self._lock:
                # some code
                if resp_code != 0:
                    # other code
                else:
                    if not register_factory:
                        record = DynArray(in_packet.read(2), session=self._session)
                    else:
                        record = register_factory(record_id, in_packet.read(2), self._session)​


    This allows to pass a lambda function to the read_uv_register method in the following way:

    ql = LambdaFile("FTPANT", dict_flag=0)
    reg = ql.read_uv_register('PROG*POSVENTA*PTPT',
                              register_factory=lambda r, d, s: UVRegister(id_reg=r,                                                                      
                                                                          datos=d,
                                                                          session=s))                         


    So, if someone wants to build a new custom class to represent a UniVerse register there is no need to touch uopy under the hood.

    The lambda takes the three parameters that read_uv_register passes to the constructor to the DynArray class. But you can also pass extra parameters as you like.

    In this case I can add a custom dictionary I built elsewhere:

    reg = ql.read_uv_register('PROG*POSVENTA*PTPT',
                              register_factory=lambda r, d, s: UVRegister(id_reg=r,
                                                                          diccionario=my_custom_dict,
                                                                          datos=d,
                                                                          session=s))    


    ------------------------------
    Hector Cortiguera
    Rocket Forum Shared Account
    ------------------------------