Encoding string from Universe to Python

Hi all

I'm trying to make sense of the encoding of strings coming and going between Basic and Python.

Our Universe database stores its data using OEM encoding, so to pass data to Python we do a series of transformations inside Basic to convert them to UTF-8.

Now I want to pass this OEM strings, as they are, directly into Python and do the conversion there, but I'm having some trouble.

ALL.CHAR.STR=''
ALL.CHAR.STR=ALL.CHAR.STR:CHAR(1):CHAR(2):CHAR(3):CHAR(4):CHAR(5):CHAR(6):CHAR(7):CHAR(8):CHAR(9):CHAR(10):CHAR(11):CHAR(12):CHAR(13):CHAR(14):CHAR(15)
* rest of characters...
ALL.CHAR.STR=ALL.CHAR.STR:CHAR(241):CHAR(242):CHAR(243):CHAR(244):CHAR(245):CHAR(246):CHAR(247):CHAR(248):CHAR(249):CHAR(250):CHAR(251):CHAR(252):CHAR(253):CHAR(254):CHAR(255)

RESPUESTA=PyCallFunction('encoding_test','test_encoding', ALL.CHAR.STR)

On the Python side I'm doing this

def test_encoding(universe_str: str) -> str:
    for c in universe_str:
        print(f'{ord(c)}')
    return 'OK'

This prints the following values:

1
2
3
4
...
125
126
127
65533
65533
...
65533
65533

How can I get the characters in Python and convert them to UTF-8?

Hi Héctor,

Since you're going to Python it might be easier and more efficient to do the conversion there.  Strings in Python are already Unicode so there might already be a conversion going on in PyCallFunction. If it is, it's a simple matter of using the Python string encode method:

utf8_string = universe_str.encode('utf-8')

If it's coming in as a byte string you can first convert it to Unicode and then back to UTF-8 like this:

unicode_string  = universe_str.decode('ascii')

utf8_string = unicode_string.encode('utf-8')

If you have a normal python string you want to convert back to ascii to pass back to Universe:

ascii_string   = normal_unicode_python_string.encode('ascii')

Hi Héctor,

Since you're going to Python it might be easier and more efficient to do the conversion there.  Strings in Python are already Unicode so there might already be a conversion going on in PyCallFunction. If it is, it's a simple matter of using the Python string encode method:

utf8_string = universe_str.encode('utf-8')

If it's coming in as a byte string you can first convert it to Unicode and then back to UTF-8 like this:

unicode_string  = universe_str.decode('ascii')

utf8_string = unicode_string.encode('utf-8')

If you have a normal python string you want to convert back to ascii to pass back to Universe:

ascii_string   = normal_unicode_python_string.encode('ascii')

Hi Joe

My problem is that the data that I'm getting in Python is not what I expect.

If I try to encode the string as UTF-8 this is what I get:

b'\\x01\\x02\\x03\\x04\\x05\\x06\\x07\\x08\\t\\n\\x0b\\x0c\\r\\x0e\\x0f\\x10\\x11\\x12\\x13\\x14\\x15\\x16\\x17\\x18\\x19\\x1a\\x1b\\x1c\\x1d\\x1e\\x1f
 !"#$%&\\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\\\]^_`abcdefghijklmnopqrstuvwxyz{|}~
\\x7f\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd
\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd\\xef\\xbf\\xbd'

As soon as I run off of the ASCII space I only get \\xef\\xbf\\xbd

Hi Hector,

That confused me for a bit. Then I figured out what you are trying to do. It looks like you've passed a string with all the ascii characters from 1 to 255. Python tried to convert them to utf-8 but characters 128 to 255 don't have utf-8 defined equivalents. The Python converter has two options to deal with those characters, ignore or replace. If you pick ignore they're just stripped out. If you pick replace, it replaces each character with the unicode error subsitute character which happens to be "\\xef\\xfb\\xbd". That's what you're seeing in your string. All the high number characters converted to that sequence

You can duplicate this with a short bit of Python:

Hi Joe

My trouble is not on the Python side, but on BASIC.

I have data in OEM encoding on BASIC that I need to pass to Python, and the PyCall function is converting the OEM data to UTF-8 incorrectly. I'd like to know how to pass this characters to Python without losing any information.

Hector,

Have you tried the PyByteCallFunction, this will send the data as a byte string and not a string?

Hi Mike,

I don't see PyByteCallFunction documented anywhere. I downloaded the latest Universe Python pdf but there's no mention of it. Do you have newer documentation?

Hi Hector,

From what I see, the Universe python interface is trying to automatically convert the ASCII strings from Basic to Unicode. The encoding method it's using must not understand characters 128+ so it's converting them to the Unicode "unknown character" sequence mentioned above.

The documentation mentions a "config.encoding" variable that defines the type of encoding used. Whatever that is, it's not a good match. The documentation says you can change it but i don't see any instructions as to how and I can't test it. It might be as simple as putting "config.encoding=xxxx" somewhere in your code. The u2py.py routine use "config.encoding" in a number of places.

If you import this in your code " from _u2py import *" it looks like you'll get the config object and can look at it. (When I try it I get a "U2 Python Package is not licensed.")

It looks like the encoding you want to use is "cp437".  On my system the encoding files are all stored in /usr/uv/python/lib/python3.4/encodings.

Maybe you can play with those  and figure it out.

Joe G.

Sorry about that, the PyByteCallFunction is in UniData.  UniVerse is a bit different.   Another thing to check is if you are using NLS and what codepage you are using if it is on.

Sorry about that, the PyByteCallFunction is in UniData.  UniVerse is a bit different.   Another thing to check is if you are using NLS and what codepage you are using if it is on.