Unidata and UTF-8

Hello community, I am facing this problem and hope anybody could help me. My site is running Unidata over Windows Server and MVIS to provide webservices.
The problem is when the webservice receive extended characters, for example: á,é,ñ,Ñ and so on, the webservice subroutine uses UDO object to get the data from JSON parameter and write it to the file, but in this case of special characters, the data saved is completely different with other characters. And it is worse when that same data is pulled out by other webservices. All third-party providers that my site interact with indicate that the application cannot handle UTF-8 encoding properly.

I will appreciate any help you could provide me.

Thanks.

We have been using UOJ for many years now. Our middleware runs on Apache Tomcat interfaces with Universe or Unidata.
On the web service side of using Java, in this case, here is a snippet of code we have used for our web services to open the connection:

UniSession session = ConnectionManager.getInstance().getMasterUniJava().openSession();
session.setSessionEncoding("ISO-8859-1");

Posted: 08-04-2022 18:27

Hello community, I am facing this problem and hope anybody could help me. My site is running Unidata over Windows Server and MVIS to provide webservices.
The problem is when the webservice receive extended characters, for example: á,é,ñ,Ñ and so on, the webservice subroutine uses UDO object to get the data from JSON parameter and write it to the file, but in this case of special characters, the data saved is completely different with other characters. And it is worse when that same data is pulled out by other webservices. All third-party providers that my site interact with indicate that the application cannot handle UTF-8 encoding properly.

I will appreciate any help you could provide me.

Thanks.

Hi Jamie,

I have the same problem, using UniVerse.  I want to read a record, and do nothing with it except pass it to the client for processing, and for client to pass similar back to UV server to be recorded elsewhere, without touching anything in the record. 

Current requirement is for pictures to be stored on the server and shown through the Microsoft client.  There must be many others where we connect with functionality outside UV, like yours.  Workarounds I have had suggested are ridiculously complex.

Our UV supplier tells me we are too small for Rocket to consider adding something like that to UniVerse - we would need to have thousands of users before being listened to.

QM has it - a different READx verb.  Sounds pretty simple to me.

If others have this problem, please ask the question - there may be enough requirement for it to make the small change.

All the best, Kate

Hello community, I am facing this problem and hope anybody could help me. My site is running Unidata over Windows Server and MVIS to provide webservices.
The problem is when the webservice receive extended characters, for example: á,é,ñ,Ñ and so on, the webservice subroutine uses UDO object to get the data from JSON parameter and write it to the file, but in this case of special characters, the data saved is completely different with other characters. And it is worse when that same data is pulled out by other webservices. All third-party providers that my site interact with indicate that the application cannot handle UTF-8 encoding properly.

I will appreciate any help you could provide me.

Thanks.

@Jaime Luna

As we discussed in the support case if you just wish to read and write the data in UniData without having the read / write statements do any conversion, then you can use the OSREAD / OSWRITE / OSBREAD / OSBWRITE and the NOCONVERT statement. I can't remember if you fully explored that option. In terms of displaying the data that is a different discussion. You may also find the tip for Kate below useful.

@Kate Stanton

In terms of storing pictures in UniVerse and UniData we have several customers who do this now in their application. One of the largest UniData customers does this for any items returned to it stores as a proof of condition on return. It is not as complex as you may think to do this. The principle is simple to demonstrate using a .NET application front end, the customers who are displaying these pictures have some type of .NET or Web Based front end.

There is an article in the knowledge base covering the steps https://my.rocketsoftware.com/RocketCommunity/articles/Tech_Note/How-to-save-and-retrieve-image-data-in-a-U2-Database-via-U2-ADO-NET-provider-1478610310159

Images are in binary format, and If you attempt to store binary data in a U2 file, the binary data may conflict with the various marks used by U2. This prevents you from storing binary data directly (UniData does have the statements described to Jaime above as well). You can use the .NET Convert.FromBase64String and Convert.ToBase64String functions to convert the binary data to and from a hex string that can be easily saved to and retrieved from a U2 file using a "READWRITE_IMAGE" subroutine.

To Convert to Base64 string

Convert the picture box image to a memory string, and then convert it to Base64 using the .Net ToBase64String function.

Dim ms As New IO.MemoryStream()
Dim base64str As String
PictureBox1.Image.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg)
'
Dim bt() As Byte ReDim bt(ms.Length)
Dim br As BinaryReader = New BinaryReader(ms) ms.Seek(0, SeekOrigin.Begin)
bt = br.ReadBytes(ms.Length)
br.Close()
'
base64str = Convert.ToBase64String(bt)

To Convert from Base64 string

Convert the Base64 data in a U2 file back to a byte array using the .Net FromBase64String function. The byte array data is then converted back to the image data displayed in a picture box.

Dim base64str As String
Dim byteary() As Byte
PictureBox1.Image = Nothing ' base64str data is from a U2 Database file
byteary = System.Convert.FromBase64String(base64str)
Dim ms As New IO.MemoryStream(byteary) Me.PictureBox1.Image = Image.FromStream(ms)

The sample .NET application in the knowledge base article then uses this READWRITE_IMAGE subroutine
​

Here is the sample READWRITE_IMAGE subroutine with five parameters.

SUBROUTINE READWRITE_IMAGE(FILENAME,KEY,IMAGE,STATENAME,ACTION)
ERRORNO = 0
OPEN FILENAME TO FILE.IMAGE ELSE ERRORNO = 9
IF ERRORNO = 0 THEN
READ STATE.REC FROM FILE.IMAGE, KEY ELSE ERRORNO = 9
IF ERRORNO = 9 THEN
IMAGE=""
CLOSE FILE.IMAGE
RETURN
END
*
IF ACTION = "2" THEN
STATE.REC = STATENAME:@FM:IMAGE
WRITE STATE.REC ON FILE.IMAGE, KEY
END ELSE
STATENAME = STATE.REC<1>
IMAGE = STATE.REC<2>
END
CLOSE FILE.IMAGE
END
RETURN

I would also think that with the addition of python to both UniVerse and UniData doing a Base64 conversion could be done in python as well.
​

Hello community, I am facing this problem and hope anybody could help me. My site is running Unidata over Windows Server and MVIS to provide webservices.
The problem is when the webservice receive extended characters, for example: á,é,ñ,Ñ and so on, the webservice subroutine uses UDO object to get the data from JSON parameter and write it to the file, but in this case of special characters, the data saved is completely different with other characters. And it is worse when that same data is pulled out by other webservices. All third-party providers that my site interact with indicate that the application cannot handle UTF-8 encoding properly.

I will appreciate any help you could provide me.

Thanks.

Hi,
We have exactly the same problem and we need a solution from Rocket to deal with UTF-8 characters.  Please note that although it is relatively easy to deal with the data storage, there are other related issues such as sorting data for reports, lookups; Displaying the data in SBClient etc.

@Jonathan Smith are there any plans for this to be handled "natively" within Unidata?

Samppa​