Skip to main content

I have a problem with control characters in records. Particularly LF, CR and TAB.

Is there a way to find records that contain them?

LF messes with any statements I have tried in UniVerse SQL and TCL.



------------------------------
Greg Clitheroe
Rocket Forum Shared Account
------------------------------

I have a problem with control characters in records. Particularly LF, CR and TAB.

Is there a way to find records that contain them?

LF messes with any statements I have tried in UniVerse SQL and TCL.



------------------------------
Greg Clitheroe
Rocket Forum Shared Account
------------------------------

I ended up doing it in SQL Server through the ODBC linked server.

The criteria can be set up as an I-descriptor (or EVAL) to avoid the line feed being acted on but AFAIK an I-descriptor cannot be used in a LIKE expression.

I would still be curious if it can be done in UniVerse on a terminal.



------------------------------
Greg Clitheroe
Rocket Forum Shared Account
------------------------------


I have a problem with control characters in records. Particularly LF, CR and TAB.

Is there a way to find records that contain them?

LF messes with any statements I have tried in UniVerse SQL and TCL.



------------------------------
Greg Clitheroe
Rocket Forum Shared Account
------------------------------

UniVerse or UniData?

I'm assuming you are having a problem with a hashed file and not a directory although the solution will work for any file type.  The quickest solution would be to create an I-Type or Virtual dictionary item that scans a record for a given character.  That will allow you to build selectlists to be fed into other cleanup routines.  

In the dictionary of the suspect data file, create a dictionary item that searches for the offending character.  Below is an example of a UniVerse I-Type that looks for a tab character.

>ED DICT DATA.FILE TAB.CHAR
This is a Type "I" Descriptor last compiled on 01/02/25 at 19:43.
20 lines long.

----: P
0001: I
0002: IF INDEX(@RECORD,CHAR(9),1) THEN 1 ELSE ''
0003:
0004: Tab
0005: 3R
0006: S
0007:
0008:

You can then execute a SELECT statement to find your bad records.

Example: SELECT DATA.FILE WITH TAB.CHAR

The result will be an active selectlist of records that contain a tab character.

You can create separate dict items for each of the characters or combine your character search in the I-Type by nesting your IF statement.

If you don't know exactly what non-printing character your looking for, the solution is a bit different.  Let me know if that's the case and I'll offer a different solution.

If the problem exists in multiple files, copy your bad character dictionary item to the VOC file and it will be usable on all of your data files.



------------------------------
Steve Murray-Wolf
Owner
Rocket Forum Shared Account
Firestone CO US
------------------------------

I have a problem with control characters in records. Particularly LF, CR and TAB.

Is there a way to find records that contain them?

LF messes with any statements I have tried in UniVerse SQL and TCL.



------------------------------
Greg Clitheroe
Rocket Forum Shared Account
------------------------------

In the past I have used the following select to find unprintable characters in a file,

SELECT "filename" WITH EVAL "CONVERT(@AM:@VM:@SM,'',@RECORD)" NE EVAL "OCONV(CONVERT(@AM:@VM:@SM,'',@RECORD),'MCP')"



------------------------------
Hans Morawa
n/a
Rocket Forum Shared Account
------------------------------

In the past I have used the following select to find unprintable characters in a file,

SELECT "filename" WITH EVAL "CONVERT(@AM:@VM:@SM,'',@RECORD)" NE EVAL "OCONV(CONVERT(@AM:@VM:@SM,'',@RECORD),'MCP')"



------------------------------
Hans Morawa
n/a
Rocket Forum Shared Account
------------------------------

I was just about to post a similar answer!  The trick is that the "MCP" conversion converts all non-printable characters to periods. It makes for a quick way to find strings with any non-printable characters.



------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US
------------------------------