Open-source Languages & Tools for z/OS

Hi,

I am working on a task that requires a few file conversions and copying / writing to other files.

1. Use JCL to convert an MVS file to a text file.
2. Executes a python program on the mainframe.
3. The python program takes the text file as input, performs some operations on it and writes output to another text file.
4. Resume to JCL which converts the new text file to another MVS file.

The problem I am having is that when the new text file is being converted / written to another MVS file, the newline character \n is not recognised.

Even when using ‘a+’ as such:

with open(‘file.txt’, ‘a+’ ) as f:

f.write(rowOutput)

The final MVS file keeps having all the content on one line, instead of each row of the text file on a separate line in the MVS file as desired.

I have tried adding code that appends “\n”, “\n” and even “\\n\n” at the end of every rowOutput. but the MVS file still shows all content as one line.

Output of text file using \\n\n:
{key1:value1, key2:value2}\n
{key3:value3, key4:value4}\n

Output of MVS file being written to using \\n\n:
{key1:value1,key2:value2}\n□{key3:value3,key4:value4)\n□

I assume the □ character is supposed to be the ‘\n’ in ‘\\n\n’.

The operations being performed are splitting the initial MVS file row content, and creating a dictionary using a key list and a value list, and writing the created dictionary for that row in the new text file.

There may be something to do with the encoding that I am unsure about.

Can anyone assist with this?

Hope this makes sense.

Hi Zachary,
What version of python are you using?
And could you provide JCL listing- this error might be caused by JCL.

Alex

Hi Alex,
The version of python being used is 3.7.2.
Here is the JCL listing:

//XXXXXXXX JOB (,XXXXXX),MSGCLASS=V,NOTIFY=XXXXXX,
// CLASS=T
//MAIN SYSTEM=ANZH
//OCOPY EXEC PGM=IKJEFT01
//SYSPRINT DD SYSOUT=
//SYSTSPRT DD SYSOUT=*
//INDD DD DISP=SHR,DSN=FILE.INPUT
//OUTDD DD PATH=’/path/to/file.csv’,
// PATHOPTS=(OWRONLY,OCREAT),
// PATHMODE=(SIRWXU,SIRGRP),
// PATHDISP=(KEEP,DELETE)
//SYSTSIN DD *
OCOPY INDD(INDD) OUTDD(OUTDD) TEXT CONVERT(YES) PATHOPTS(USE) FROM1047
/*
//* BATCH UNIX PROGRAM SPAWN LOCAL (BPXBATSL)
//*
//BATCHPGM EXEC PGM=BPXBATSL
//*
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//STDENV DD *
LIBPATH=/var/python/python36:/var/python/python36/lib
PATH=/var/python/python36/bin
_bpxk_autocvt=“on”
_bpx_shareas=must
_bpx_batch_spawn=yes
/*
//STDPARM DD *
PGM /var/python/python36/bin/python /path/to/program.py
/*
//OCOPY2 EXEC PGM=IKJEFT01
//SYSPRINT DD SYSOUT=*
//SYSTSPRT DD SYSOUT=*
//OUTDD DD DSN=FILE.OUTPUT,
// DISP=(NEW,CATLG,DELETE),RECFM=VB,LRECL=32000,
// SPACE=(CYL,(10,10),RLSE)
//INDD DD PATH=’/output/file.txt’,
// PATHOPTS=(ORDONLY),
// PATHMODE=(SIRWXU,SIRGRP),
// PATHDISP=(KEEP,KEEP)
//SYSTSIN DD *
OCOPY INDD(INDD) OUTDD(OUTDD) TEXT CONVERT(YES) PATHOPTS(USE) TO1047
/*
//

Hello Zachary,

I’ve tried to recreate your issue but unfortunately could not get similar results. The issue indeed appears to be related to encoding, so let’s check the encoding of your data:

1. What is the encoding of your input and output datasets (i.e. MVS files)? What do you use to view them? Is it ISPF View, or Browse, or something else?

2. For the output USS file created by python:

• What does ‘ls -T file.txt’ report?
• Does the cat command show readable file contents?

Also, your JCL looks like it’s supposed to convert the data between ASCII and EBCDIC. However, the parameters “CONVERT(YES) FROM|TO1047” do not necessarily mean conversion between ASCII and EBCDIC. By default, this enables conversion between the IBM-037 and IBM-1047 code pages; defaults on your system may vary. You can use CONVERT((BPXFX311)) to convert between ASCII and EBCDIC; however, this creates an untagged file on USS which might not be read properly by Python. Please notice the double parentheses around BPXFX311.

Regards,
Vladimir

I don’t know why this happens, but I did hack up a fix. The key is that where Python (and us programmers) expect/provide a \n, z/OS is actually providing/expecting a \x85, which is rendered as □. Observe the following Python, which creates working results when run with Zachary’s JCL:

ACTUAL_NEWLINE = '\x85' # ugh

def convert(infile, outfile):
    # There's meant to be a lineterminator= parameter for this,
    # but it doesn't work for readers, so we have an ugly hack instead.
    reader = csv.DictReader(infile.read().split(ACTUAL_NEWLINE), dialect='unix')
    for row in reader:
        json.dump(dict(row), outfile)
        outfile.write(ACTUAL_NEWLINE)

One has to be careful not to let any \ns sneak in, as the result of such will be unexpected and/or wrong.

(This \x85 is undoubtedly an extended ASCII control character, as in EBCDIC code pages it’s simply the letter e.)