Hello and thanks to everyone who have replied.
I thought I'd give you all the background to this issue. I have a tool similar to ziggy that I started writing back in 2016, yes even before Git for z/OS was announced. I had a ssh connection to my PC and automatically copied updated files to the PC and issued git commands on the PC and copied updated files back to the z/OS.
That all changed in 2017 when Git for z/OS was announced and I converted the tool to use Git for z/OS. To transfer the files from z/OS datasets to USS files, and vice versa, I've been using IBM's OGET and OPUT commands but I finally got sick of how slow the transfers took and decided to write my own in assembler.
I am no USS guru, having spent my career in the traditional MVS environment. So to my surprise, during the writing of my assembler program, I found that each USS file also had a file format as seen here from the BPXYFTYP macro:
So I coded my assembler program to support these formats. During the transfer it automatically adds/subtracts the correct delimiters based on this value.
But it now seems that my program is the only program that does. :-( Even ISPF is not supporting all the file formats correctly !!!
With Git for z/OS not preserving the file format, my ISPF tables which had a file format of binary were being transfers as text and any X'15' in the table were considered EOL characters, creating invalid tables.
I've now changed my program so that if the file format is "not specified" it will now look at the CCSID and if "binary" it will transfer with no data delimiters. If not "binary" it will assume NL as the data delimiter.
(just in case anyone is interested my assembler program has cut the transfer time for 660 member project from 127 seconds down to 4 seconds. I said OGET and OPUT were slow).
@hank oerlemansNo I did not try changing the name by adding ".bin" to the name. As far as know USS does not use the name to identify file types.
@Giancarlo FrixYour summary of the issue is correct.
I understand that the file format is a z/OS only attribute and it's not available on other platforms, but I'm sure it can be supported in exactly the same way as "zos-working-tree-encoding=" which doesn't get used on other platforms. You could have a .gitattributes entry "zos-file-format=" that could update the file format when updating the working directory.
You said "
git does not set or restore the file format flag, that is actually a known limitation". I haven't seen this limitation documented, its certainly not in the "Git 2.26.2 for z/OS release Notes".
You certainly are right in that minimal utilities support the file format, even ISPF doesn't fully support it.
@Morten KrøyerThere is another issue with git show, which I have reported
here.
Certainly z/OS datasets, that are being treated as text files, that contain X'15' characters have problems. When copied to USS the X'15' are copied fine, but on the way back from USS to z/OS datasets the copy programs have to use X'15' as the end of line, cause any imbedded X'15' to split lines.
I've removed all X'15' from my panels and replaced with some other character, this allows the panels still be treated as text file allowing git-diff to work with the files.
ISPF tables on the other hand have to be treated as binary files, as they could have imbedded X'15'
@Lionel DyckI can understand ziggy ignoring the file format as it seems every other application is too.
But am I understanding you correctly, that you are ignoring the CCSID file encoding altogether and are only using the gitattributes definitions as to whether you transfer your files as binary or text ? If the gitattribute files specifies "binary" do you update the USS tag to binary after your transfer the file.
Also, I can't get my head around supporting ISPF statistics with Git. Do you update the stats based on the USS date/time ? what happens to VRM, specially when members can be updated on different branches - you could be v01.40 on one branch only to go back to say 01.25 if you checkout an old branch.
@All
Again thanks for your support with this issue. As I've said I have now coded my assembler program to honor the file formats if supplied, and if not specified will look for CCSID binary encoding to support binary files.
------------------------------
Gary Freestone
Systems Programmer
Kyndryl Inc
Mt Helen VIC AU
------------------------------
Original Message:
Sent: 06-10-2022 07:23
From: Lionel Dyck
Subject: Is this a setup problem or a problem with Git for z/OS
I did more testing with ZIGI and found that ZIGI does not care what the tags are for a file (text or binary) as it maintains that in the .gitattributes file and that is what ZIGI uses for all copies between z/OS and OMVS. ZIGI also maintains ISPF statistics for all PDS/PDSE members that have them.
See it at GitHub - zigi/zigi: zigi: The git interface for the rest of us!GitHub | remove preview |
| GitHub - zigi/zigi: zigi: The git interface for the rest of us! | Make sure you have git installed on your Mainframe. In order to install git, head on over to https://www.rocketsoftware.com/ and create an account. Then download git, bash, gzip and perl and bring all those files to one directory in USS. | View this on GitHub > |
|
|
------------------------------
LionelDyck <><
https://github.com/zigi
https://github.com/lbdyck
Original Message:
Sent: 06-10-2022 06:29
From: Morten Kroyer
Subject: Is this a setup problem or a problem with Git for z/OS
Lionel Dyck's experience matches ours, I think that git show and git restore somehow expects to deliver the output to a terminal of some sort and inserts newlines or interprets some characters in the table as newlines instead of treating the content as binary data.
------------------------------
Morten Kroyer
Software developer
Rocket Forum Shared Account
Original Message:
Sent: 06-09-2022 10:04
From: Lionel Dyck
Subject: Is this a setup problem or a problem with Git for z/OS
I ran a test using ZIGI (the z/OS ISPF Git Interface) with an ISPF table pds with 3 tables. The table PDS was added as a binary PDS so the members were copied using binary options and tagged accordingly. After adding/committing/pushing, I then cloned the test repo back to z/OS which created a new table pds. I then did a compare (SuperC) of the members and the cloned PDS matched the original.
------------------------------
LionelDyck <><
https://github.com/zigi
https://github.com/lbdyck
Original Message:
Sent: 06-09-2022 08:23
From: Morten Kroyer
Subject: Is this a setup problem or a problem with Git for z/OS
We mange ISPF tables with git and haven't encountered this issue
I didn't know about the file-format attribute and we don't use git-restore, but I encountered a similar issue when using git show, so for ISPF tables we check out the repository to the hash or branch that contains the right version of the table and that seems to restore the content correctly.
We also use git to manage REXX's, Panels, Skeletons etc, but for those we use git show to get the content of a REXX on a specific version.
Morten Krøyer
Software Developer
BEC Financial Technologies
Roskilde - Denmark
------------------------------
Morten Kroyer
Software developer
Rocket Forum Shared Account
Original Message:
Sent: 06-08-2022 02:01
From: Gary Freestone
Subject: Is this a setup problem or a problem with Git for z/OS
Further to this issue, I believe the issue is Git for z/OS not restoring the file format attribute of the USS files, as seen in this example
In this example you can see the CCSID (circled in red) of the file before the commit is binary and again after the restore it's still binary. But the file format (circled in green) before the commit was bin and after the restore it's --------.
From my understanding the file format basically indicates to z/OS the end-of-line (EOL) character(s) used within the file. There are 9 possible values :
0. Not Specified - text data with file format not specified
- Binary - binary data
- NL - text data with lines delimited by NL X'15'
- CR - text data with lines delimited by CR X'0D'
- LF - text data with lines delimited by LF X'0A'
- CRLF - text data with lines delimited by CRLF X'0D0A'
- LFCR - text data with lines delimited by LFCR X'0A0D'
- CRNL - text data with lines delimited by CRNL X'0D15'
- Records - The file is made up of records (similar to VB records in z/OS dataset)
So it seems the git-restore has removed the Binary file format replacing it with Not specified
------------------------------
Gary Freestone
Systems Programmer
Kyndryl Inc
Mt Helen VIC AU
Original Message:
Sent: 06-07-2022 02:09
From: Gary Freestone
Subject: Is this a setup problem or a problem with Git for z/OS
I have a requirement to Git manage ISPF tables. The members have been transferred from a z/OS PDS in binary format and have a CCSID of binary and a USS file format of binary. I am using both the .git/info/attributes and .gitattributes files, with the .git/info/attributes file looking like this:
#
# Default zGit .gitattributes file specifying that all files are in EBCDIC
#
# This file is copied to .git/info/attributes during ZINIT
#
#
* zos-working-tree-encoding=ibm-1047
.gitattributes zos-working-tree-encoding=iso8859-1
.gitignore zos-working-tree-encoding=iso8859-1
and the .gitattributes file that looks like this:
#
# This file is effectively concatenated after .git/info/attributes
#
# For this repo the TABLES directory should be processed as Binary, and the
# $ACBVERS.TABLES directory should be processed as Binary
#
TABLES/** binary
$ACBVERS.TABLES/** binary
From by understanding of git attributes this should tell Git that all files in the TABLES and $ACBVARS.TABLES directory are in binary.
I can successfully issue both the git-add and git-commit to add the files to the repo. However, performing a git-restore of one of the files brings back a file in a totally different format, for example
Before being committed to Git:
BROWSE /u/u700215/zGit-Repositories/U700215/GIT/MAI/TABLES/$SSCAI Line 0000000000 Col 001 132
Command ===> Scroll ===> CSR
*********************************************************** Top of Data *****************************************
.àTIR:...{..$SSCAI ..Ø.Ìh...-........m.¤...o.".å.BOL .............&..ESEQNBR .
SSCAI.EXT.940208.0857.. .. .. .. .......Ø..ì...â............ ..#..I
..Ø..&................ ..#..ISPF.AI.CLIST..KT..Knowledge Tool.........Ø..&................ ..#..ISPF.AI.SKELS..TIRE..Creat own TIRS.
uilder/370.........Ø..&................ ..#..ISPF.AI.PANELS..TIRR..TIRS Runtime/370....Ø..ç................ ..#..ISPF.PROL.HELP..PRO
ntry .........................ry ....................................................
********************************************************** Bottom of Data ***************************************
After performing a git-restore:
BROWSE /u/u700215/zGit-Repositories/U700215/GIT/MAI/TABLES/$SSCAI Line 0000000000 Col 001 132
Command ===> Scroll ===> CSR
*********************************************************** Top of Data ****************************************
.àTIR:...{..$SSCAI ..Ø.Ìh...-........m.¤...o.".å.BOL .............&..ESEQNBR .
..ä`..ELVL ....äã..EDS ....èÁ..EXKEY ....â...ECMTS ....]å..Ø..ç............. .
SSCAI.EXT.940208.0857.. .. .. .. .......Ø..ì...â............ ..#..ISPF.AI.CLIST..ESEC..Expert System Consult........Ø..ì...â........
********************************************************** Bottom of Data **************************************
They are full of hex characters but they should at least have the same number of lines.
I also noticed that after the git-restore the file's CCSID is still binary but its USS file format has changed to from binary to "not specified".
So I guess I have a few questions:
Are my git attributes set correctly for binary files ?
If they are, I believe this is a problem with Git for z/OS. I have push the commit I made to Github and clone it to my PC. The file on the PC matches byte-to-byte to the file I committed.
Is anyone else managing binary files using Git for z/OS successfully, if yes, what are you doing differently ?
------------------------------
Gary Freestone
Systems Programmer
Kyndryl Inc
Mt Helen VIC AU
------------------------------