Open-source Languages & Tools for z/OS

 View Only
Expand all | Collapse all

Is this a setup problem or a problem with Git for z/OS

  • 1.  Is this a setup problem or a problem with Git for z/OS

    PARTNER
    Posted 06-07-2022 02:10
    Edited by Gary Freestone 06-07-2022 23:39
    I have a requirement to Git manage ISPF tables.  The members have been transferred from a z/OS PDS in binary format and have a CCSID of binary and a USS file format of binary. I am using both the .git/info/attributes and .gitattributes files, with the .git/info/attributes file looking like this:

    #
    # Default zGit .gitattributes file specifying that all files are in EBCDIC
    #
    # This file is copied to .git/info/attributes during ZINIT
    #
    #

    * zos-working-tree-encoding=ibm-1047

    .gitattributes zos-working-tree-encoding=iso8859-1
    .gitignore zos-working-tree-encoding=iso8859-1

    and the .gitattributes file that looks like this: 

    #
    # This file is effectively concatenated after .git/info/attributes
    #
    # For this repo the TABLES directory should be processed as Binary, and the
    # $ACBVERS.TABLES directory should be processed as Binary
    #
    TABLES/** binary
    $ACBVERS.TABLES/** binary

    From by understanding of git attributes this should tell Git that all files in the TABLES and $ACBVARS.TABLES directory are in binary. 

    I can successfully issue both the git-add and git-commit to add the files to the repo.  However, performing a git-restore of one of the files brings back a file in a totally different format, for example

    Before being committed to Git:
    BROWSE /u/u700215/zGit-Repositories/U700215/GIT/MAI/TABLES/$SSCAI Line 0000000000 Col 001 132
    Command ===> Scroll ===> CSR
    *********************************************************** Top of Data *****************************************
    .àTIR:...{..$SSCAI ..Ø.Ìh...-........m.¤...o.".å.BOL .............&..ESEQNBR .
    SSCAI.EXT.940208.0857.. .. .. .. .......Ø..ì...â............ ..#..I
    ..Ø..&................ ..#..ISPF.AI.CLIST..KT..Knowledge Tool.........Ø..&................ ..#..ISPF.AI.SKELS..TIRE..Creat own TIRS.
    uilder/370.........Ø..&................ ..#..ISPF.AI.PANELS..TIRR..TIRS Runtime/370....Ø..ç................ ..#..ISPF.PROL.HELP..PRO
    ntry .........................ry ....................................................
    ********************************************************** Bottom of Data ***************************************

    After performing a git-restore:
    BROWSE /u/u700215/zGit-Repositories/U700215/GIT/MAI/TABLES/$SSCAI Line 0000000000 Col 001 132
    Command ===> Scroll ===> CSR
    *********************************************************** Top of Data ****************************************
    .àTIR:...{..$SSCAI ..Ø.Ìh...-........m.¤...o.".å.BOL .............&..ESEQNBR .
    ..ä`..ELVL ....äã..EDS ....èÁ..EXKEY ....â...ECMTS ....]å..Ø..ç............. .
    SSCAI.EXT.940208.0857.. .. .. .. .......Ø..ì...â............ ..#..ISPF.AI.CLIST..ESEC..Expert System Consult........Ø..ì...â........
    ********************************************************** Bottom of Data **************************************

    They are full of hex characters but they should at least have the same number of lines.

    I also noticed that after the git-restore the file's CCSID is still binary but its USS file format has changed to  from binary to  "not specified". 

    So I guess I have a few questions:

    Are my git attributes set correctly for binary files ?

    If they are,  I believe this is a problem with Git for z/OS.  I have push the commit I made to Github and clone it to my PC.  The file on the PC matches byte-to-byte to the file I committed.

    Is anyone else managing binary files using Git for z/OS successfully, if yes, what are you doing differently ?

    ------------------------------
    Gary Freestone
    Systems Programmer
    Kyndryl Inc
    Mt Helen VIC AU
    ------------------------------


  • 2.  RE: Is this a setup problem or a problem with Git for z/OS

    PARTNER
    Posted 06-08-2022 02:01
    Further to this issue, I believe the issue is Git for z/OS not restoring the file format attribute of the USS files, as seen in this example 

    In this example you can see the CCSID (circled in red) of the file before the commit is binary and again after the restore it's still binary. But the file format (circled in green) before the commit was bin and after the restore it's --------.   

    From my understanding the file format basically indicates to z/OS the end-of-line (EOL) character(s) used within the file.   There are 9 possible values :

           0. Not Specified - text data with file format not specified
    1. Binary - binary data
    2. NL -  text data with lines delimited by NL X'15'
    3. CR - text data with lines delimited by CR X'0D'
    4. LF - text data with lines delimited by LF  X'0A'
    5. CRLF - text data with lines delimited by CRLF  X'0D0A'
    6. LFCR - text data with lines delimited by LFCR  X'0A0D'
    7. CRNL - text data with lines delimited by CRNL X'0D15'
    8. Records - The file is made up of records (similar to VB records in z/OS dataset)
    So it seems the git-restore has removed the Binary file format replacing it with Not specified


    ------------------------------
    Gary Freestone
    Systems Programmer
    Kyndryl Inc
    Mt Helen VIC AU
    ------------------------------



  • 3.  RE: Is this a setup problem or a problem with Git for z/OS

    PARTNER
    Posted 06-08-2022 02:35
    The saga continues....

    As already demonstrated, Git for z/OS doesn't preserve the binary file format, but what about the others ?   I thought I'd better do a test. And here is the result :
    And it seems the answer is no.

    How important is this ? Probably not very especially if its taken 5 years to find this issue.

    With my z/OS background I have no idea if other platforms like, linux, windows etc  have this capability of specifying file formats.  But I'm guessing no, which means its not a general git problem, it only relates to Git for z/OS. 

    Maybe it needs a z/OS only solution, like a new optional z/OS attribute for the .git/info/attributes and .gitattributes such as  zos-file-format=CRNL

    I'm very interested to see what Rocket thinks of this issue

    ------------------------------
    Gary Freestone
    Systems Programmer
    Kyndryl Inc
    Mt Helen VIC AU
    ------------------------------



  • 4.  RE: Is this a setup problem or a problem with Git for z/OS

    Posted 06-09-2022 01:40
    What happens if you transfer tabacb.bin rather than tabacb ?

    ------------------------------
    hank oerlemans
    Lead Software Engineer
    HCL America Inc.
    North Sydney NSW AU
    ------------------------------



  • 5.  RE: Is this a setup problem or a problem with Git for z/OS

    ROCKETEER
    Posted 06-09-2022 15:56
    Edited by Giancarlo Frix 06-09-2022 15:58
    Hi Gary,

    Just to make sure I am apprehending the issue correctly, let me restate the issue as I understand it. Your issue is that when you restore a file in git that originally had a non-null file format flag (as displayed with `ls -H`), it is no longer present after the restore, which causes ISPF to display the file differently. The file contents are NOT actually changing. Please let me know if I am correct so far.

    To determine if the file has changed, you can run `sha512 filename` on the file before and after the restore. If it gives identical output, then the file likely has identical contents.

    If your issue is solely that git does not set or restore the file format flag, that is actually a known limitation. The file format flag is specific to z/OS and not present on other platforms, and git does not natively have a way to save that metadata. Furthermore, very few utilities on the system respect that flag at all, so it is generally considered of minimal import or use. This is actually the first time that a customer has brought up this issue to us. In your situation, what is driving the need for this flag to be maintained?

    ------------------------------
    Giancarlo Frix
    Lead Open-Source Porting Engineer
    Rocket Software Inc.
    ------------------------------



  • 6.  RE: Is this a setup problem or a problem with Git for z/OS

    ROCKETEER
    Posted 06-09-2022 15:56
    Edited by Giancarlo Frix 06-09-2022 15:59
    [Ignore this post}


  • 7.  RE: Is this a setup problem or a problem with Git for z/OS

    Posted 06-09-2022 09:57

    We mange ISPF tables with git and haven't encountered this issue

    I didn't know about the file-format attribute and we don't use git-restore, but I encountered a similar issue when using git show, so for ISPF tables we check out the repository to the hash or branch that contains the right version of the table and that seems to restore the content correctly.

    We also use git to manage REXX's, Panels, Skeletons etc, but for those we use git show to get the content of a REXX on a specific version.

    Morten Krøyer
    Software Developer
    BEC Financial Technologies
    Roskilde - Denmark



    ------------------------------
    Morten Kroyer
    Software developer
    Rocket Forum Shared Account
    ------------------------------



  • 8.  RE: Is this a setup problem or a problem with Git for z/OS

    Posted 06-09-2022 10:05
    I ran a test using ZIGI (the z/OS ISPF Git Interface) with an ISPF table pds with 3 tables.   The table PDS was added as a binary PDS so the members were copied using binary options and tagged accordingly. After adding/committing/pushing, I then cloned the test repo back to z/OS which created a new table pds. I then did a compare (SuperC) of the members and the cloned PDS matched the original.

    ------------------------------
    LionelDyck <><
    https://github.com/zigi
    https://github.com/lbdyck
    ------------------------------



  • 9.  RE: Is this a setup problem or a problem with Git for z/OS

    Posted 06-10-2022 06:29

    Lionel Dyck's experience matches ours, I think that git show and git restore somehow expects to deliver the output to a terminal of some sort and inserts newlines or interprets some characters in the table as newlines instead of treating the content as binary data. 



    ------------------------------
    Morten Kroyer
    Software developer
    Rocket Forum Shared Account
    ------------------------------



  • 10.  RE: Is this a setup problem or a problem with Git for z/OS

    Posted 06-10-2022 07:23
    I did more testing with ZIGI and found that ZIGI does not care what the tags are for a file (text or binary) as it maintains that in the .gitattributes file and that is what ZIGI uses for all copies between z/OS and OMVS.  ZIGI also maintains ISPF statistics for all PDS/PDSE members that have them.

    See it at GitHub - zigi/zigi: zigi: The git interface for the rest of us!
    GitHub remove preview
    GitHub - zigi/zigi: zigi: The git interface for the rest of us!
    Make sure you have git installed on your Mainframe. In order to install git, head on over to https://www.rocketsoftware.com/ and create an account. Then download git, bash, gzip and perl and bring all those files to one directory in USS.
    View this on GitHub >


    ------------------------------
    LionelDyck <><
    https://github.com/zigi
    https://github.com/lbdyck
    ------------------------------



  • 11.  RE: Is this a setup problem or a problem with Git for z/OS

    PARTNER
    Posted 06-14-2022 00:45
    Hello and thanks to everyone who have replied. 

    I thought I'd give you all the background to this issue.  I have a tool similar to ziggy that I started writing back in 2016, yes even before Git for z/OS was announced.  I had a ssh connection to my PC and automatically copied updated files to the PC and issued git commands on the PC and copied updated files back to the z/OS. 

    That all changed in 2017 when Git for z/OS was announced and I converted the tool to use Git for z/OS.   To transfer the files from z/OS datasets to USS files, and vice versa, I've been using IBM's OGET and OPUT commands but I finally got sick of how slow the transfers took and decided to write my own in assembler. 

    I am no USS guru, having spent my career in the traditional MVS environment. So to my surprise, during the writing of my assembler program, I found that each USS file also had a file format as seen here from the BPXYFTYP macro:


    So I coded my assembler program to support these formats.  During the transfer it automatically adds/subtracts the correct delimiters based on this value. 

    But it now seems that my program is the only program that does. :-(  Even ISPF is not supporting all the file formats correctly !!!

    With Git for z/OS not preserving the file format, my ISPF tables which had a file format of binary were being transfers as text and any X'15' in the table were considered EOL characters, creating invalid tables. 

    I've now changed my program so that if the file format is "not specified" it will now look at the CCSID  and if "binary" it will transfer with no data delimiters. If not "binary" it will assume NL as the data delimiter. 

    (just in case anyone is interested my assembler program has cut the transfer time for 660 member project from 127 seconds down to 4 seconds.  I said OGET and OPUT were slow).

    @hank oerlemans

    No I did not try changing the name by adding ".bin" to the name.  As far as know USS does not use the name to identify file types.

    @Giancarlo Frix

    Your summary of the issue is correct. 

    I understand that the file format is a z/OS only attribute and it's not available on other platforms, but I'm sure it can be supported in exactly the same way as "zos-working-tree-encoding=" which  doesn't get used on other platforms.   You could have a .gitattributes entry "zos-file-format=" that could update the file format  when updating the working directory. 

    You said "git does not set or restore the file format flag, that is actually a known limitation". I haven't seen this limitation documented, its certainly not in the "Git 2.26.2 for z/OS release Notes".  
     
    You certainly are right in that minimal utilities support the file format, even ISPF doesn't fully support it. 

    @Morten Krøyer

    There is another issue with git show, which I have reported here.

    Certainly z/OS datasets, that are being treated as text files, that contain X'15' characters have problems. When copied to USS the X'15' are copied fine, but on the way back from USS to z/OS datasets the copy programs have to use X'15' as the end of line, cause any imbedded X'15' to split lines. 

    I've removed all X'15' from my panels and replaced with some other character, this allows the panels still be treated as text file allowing git-diff to work with the files. 

    ISPF tables on the other hand have to be treated as binary files, as they could have imbedded X'15'

    @Lionel Dyck

    I can understand ziggy ignoring the file format as it seems every other application is too.

    But am I understanding you correctly,  that you are ignoring the CCSID file encoding altogether and are only using the gitattributes definitions as to whether you transfer your files as binary or text ?  If the gitattribute files specifies "binary" do you update the USS tag to binary after your transfer the file. 

    Also, I can't get my head around supporting ISPF statistics with Git.  Do you update the stats based on the USS date/time ? what happens to VRM, specially when members can be updated on different branches - you could be v01.40 on one branch only to go back to say 01.25 if you checkout an old branch.   

    @All

    Again thanks for your support with this issue.  As I've said I have now coded my assembler program to honor the file formats if supplied, and if not specified will look for CCSID binary encoding to support binary files. 




     

     ​​​​​

    ------------------------------
    Gary Freestone
    Systems Programmer
    Kyndryl Inc
    Mt Helen VIC AU
    ------------------------------



  • 12.  RE: Is this a setup problem or a problem with Git for z/OS

    Posted 06-14-2022 02:31
    Actually ZIGI uses the code page defined in the gitattributes as well.

    ------------------------------
    LionelDyck <><
    https://github.com/zigi
    https://github.com/lbdyck
    ------------------------------



  • 13.  RE: Is this a setup problem or a problem with Git for z/OS

    PARTNER
    Posted 06-14-2022 02:36
    Thanks Lionel

    ------------------------------
    Gary Freestone
    Systems Programmer
    Kyndryl Inc
    Mt Helen VIC AU
    ------------------------------



  • 14.  RE: Is this a setup problem or a problem with Git for z/OS

    Posted 06-14-2022 18:45
    git-newbie here....I only mentioned the .bin as while USS does not care I've seen "Unix" tools be sensitive to this.....and I couldn't find anything significant about a binary file transfer that git does for us other than that. I've pulled the files back and they operate correctly.
    The last really irritating binary problem I had was with RTC-Ant builds and I had to ensure the binary files were all properly fixed 80 and treated that way all the way. i.e. ensure no VB and no Cr Lf Nl issues as a result and blowing away any kind of CCSID or implied format issues. Too long ago to be more specific.

    ------------------------------
    hank oerlemans
    Lead Software Engineer
    HCL America Inc.
    North Sydney NSW AU
    ------------------------------