Open-source Languages & Tools for z/OS

 View Only
  • 1.  Git 2.26.2-84 - Presence of an extra x0D character at the end of the line on files transcoded in EBCDIC

    Posted 05-10-2022 06:20
    Hi,

    By cloning by Git for z/OS, in SSH protocol, a repository stored in our Bitbucket manager, and created on Windows, I notice the presence of an x0D character at the end of each line of the files encoded in EBCDIC.
    Files encoded in ASCII, or more precisely files that are not subject to codepage conversion, do not have this x0D character at the end of the line.

    I had to add an explicit "eol=lf" option in the .gitattributes file on the "*" pattern before cloning on USS to work around this problem.

    This is not satisfactory, because we use our Git repositories on various platforms, mainly Windows, and also z/OS-USS, and we don't want to force the handling of line endings in the .gitattributes file.
    We want each platform to be free to use whatever line endings it wants.

    All of our Git installations, Windows or USS, use the core.autocrlf=true configuration. This has the consequence of storing text files in internal format with an CRLF end-of-line sequence.
    When fetching to the working directory, the end-of-line sequence must be adapted to the platform: CRLF for Windows, LF for Unix/Linux, nothing (visible) for USS.

    The fact that only transcoded files (ASCII to EBCDIC) are affected by this problem leads me to suspect that it is a Git for z/OS specific anomaly that occurs at the time of this transcoding.

    Git version: Git 2.26.2-84

    ------------------------------
    Fallai Denis
    M
    Self Registered
    L'ISLE-ADAM FR
    ------------------------------


  • 2.  RE: Git 2.26.2-84 - Presence of an extra x0D character at the end of the line on files transcoded in EBCDIC

    PARTNER
    Posted 05-12-2022 08:56
    I've only been into git since about 2017 when Rocket brought it to z/OS.  It's only in the last year or so that I've ventured out using VScode to edit essentially z/OS PDS members. I use VScode on windows, my colleague uses it running on MAC.   In all this time I've never had a problem with EOL characters in any of the environments.

    Our PDS members start out FB80  with no EOL characters. They get transferred to USS with a NL X'15' character being added to the end of each line.  Rocket encodes the file in UTF-8 within Git, which gets pushed to the GitHub.   If I clone the file to windows and dump the file in hex I see the EOL characters has been changed to CRLF as expected in windows.  My colleague has LF as the end of line character.  

    Interesting he also can run VScode on Windows under VMWARE on this MAC. When he looks at these files under windows they have the LF characters as the EOL character. This is understandable given the file was created on MAC. But he is still able to edit and save on his windows with just the LF as the EOL character. VScode even when running on windows handles just LF as the end of line character.

    As I said, I am relatively new to Git, especially using it in multiple different operating system, by so far we have no had issues running the default configuration everywhere.

    Regards,
    Gary

    ------------------------------
    Gary Freestone
    Systems Programmer
    Kyndryl Inc
    Mt Helen VIC AU
    ------------------------------



  • 3.  RE: Git 2.26.2-84 - Presence of an extra x0D character at the end of the line on files transcoded in EBCDIC

    Posted 05-12-2022 11:19
    Hi Gary,

    I also use VS Code, with Zowe CLI + z/OSMF, to work with PDS files, and I don't have any problem either.

    I didn't quite understand if you were using Git for z/OS to clone to USS....
    If so, could you communicate the .gitattributes file used in your Git repositories?

    ------------------------------
    Fallai Denis
    M
    Self Registered
    L'ISLE-ADAM FR
    ------------------------------



  • 4.  RE: Git 2.26.2-84 - Presence of an extra x0D character at the end of the line on files transcoded in EBCDIC

    Posted 05-29-2022 04:26

    Solved ! Presence of "core.autocrlf=true" in /u/<my_user>/.gitconfig file

    Information about Git config on z/OS:

    >git config --system --list
    error: cannot run less: EDC5129I No such file or directory.
    http.sslverify=true
    http.sslcainfo=/service/OADV100/bin/ca-bundle.crt

    >git config --global --list
    error: cannot run less: EDC5129I No such file or directory.
    user.name=<my_user>
    user.email=<my_email>
    core.autocrlf=true          <-- problem! ok after removing this option

    >git config --local --list
    error: cannot run less: EDC5129I No such file or directory.
    core.repositoryformatversion=0
    core.filemode=true
    core.bare=false
    core.logallrefupdates=true
    remote.origin.url=ssh://<my_server>/<path_to_my_repo>
    remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
    branch.master.remote=origin
    branch.master.merge=refs/heads/master

    Question: should we force core.autocrlf=input for Git for z/OS, either at the system level, or at the global (user) level, to avoid any end-of-line management problem?





    ------------------------------
    Fallai Denis
    M
    Self Registered
    L'ISLE-ADAM FR
    ------------------------------



  • 5.  RE: Git 2.26.2-84 - Presence of an extra x0D character at the end of the line on files transcoded in EBCDIC

    PARTNER
    Posted 06-05-2022 20:02
    Edited by Gary Freestone 06-09-2022 19:22
    Fallai,

    Sorry I didn't reply to your post earlier but I've been on vacation. 

    I'm glad you have solved your problem, this whole CRLF/LF/CR stuff  is confusing  because z/OS doesn't, by default,  use any of those characters for the EOL but NL (New Line X'15') instead.   

    By removing core.autocrlf=true you have reverted back to the default, which on z/OS is NL.   

    From my understanding, the EOL character used by Git internally is LF.   It is when a checkout is performed Git looks at core.autocrlf to decide what it will add as the EOL character in the working directory.    With autocrlf=true, Git would normally put a CRLF as the EOL character but, on z/OS having autocrlf=true results in CRNL as the EOL characters and not CRLF.  

    So in your case Git adds CRNL, but z/OS only recognizes the NL so you end up with the CR characters at the end of each line. 

    Note that you could have continued to use autocrlf=true if your USS files had a file format of CRNL.   Each file in USS can have 1 of 9 file formats, 0=Default, 1=binary, 2=NL, 3=CR, 4=LF, 5=CRLF, 6=LFCR, 7=CRNL, 8=LRECL.   

    I changed all my files to type 7 and changed autocrlf=true and seemed to work fine with Git IBM's OGET and OPUT commands did not recognize the CRNL even when the file format clearly said it was CRNL.

    I also tried core.autocrlf=input and core.autocrlf=false, which with Git for z/OS at least created files with a EOL of NL.

    If you delete core.autocrlf from the config then the default is core.autocrlf=false

    Personally I don't think core.autocrlf=input should be forced.  By default Git for z/OS does not have autocrlf= coded so everything works as designed.   If you didn't add core.autocrlf=trueto your config I have no idea how you got it.  




    ------------------------------
    Gary Freestone
    Systems Programmer
    Kyndryl Inc
    Mt Helen VIC AU
    ------------------------------