Git shows modified files after git clone

Hi,

in our comnpany we want to move our deployment pipline from jenkins agent to USS and so we are discovering the rocket git port for z/OS.
But we have some problems with the zos-working-tree-encoding translation. After git clone, the entire working tree ist marked as modified.

The files in our git repo are in ANSI (WINDOWS 1252) codepage.
We are using on our Mainframe the german EBCDIC codepage 1141.

git --version
git version 2.26.2-78

z/OS 2.3

.gitattributes

*.CBL zos-working-tree-encoding=IBM-1141 git-encoding=IBM-1252
*.CPY zos-working-tree-encoding=IBM-1141 git-encoding=IBM-1252
*.REX zos-working-tree-encoding=IBM-1141 git-encoding=IBM-1252
*.BMS zos-working-tree-encoding=IBM-1141 git-encoding=IBM-1252​

09:01:27 uuuuuu@ssssssss~/gitrepo $ git clone https://github.com/xxxxxx

Cloning into 'mainframe'...
remote: Enumerating objects: 36495, done.
remote: Counting objects: 100% (4369/4369), done.
remote: Compressing objects: 100% (2050/2050), done.
remote: Total 36495 (delta 2564), reused 3846 (delta 2110), pack-reused 32126
Receiving objects: 100% (36495/36495), 81.42 MiB | 3.39 MiB/s, done.
Resolving deltas: 100% (25314/25314), done.
Updating files: 100% (22956/22956), done.
09:02:13 uuuuuu@ssssssss~/gitrepo $ cd mainframe/
09:02:19 uuuuuu@ssssssss~/gitrepo/mainframe $ git status
Refresh index: 100% (22956/22956), done.
On branch produktion
Your branch is up to date with 'origin/produktion'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   xxx/cobol/UPVB0020.CBL
        modified:   xxx/cobol/UPVB0021.CBL
        modified:   xxx/cobol/UPVB0022.CBL
        modified:   xxx/cobol/UPVB0023.CBL
        modified:   xxx/cobol/UPVB0024.CBL
        modified:   xxx/cobol/UPVB0025.CBL
        modified:   xxx/cobol/UPVB0026.CBL
....
many other files
....

ls -laT for those files:

10:04:48 uuuuuuuu@sssssss~/gitrepo/mainframe/xxx/cobol $ ls -laT UPVB002*
t IBM-1141    T=on  -rw-rw-r--   1 uuuuuuu  GGGGG      46467 Mar 24 09:12 UPVB0020.CBL
t IBM-1141    T=on  -rw-rw-r--   1 uuuuuuu  GGGGG      24690 Mar 24 09:12 UPVB0021.CBL
t IBM-1141    T=on  -rw-rw-r--   1 uuuuuuu  GGGGG      20514 Mar 24 09:12 UPVB0022.CBL
t IBM-1141    T=on  -rw-rw-r--   1 uuuuuuu  GGGGG      18463 Mar 24 09:12 UPVB0023.CBL
t IBM-1141    T=on  -rw-rw-r--   1 uuuuuuu  GGGGG      15993 Mar 24 09:12 UPVB0024.CBL

does anybody have an idea why those sources are marked as modified?

kind regards, david

------------------------------
David Dylong
Databaseadmnistrator
Deutsche Vermögensberatung Aktiengesellschaft · DVAG
Frankfurt am Main DE
------------------------------

Hi David,

Are you moving from an old version of Git to 2.26.2?

The old versions of Git have ISO8859-1 as a default git-encoding, but Git 2.26.2 keeps all files in index in UTF-8 and ignores git-encoding attribute.
In this case, the files can be showed as modified because Git exepcts that they are saved in UTF-8 insted of the current value IBM-1252.
You can renolmalize your repository and update the index:
git add --all --renormalize
And you can also remove git-encoding from .gitattributes (keep zos-working-tree-encoding only).

The quote from release notes:
If you have been working with previous releases Git for z/OS, you must push recent changes to the remote repository with the old version of Git and clone the repository with Git for z/OS 2.26.2.

Note: Once you change a repository with Git for z/OS 2.26.2, this repository won't be compatible with previous versions of Git for z/OS. It is vital that entire team will move to the new version of Git.

When you clone the repository with Git for z/OS 2.26.2, you need to run git status command. You might see that Git marks some files with the attribute as modified, even if you didn't modify them.

Usually it happens if you work in the repository, which had files encoded in ISO8859-1 in the index and the files have bytes which are converted to bytes greater than x7F in ISO88859-1. This is an expected behavior. Git for z/OS 2.26.2 does real-time conversion to UTF-8 to check if there is a difference between a file in the repository index and working directory. The files used to be encoded in ISO8859-1 in the repository index. If the ISO8859-1 representation doesn't match that in UTF-8, the file will be marked as modified. We recommend to update the files in the index (commands: git add <filename>; git commit).

Also it can happen if the file in the index has a character which doesn't have a corresponding character in the encoding defined in (zos-)working-tree-encoding attribute. Note that iconv on z/OS replaces such characters with a substitution character without any warning or error message. You need to exclude such characters from the file or change (zos-)working-tree-encoding attribute.

Please, read Git 2.26.2 release notes for complete information:
https://community.rocketsoftware.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=2bf3f34f-fbee-467a-ab95-4aa35db76519&forceDialog=0

Thanks,
Sergei

------------------------------
Sergei Rezepin
Rocket Software
------------------------------

Hi David,

Are you moving from an old version of Git to 2.26.2?

The old versions of Git have ISO8859-1 as a default git-encoding, but Git 2.26.2 keeps all files in index in UTF-8 and ignores git-encoding attribute.
In this case, the files can be showed as modified because Git exepcts that they are saved in UTF-8 insted of the current value IBM-1252.
You can renolmalize your repository and update the index:
git add --all --renormalize
And you can also remove git-encoding from .gitattributes (keep zos-working-tree-encoding only).

The quote from release notes:
If you have been working with previous releases Git for z/OS, you must push recent changes to the remote repository with the old version of Git and clone the repository with Git for z/OS 2.26.2.

Note: Once you change a repository with Git for z/OS 2.26.2, this repository won't be compatible with previous versions of Git for z/OS. It is vital that entire team will move to the new version of Git.

When you clone the repository with Git for z/OS 2.26.2, you need to run git status command. You might see that Git marks some files with the attribute as modified, even if you didn't modify them.

Usually it happens if you work in the repository, which had files encoded in ISO8859-1 in the index and the files have bytes which are converted to bytes greater than x7F in ISO88859-1. This is an expected behavior. Git for z/OS 2.26.2 does real-time conversion to UTF-8 to check if there is a difference between a file in the repository index and working directory. The files used to be encoded in ISO8859-1 in the repository index. If the ISO8859-1 representation doesn't match that in UTF-8, the file will be marked as modified. We recommend to update the files in the index (commands: git add <filename>; git commit).

Also it can happen if the file in the index has a character which doesn't have a corresponding character in the encoding defined in (zos-)working-tree-encoding attribute. Note that iconv on z/OS replaces such characters with a substitution character without any warning or error message. You need to exclude such characters from the file or change (zos-)working-tree-encoding attribute.

Please, read Git 2.26.2 release notes for complete information:
https://community.rocketsoftware.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=2bf3f34f-fbee-467a-ab95-4aa35db76519&forceDialog=0

Thanks,
Sergei

------------------------------
Sergei Rezepin
Rocket Software
------------------------------

One of the issues I struggle with is when I want to work with a repo that I don't have the authority to make global to - for example I want to fix bugs but the owner of the group isn't keen to add git attributes just for z/OS. Has there been any thought to support some sort of environment variable so that we could request the old tagging be done so that we could tag everything in ISO8859-1? 

I understand the issues of tagging for other repos like an hlasm repo that might be more naturally encoded in EBCDIC but it would nice to be able to have multi platform repos 'just work' 

------------------------------
mike fulton
VP/CIO
IBM Canada Ltd
brackendale BC CA
------------------------------

Hi David,

Are you moving from an old version of Git to 2.26.2?

The old versions of Git have ISO8859-1 as a default git-encoding, but Git 2.26.2 keeps all files in index in UTF-8 and ignores git-encoding attribute.
In this case, the files can be showed as modified because Git exepcts that they are saved in UTF-8 insted of the current value IBM-1252.
You can renolmalize your repository and update the index:
git add --all --renormalize
And you can also remove git-encoding from .gitattributes (keep zos-working-tree-encoding only).

The quote from release notes:
If you have been working with previous releases Git for z/OS, you must push recent changes to the remote repository with the old version of Git and clone the repository with Git for z/OS 2.26.2.

Note: Once you change a repository with Git for z/OS 2.26.2, this repository won't be compatible with previous versions of Git for z/OS. It is vital that entire team will move to the new version of Git.

When you clone the repository with Git for z/OS 2.26.2, you need to run git status command. You might see that Git marks some files with the attribute as modified, even if you didn't modify them.

Usually it happens if you work in the repository, which had files encoded in ISO8859-1 in the index and the files have bytes which are converted to bytes greater than x7F in ISO88859-1. This is an expected behavior. Git for z/OS 2.26.2 does real-time conversion to UTF-8 to check if there is a difference between a file in the repository index and working directory. The files used to be encoded in ISO8859-1 in the repository index. If the ISO8859-1 representation doesn't match that in UTF-8, the file will be marked as modified. We recommend to update the files in the index (commands: git add <filename>; git commit).

Also it can happen if the file in the index has a character which doesn't have a corresponding character in the encoding defined in (zos-)working-tree-encoding attribute. Note that iconv on z/OS replaces such characters with a substitution character without any warning or error message. You need to exclude such characters from the file or change (zos-)working-tree-encoding attribute.

Please, read Git 2.26.2 release notes for complete information:
https://community.rocketsoftware.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=2bf3f34f-fbee-467a-ab95-4aa35db76519&forceDialog=0

Thanks,
Sergei

------------------------------
Sergei Rezepin
Rocket Software
------------------------------

Hi Sergei,

wo didn't move from an old version of Git to 2.26.2, we installed it for the first time.

But in the meantime we figured out what the root of the problem was.
Our repo is in ANSI and its the same problme as with ISO8859-1 representation.

If we move our repo to UTF-8 everything is fine.


Thanks,

------------------------------
David Dylong
Databaseadmnistrator
Deutsche Vermögensberatung Aktiengesellschaft · DVAG
Frankfurt am Main DE
------------------------------