Git for z/os - Chinese Character code page 935 translation problem

hi dear Rocket supporter,

We are testing Git for z/os to try to migrate our mainframe source code from ClearCase to Git. As we are a banking in China, so we need to make sure Git for z/os support Chinese character. We tried the below configuration:


Then, I update the txt file on git server,  add a new line with "中国银行" which are four Chinese characters, as below

Then, I checkout the update on z/os uss, I got the wrong code page translation, the ending '0f' is lost.

fyi, you can see "中国" is correctly displayed, because I added the "中国" from the z/os uss end, then push to git server. But if I do the update from the git server end and then checkout the update from the z/os uss end, the error occurred. 

Please could you help to take a look at this problem? Thanks a lot!

Xiaohang Wang
IBM z software presales

------------------------------
Doria Wang
client support
IBM China Company Limited
BEI JING China
------------------------------

Hi Doria,

Thank you for reporting the issue. I've tried to reproduce it, but it wasn't successfully yet.
Could you say which version and build of git you are using and send the output of 'env | sort' command if possible?

Thanks,
Sergey

------------------------------
Sergey Rezepin
Rocket Software
------------------------------

Hi Doria,

Thank you for reporting the issue. I've tried to reproduce it, but it wasn't successfully yet.
Could you say which version and build of git you are using and send the output of 'env | sort' command if possible?

Thanks,
Sergey

------------------------------
Sergey Rezepin
Rocket Software
------------------------------

hi Sergey,

Thanks for your prompt response, please check the below info, thanks so much!
(gitenv) -bash-4.3¥ git version
git version 2.26.2-78

(gitenv) -bash-4.3¥ env|sort
ANT_HOME=/u/rocket/miniconda/envs/gitenv
ANT_HOME_CONDA_BACKUP=
ASCII_TERMINFO=/u/rocket/miniconda/envs/gitenv/share/terminfo
COLUMNS=160
CONDA_DEFAULT_ENV=gitenv
CONDA_EXE=/u/rocket/miniconda/bin/conda
CONDA_PREFIX=/u/rocket/miniconda/envs/gitenv
CONDA_PROMPT_MODIFIER=(gitenv)
CONDA_PYTHON_EXE=/u/rocket/miniconda/bin/python
CONDA_SHLVL=1
CURL_CA_BUNDLE=/u/rocket/miniconda/envs/gitenv/etc/ssl/cacert.pem
FFI_LIB=/u/rocket/miniconda/envs/gitenv/lib/ffi
GIT_EXEC_PATH=/u/rocket/miniconda/envs/gitenv/libexec/git-core
GIT_SHELL=/u/rocket/miniconda/envs/gitenv/bin/bash
GIT_TEMPLATE_DIR=/u/rocket/miniconda/envs/gitenv/share/git-core/templates
HOME=/u/tmp/tso03
JAVA_HOME=/usr/lpp/java/J8.0_64
LANG=C
LIBPATH=/u/rocket/miniconda/envs/gitenv/lib:/u/rocket/miniconda/lib:/lib:/usr/lib:.:/u/rocket/miniconda/envs/gitenv/lib/perl5/5.24.4/os390/CORE
LINES=58
LOGNAME=TSO03
MAIL=/usr/mail/TSO03
MANPATH=/u/rocket/miniconda/envs/gitenv/share/man:/u/rocket/miniconda/envs/gitenv/man:/usr/man/%L
NLSPATH=/usr/lib/nls/msg/%L/%N
OLDPWD=/u/tmp/tso03
OPENSSL_CONF=/u/rocket/miniconda/envs/gitenv/ssl/openssl.cnf
PATH=/u/rocket/miniconda/envs/gitenv/bin:/usr/lpp/java/J8.0_64/bin:/bin
PERL5LIB=/u/rocket/miniconda/envs/gitenv/lib/perl5:/u/rocket/miniconda/envs/gitenv/lib/perl5/site_perl
PWD=/u/tmp/tso03/test001
PYTHON_DVERSION=python3.7
PYTHON_VERSION=python37
PYV=py37
SHELL=/u/rocket/miniconda/bin/bash
SHLVL=1
SSL_CERT_DIR=/u/rocket/miniconda/envs/gitenv/ssl/certs
SSL_CERT_FILE=/u/rocket/miniconda/envs/gitenv/ssl/cert.pem
STEPLIB=none
TERM=dumb
TZ=TAIST-8
_=/bin/env
_BPXK_AUTOCVT=ON
_BPX_TERMPATH=OMVS
_CEE_RUNOPTS=FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)
_TAG_REDIR_ERR=TXT
_TAG_REDIR_IN=TXT
_TAG_REDIR_OUT=TXT


------------------------------
Doria Wang
client support
IBM China Company Limited
BEI JING China
------------------------------

Hi Doria,

Thank you for reporting the issue. I've tried to reproduce it, but it wasn't successfully yet.
Could you say which version and build of git you are using and send the output of 'env | sort' command if possible?

Thanks,
Sergey

------------------------------
Sergey Rezepin
Rocket Software
------------------------------

hi Sergey,

The problem looks same as this apar https://www.ibm.com/support/pages/apar/IV07477, which for java.
You can reference this apar to check whether we have same problem in python code. Thanks so much!

------------------------------
Doria Wang
client support
IBM China Company Limited
BEI JING China
------------------------------

hi Sergey,

The problem looks same as this apar https://www.ibm.com/support/pages/apar/IV07477, which for java.
You can reference this apar to check whether we have same problem in python code. Thanks so much!

------------------------------
Doria Wang
client support
IBM China Company Limited
BEI JING China
------------------------------

Hi Doria,

Thank you for the detail information, I reproduced the behavior with 'git merge' locally, without using a git server. Git truncates the last SHIFT_IN (\\x0f) character when merging for adding the new line with the double byte characters.

It looks, the closing SHIFT_IN in the end of file is not necessary required for the conversion from IBM-935 to UTF-8 back properly, because the file converts well, but the editor mentioned in your post can't read the last sentence correctly, because it adds single byte \\x40 characters after that. In this case, SHIFT_IN is required to split double bytes and single bytes characters for correct reading. Actually, the original file have no \\x40 characters at the end of file.

Anyway, we need to research this issue. Now, I can only recommend to add a single byte character at the end of the file as a workaround. For example, add 'new line' character at the end of file after adding a last line with double byte characters. It will help to avoid the issue.

Thanks,
Sergey

------------------------------
Sergey Rezepin
Rocket Software
------------------------------