git anomaly on test lpar

Jorn Thyssen
Rocketeer
22 replies
Forum|Forum|4 years ago
March 2, 2021

I'm having an issue on a test lpar with git for z/OS – both version 2.14 and the latest 2.26 ports.

I've looked at this as have several others without success.

When I attempt a git clone I get this error:

fatal: protocol error: bad line length character: ---Á

When I do a ssh git@github.com git-receive-pack lbdyck/qtab.git I get this:

--ÄÁ---Ä--Ã----Ã-À--ÄÂÂ--Á-/ÄÁ-------À----/--ÊÁÃËÇÁ/ÀË_/ËÈÁÊÊÁø?ÊÈ-ËÈ/ÈÍË-ÊÁø?ÊÈ-ËÈ/ÈÍË-Î--ÀÁ%ÁÈÁ-ÊÁÃË-ËÑÀÁ-Â/>À---,-ÉÍÑÁÈ-/È?_ÑÄ-?Ã

Ë-ÀÁ%È/-øÍËÇ-?øÈÑ?>Ë-?Â¦ÁÄÈ-Ã?Ê_/È-ËÇ/--/ÅÁ>È-ÅÑÈÅÑÈÇÍÂ-Å-ÃÃ-Ä/À----------

If I do the same ssh git@github.com git-receive-pack lbdyck/qtab.git on one of my other systems it looks like this:

00ce342c27f0363f6d24cbb68e8ace7454218d4828a2 refs/heads/masterreport-status report-status-v2 delete-refs side-band-64k quiet atomic

ofs-delta push-options object-format=sha1 agent=git/github-g2ff1cad44179

I have these in the environment:

_BPXK_AUTOCVT=ON
_CEE_RUNOPTS=FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)
_TAG_REDIR_IN=txt
_TAG_REDIR_OUT=txt
_TAG_REDIR_ERR=txt

We've looked at the TCP/IP setup, the SSH setup, and the Git setup with no joy. We've also confirmed that the ssh public key has been properly added to github.

Can anyone provide any suggestions that will help resolve this?

------------------------------
Lionel Dyck
lbdyck
------------------------------

Hi Lionel,

Disclaimer: I am just a regular git for z/OS user.

I have the same environment variables set as you and does not have any issues (I am using gitlab not github, though).

Do you have the same version of ssh on both LPARs? (I assume "my other systems" refer to other z/OS systems?)

I've previously used
export GIT_SSH_COMMAND="ssh -v -v"
to debug ssh related issues.

------------------------------
Joern Thyssen
Rocket Software
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 2, 2021

Hi Lionel,

Disclaimer: I am just a regular git for z/OS user.

I have the same environment variables set as you and does not have any issues (I am using gitlab not github, though).

Do you have the same version of ssh on both LPARs? (I assume "my other systems" refer to other z/OS systems?)

I've previously used
export GIT_SSH_COMMAND="ssh -v -v"
to debug ssh related issues.

------------------------------
Joern Thyssen
Rocket Software
------------------------------

I have the same SSH on all systems.

I tried what you suggested and I get this on the bad lpar - strange.

debug1: identity file /u/lbdyck/.ssh/id_rsa type 0
debug1: key_load_public: EDC5129I No such file or directory. (errno2=0x05620062)
debug1: identity file /u/lbdyck/.ssh/id_rsa-cert type -1
debug1: key_load_public: EDC5129I No such file or directory. (errno2=0x05620062)
debug1: identity file /u/lbdyck/.ssh/id_dsa type -1
debug1: key_load_public: EDC5129I No such file or directory. (errno2=0x05620062)

------------------------------
Lionel Dyck
lbdyck
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 2, 2021

I have the same SSH on all systems.

I tried what you suggested and I get this on the bad lpar - strange.

debug1: identity file /u/lbdyck/.ssh/id_rsa type 0
debug1: key_load_public: EDC5129I No such file or directory. (errno2=0x05620062)
debug1: identity file /u/lbdyck/.ssh/id_rsa-cert type -1
debug1: key_load_public: EDC5129I No such file or directory. (errno2=0x05620062)
debug1: identity file /u/lbdyck/.ssh/id_dsa type -1
debug1: key_load_public: EDC5129I No such file or directory. (errno2=0x05620062)

------------------------------
Lionel Dyck
lbdyck
------------------------------

Then later I see

debug2: key: /u/lbdyck/.ssh/id_rsa (B17EAA8)
debug2: key: /u/lbdyck/.ssh/id_dsa (0)
debug2: key: /u/lbdyck/.ssh/id_ecdsa (0)
debug2: key: /u/lbdyck/.ssh/id_ed25519 (0)

strange.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 2, 2021

Then later I see

debug2: key: /u/lbdyck/.ssh/id_rsa (B17EAA8)
debug2: key: /u/lbdyck/.ssh/id_dsa (0)
debug2: key: /u/lbdyck/.ssh/id_ecdsa (0)
debug2: key: /u/lbdyck/.ssh/id_ed25519 (0)

strange.

------------------------------
Lionel Dyck
lbdyck
------------------------------

And further down this:

debug1: Authentication succeeded (publickey).
Authenticated to github.com ([140.82.113.4]:22).

Looks good - no anomalies here.

------------------------------
Lionel Dyck
lbdyck
------------------------------

David Crayford
Rocketeer
11 replies
Forum|Forum|4 years ago
March 3, 2021

And further down this:

debug1: Authentication succeeded (publickey).
Authenticated to github.com ([140.82.113.4]:22).

Looks good - no anomalies here.

------------------------------
Lionel Dyck
lbdyck
------------------------------

We're stumped by this! Is the problem LPAR in a sysplex with the LPARs that work? Can you dump the BPXPRMnn parameter member?

------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 4, 2021

We're stumped by this! Is the problem LPAR in a sysplex with the LPARs that work? Can you dump the BPXPRMnn parameter member?

------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------

How about a D OMVS,O

OMVS 000F ACTIVE OMVS=(00,FS,F2,GT)
CURRENT UNIX CONFIGURATION SETTINGS:
MAXPROCSYS = 900 MAXPROCUSER = 25
MAXFILEPROC = 64000 MAXFILESIZE = NOLIMIT
MAXCPUTIME = 1000 MAXUIDS = 200
MAXPTYS = 800 MAXIOBUFUSER = 2048
MAXMMAPAREA = 40960 MAXASSIZE = 209715200
MAXTHREADS = 200 MAXTHREADTASKS = 1000
MAXCORESIZE = 4194304 MAXSHAREPAGES = 131072
IPCMSGQBYTES = 2147483647 IPCMSGQMNUM = 10000
IPCMSGNIDS = 500 IPCSEMNIDS = 500
IPCSEMNOPS = 25 IPCSEMNSEMS = 1000
IPCSHMMPAGES = 25600 IPCSHMNIDS = 500
IPCSHMNSEGS = 500 IPCSHMSPAGES = 262144
SUPERUSER = BPXROOT FORKCOPY = COPY
STEPLIBLIST =
USERIDALIASTABLE=
PRIORITYPG VALUES: NONE
PRIORITYGOAL VALUES: NONE
MAXQUEUEDSIGS = 1000 SHRLIBRGNSIZE = 67108864
SHRLIBMAXPAGES = 4096 VERSION = / ,N
SYSCALL COUNTS = NO TTYGROUP = TTY
SYSPLEX = NO BRLM SERVER = N/A
LIMMSG = NONE AUTOCVT = OFF
RESOLVER PROC = RESOLVER LOSTMSG = ON
AUTHPGMLIST = NONE
SWA = BELOW NONEMPTYMOUNTPT = NOWARN
SERV_LINKLIB =
SERV_LPALIB =
MAXUSERMOUNTSYS = 0 MAXUSERMOUNTUSER= 0
MAXPIPEUSER = 8730 PWT = ENV
KERNELSTACKS = ABOVE UMASK = NONE
SC_EXITTABLE =

------------------------------
Lionel Dyck
lbdyck
------------------------------

+1

Vladimir Ein
Rocketeer
110 replies
Forum|Forum|4 years ago
March 5, 2021

How about a D OMVS,O

OMVS 000F ACTIVE OMVS=(00,FS,F2,GT)
CURRENT UNIX CONFIGURATION SETTINGS:
MAXPROCSYS = 900 MAXPROCUSER = 25
MAXFILEPROC = 64000 MAXFILESIZE = NOLIMIT
MAXCPUTIME = 1000 MAXUIDS = 200
MAXPTYS = 800 MAXIOBUFUSER = 2048
MAXMMAPAREA = 40960 MAXASSIZE = 209715200
MAXTHREADS = 200 MAXTHREADTASKS = 1000
MAXCORESIZE = 4194304 MAXSHAREPAGES = 131072
IPCMSGQBYTES = 2147483647 IPCMSGQMNUM = 10000
IPCMSGNIDS = 500 IPCSEMNIDS = 500
IPCSEMNOPS = 25 IPCSEMNSEMS = 1000
IPCSHMMPAGES = 25600 IPCSHMNIDS = 500
IPCSHMNSEGS = 500 IPCSHMSPAGES = 262144
SUPERUSER = BPXROOT FORKCOPY = COPY
STEPLIBLIST =
USERIDALIASTABLE=
PRIORITYPG VALUES: NONE
PRIORITYGOAL VALUES: NONE
MAXQUEUEDSIGS = 1000 SHRLIBRGNSIZE = 67108864
SHRLIBMAXPAGES = 4096 VERSION = / ,N
SYSCALL COUNTS = NO TTYGROUP = TTY
SYSPLEX = NO BRLM SERVER = N/A
LIMMSG = NONE AUTOCVT = OFF
RESOLVER PROC = RESOLVER LOSTMSG = ON
AUTHPGMLIST = NONE
SWA = BELOW NONEMPTYMOUNTPT = NOWARN
SERV_LINKLIB =
SERV_LPALIB =
MAXUSERMOUNTSYS = 0 MAXUSERMOUNTUSER= 0
MAXPIPEUSER = 8730 PWT = ENV
KERNELSTACKS = ABOVE UMASK = NONE
SC_EXITTABLE =

------------------------------
Lionel Dyck
lbdyck
------------------------------

Hi Lionel,

This looks like a problem with SSH configuration, probably unrelated to Git altogether. In your command 'ssh git@github.com git-receive-pack lbdyck/qtab.git', z/OS SSH client (/bin/ssh) runs the command git-receive-pack remotely on GitHub (that is, GitHub's own version of git-receive-pack), and for some reason interprets the output in a wrong encoding.

Can you please check a couple more things on the 'bad' LPAR?

1. Try 'git clone' with a different server, not github.com

2. If you have a Linux machine, try to SSH to it from the mainframe. On the bad LPAR, issue the following command:

ssh linux-user@your-linux-machine-name env

This should run the 'env' command on Linux; even though Linux produces ASCII output, ssh should display it correctly in the terminal. Unfortunately this is not going to work in TSO OMVS; ssh cannot input passwords from 3270 terminals. You'll need to connect to mainframe via Putty or a similar terminal.

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Software
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 5, 2021

Hi Lionel,

This looks like a problem with SSH configuration, probably unrelated to Git altogether. In your command 'ssh git@github.com git-receive-pack lbdyck/qtab.git', z/OS SSH client (/bin/ssh) runs the command git-receive-pack remotely on GitHub (that is, GitHub's own version of git-receive-pack), and for some reason interprets the output in a wrong encoding.

Can you please check a couple more things on the 'bad' LPAR?

1. Try 'git clone' with a different server, not github.com

2. If you have a Linux machine, try to SSH to it from the mainframe. On the bad LPAR, issue the following command:

ssh linux-user@your-linux-machine-name env

This should run the 'env' command on Linux; even though Linux produces ASCII output, ssh should display it correctly in the terminal. Unfortunately this is not going to work in TSO OMVS; ssh cannot input passwords from 3270 terminals. You'll need to connect to mainframe via Putty or a similar terminal.

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Software
------------------------------

Unfortunately this LPAR in question is behind a firewall and SSH is not allowed from outside :(

I will try to setup a different git server and try over the weekend. I have no linux machine that I have access to from this LPAR.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 5, 2021

Hi Lionel,

This looks like a problem with SSH configuration, probably unrelated to Git altogether. In your command 'ssh git@github.com git-receive-pack lbdyck/qtab.git', z/OS SSH client (/bin/ssh) runs the command git-receive-pack remotely on GitHub (that is, GitHub's own version of git-receive-pack), and for some reason interprets the output in a wrong encoding.

Can you please check a couple more things on the 'bad' LPAR?

1. Try 'git clone' with a different server, not github.com

2. If you have a Linux machine, try to SSH to it from the mainframe. On the bad LPAR, issue the following command:

ssh linux-user@your-linux-machine-name env

This should run the 'env' command on Linux; even though Linux produces ASCII output, ssh should display it correctly in the terminal. Unfortunately this is not going to work in TSO OMVS; ssh cannot input passwords from 3270 terminals. You'll need to connect to mainframe via Putty or a similar terminal.

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Software
------------------------------

I was able to ssh from the bad lpar to a different z/OS lpar using the ssh userid@xxx.xxx.xxx.xx env

and the results came back as ascii.

I saved the results in a file and looked at it as both ebcdic and as ascii - and the data was definitely ascii.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 7, 2021

Hi Lionel,

This looks like a problem with SSH configuration, probably unrelated to Git altogether. In your command 'ssh git@github.com git-receive-pack lbdyck/qtab.git', z/OS SSH client (/bin/ssh) runs the command git-receive-pack remotely on GitHub (that is, GitHub's own version of git-receive-pack), and for some reason interprets the output in a wrong encoding.

Can you please check a couple more things on the 'bad' LPAR?

1. Try 'git clone' with a different server, not github.com

2. If you have a Linux machine, try to SSH to it from the mainframe. On the bad LPAR, issue the following command:

ssh linux-user@your-linux-machine-name env

This should run the 'env' command on Linux; even though Linux produces ASCII output, ssh should display it correctly in the terminal. Unfortunately this is not going to work in TSO OMVS; ssh cannot input passwords from 3270 terminals. You'll need to connect to mainframe via Putty or a similar terminal.

Regards,
Vladimir

------------------------------
Vladimir Ein
Rocket Software
------------------------------

Found the issue - on this system (z/OS 2.4) the zos_ssh_config had

ChannelConvert shell

It should have had

ChannelConvert shell,exec

Making this change and then doing this solved the problem

kill -s HUP $(cat /var/run/sshd.pid)

------------------------------
Lionel Dyck
lbdyck
------------------------------

David Crayford
Rocketeer
11 replies
Forum|Forum|4 years ago
March 8, 2021

Found the issue - on this system (z/OS 2.4) the zos_ssh_config had

ChannelConvert shell

It should have had

ChannelConvert shell,exec

Making this change and then doing this solved the problem

kill -s HUP $(cat /var/run/sshd.pid)

------------------------------
Lionel Dyck
lbdyck
------------------------------

Thanks for letting us know Lionel.Did you need to specify exec to get it to work with bpxwunix()?

Can you also post your resolution to the IBMMAIN thread for the archives?

Thanks

------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 8, 2021

Thanks for letting us know Lionel.Did you need to specify exec to get it to work with bpxwunix()?

Can you also post your resolution to the IBMMAIN thread for the archives?

Thanks

------------------------------
David Crayford
Senior Software Engineer
Rocket Software
------------------------------

One other interesting fact on this. On another z/OS 2.4 LPAR there is no problem and there is no zos_ssh_config file.

Curious....

------------------------------
Lionel Dyck
lbdyck
------------------------------

Jorn Thyssen
Rocketeer
22 replies
Forum|Forum|4 years ago
March 8, 2021

One other interesting fact on this. On another z/OS 2.4 LPAR there is no problem and there is no zos_ssh_config file.

Curious....

------------------------------
Lionel Dyck
lbdyck
------------------------------

Hi Lionel,

The default is shell,exec according to
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.foto100/fotz1141.htm

------------------------------
Joern Thyssen
Rocket Software
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 8, 2021

Hi Lionel,

The default is shell,exec according to
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.foto100/fotz1141.htm

------------------------------
Joern Thyssen
Rocket Software
------------------------------

That would explain why it works on the system without the zos_ssh_config. Why it was created on the system I had the issue on is a question that probably won't get an answer.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Jorn Thyssen
Rocketeer
22 replies
Forum|Forum|4 years ago
March 8, 2021

That would explain why it works on the system without the zos_ssh_config. Why it was created on the system I had the issue on is a question that probably won't get an answer.

------------------------------
Lionel Dyck
lbdyck
------------------------------

What if the sysprog used git to version control the config files and parmlib members... :) Infrastructure-as-code...

------------------------------
Joern Thyssen
Rocket Software
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 8, 2021

What if the sysprog used git to version control the config files and parmlib members... :) Infrastructure-as-code...

------------------------------
Joern Thyssen
Rocket Software
------------------------------

that would be an interesting exercise.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Jorn Thyssen
Rocketeer
22 replies
Forum|Forum|4 years ago
March 8, 2021

that would be an interesting exercise.

------------------------------
Lionel Dyck
lbdyck
------------------------------

I think long term it is a necessary exercise.

The process should really be:
1) Sysprog clones git repo
2) Sysprog changes ssh config file; git commit + git push
3) Sysprog starts pipeline (maybe ansible playbook) to deploy updated config (in this case it should deploy to /etc/ssh and issue the kill to refresh the daemon)

This is business-as-usual for developers and the sysprog is really just a infrastructure-developer :)

------------------------------
Joern Thyssen
Rocket Software
------------------------------

Lionel Dyck
Author
Participating Frequently
72 replies
Forum|Forum|4 years ago
March 8, 2021

I think long term it is a necessary exercise.

The process should really be:
1) Sysprog clones git repo
2) Sysprog changes ssh config file; git commit + git push
3) Sysprog starts pipeline (maybe ansible playbook) to deploy updated config (in this case it should deploy to /etc/ssh and issue the kill to refresh the daemon)

This is business-as-usual for developers and the sysprog is really just a infrastructure-developer :)

------------------------------
Joern Thyssen
Rocket Software
------------------------------

I agree it would be worthy exercise. I would strong encourage the git clone repo to be a unique directory and not the production location for those files to prevent overlaying the production files with an update from another system that may not work on the local system. I would like to see a 'read-only' repository which could reference the production ssh configs and other key files (perhaps most of /etc) that would add/commmit/push the updates but would not allow a pull.

Just me being very cautious.

------------------------------
Lionel Dyck
lbdyck
------------------------------

Sign up

Please log in or register:

Welcome to the Rocket Forum!

Please log in or register:

Scanning file for viruses.

This file cannot be downloaded