We are currently experimenting with using Jenkins remote slave on USS to issue Git commands to Git installation. We have two test systems. On one of the test systems everything works as expected. On the second test system the Jenkins build fails during the Git fetch because it cannot find any remote branches. Both test systems are being invoked by the same Jenkins server and are connecting to the same remove Git server. We have not been able to identify what if any differences between the systems may be causing the inconsistent behavior.

We are using a standard Jenkins Git Client plugin to access the remote repository. It issues four commands at the beginning of each build. Jenkins Build Log from the failing test system:

Building remotely on tvt6031Dan in workspace /u/dbruce/test/jenkins/workspace/DansGitPull
 > /u/dbruce/test/dbb/sample/git-jenkins.sh rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > /u/dbruce/test/dbb/sample/git-jenkins.sh config remote.origin.url ssh://drbruce@abc.com/git/mortgage-repo.git # timeout=10
Fetching upstream changes from ssh://drbruce@abc.com/git/mortgage-repo.git
 > /u/dbruce/test/dbb/sample/git-jenkins.sh --version # timeout=10
using GIT_SSH to set credentials Dans ee-jenkins git credentials
 > /u/dbruce/test/dbb/sample/git-jenkins.sh fetch --tags ssh://drbruce@abc.com/git/mortgage-repo.git +refs/heads/*:refs/remotes/origin/*
Seen 0 remote branches
ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.

These commands work successfully when issued directly via the command line. After the build, the following error message appears in the Jenkins Slave log:

Jul 24, 2017 6:49:35 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '/var/rocket/bin', return code '128', error message '�atal: Could not switch to '/rsusr/rocket/etc/': EDC5129I No such file or directory.
'

We installed Git in /var/rocket and we invoke a gitenv.sh script at the beginning of the build so that everything is pointing to /var/rocket/* installation. We do not understand where the reference to /rsusr/rocket/etc is coming from. Is there anyway to redirect that reference to our installation directories in case it is trying to write some error or diagnostic messages?

Hi,
Thanks for report. We have received your request and we will work on this problem.

Can you please do the following and provide the output?

• Set the environment variable EDITOR to “env|sort; echo” – this will dump the environment at the time of the error and echo the argument to the editor.
• Set GIT_TRACE=<an absolute path to a trace file, like “/u/username/git_trace.txt”> – this will log some additional information about the command.
• Provide the contents of the gitenv.sh script.

Thanks!

There are no additional statements added to the Jenkins slave log or the build log (not sure if any expected). The contents of the trace file:

18:11:20.192077 ./git.c:351             trace: built-in: git 'config' '--global' 'core.editor' 'env|sort; echo'
18:12:03.264553 ./git.c:351             trace: built-in: git 'config' '--global' 'core.editor' 'env|sort; echo'
18:14:03.262706 ./git.c:351             trace: built-in: git 'config' '--global' 'core.editor' 'env|sort; echo'
18:16:17.099597 ./git.c:351             trace: built-in: git 'rev-parse' '--is-inside-work-tree'
18:16:17.364039 ./git.c:351             trace: built-in: git 'config' 'remote.origin.url' 'ssh://drbruce@ee-jenkins.rtp.raleigh.ibm.com/git/mortgage-repo.git'
18:16:17.654265 ./git.c:351             trace: built-in: git 'version'
18:16:19.148319 ./git.c:351             trace: built-in: git 'fetch' '--tags' '--progress' 'ssh://drbruce@abc.com/git/mortgage-repo.git' '+refs/heads/*:refs/remotes/origin/*'
18:16:19.149436 ./run-command.c:369     trace: run_command: '/tmp/ssh5001375550365929388.sh' 'drbruce@abc.com' 'git-upload-pack '\''/git/mortgage-repo.git'\'''
18:16:19.161524 ./run-command.c:131     trace: execvp: '/tmp/ssh5001375550365929388.sh' 'drbruce@abc.com' 'git-upload-pack '\''/git/mortgage-repo.git'\'''
18:16:21.955577 ./git.c:351             trace: built-in: git 'version'
18:16:22.035495 ./git.c:351             trace: built-in: git 'config' '--system' '--edit'

There are no ssh*.sh files in /tmp. I assume they were cleaned up automatically by someone. The contents of gitenv.sh is:

#!/bin/sh
git $@ | iconv -f ibm-1047 -t ibm-1047

We have found that this is necessary when using the Jenkins slave and the Bash version provided by Rocket on this test system. We do not know why. If we do not use the iconv command, the git fetch command fails with a return code status of 128 followed by a lot of garbage characters. Requiring the use of the ICONV command will be the subject of our next topic once this issue is resolved. :^)

Hmmm, well THAT didn’t do what I expected it to. Can you try again, but WITHOUT setting EDITOR? That seems to have backfired.

I’m confused about the use of gitenv.sh What I expect to see there was a bunch of export commands setting up the environment for git. Instead, it’s doing something odd WRT character encoding.

There are too many moving parts here that I can’t see to really understand what’s going on, but the z/OS git port relies on enhanced ASCII support for character conversion. Is something somewhere setting all the environment variables that are specified in the release notes? What does /u/dbruce/test/dbb/sample/git-jenkins.sh look like?

Sorry! My mistake. I mixed up the gitenv.sh file with the git-jenkins.sh file. /u/dbruce/test/dbb.sample.git-jenkins.sh file contains the pipe to iconv posted earlier. Then contents of gitenv.sh is:

#!/bin/sh

# git the environment variables
export GIT_SHELL=/var/rocket/bin/bash
export GIT_EXEC_PATH=/var/rocket/libexec/git-core
export GIT_TEMPLATE_DIR=/var/rocket/share/git-core/templates
export PREFIX=/u/dbruce/prefix
#export EDITOR="env|sort; echo"
export GIT_TRACE=/u/dbruce/git_trace.txt

#common the environment variables
export PATH=/var/rocket/bin:$PATH
export MANPATH=$MANPATH:/var/rocket/man
export PERL5LIB=$PERL5LIB:/var/rocket/lib/perl5
export LIBPATH=$LIBPATH:/var/rocket/lib/per15/5.24.0/os390/CORE

#ASCII support the environment variables
export _BPXK_AUTOCVT=ON
export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
export _TAG_REDIR_ERR=txt
export _TAG_REDIR_IN=txt
export _TAG_REDIR_OUT=txt

#set git editor to create comments on encdoing ISO8859-1
#git config --global core.editor "/bin/vi -W filecodeset=ISO8859-1"

Here is the updated contents of the trace file:

DBRUCE:/u/dbruce>cat git_trace.txt
19:26:15.077296 ./git.c:351             trace: built-in: git 'rev-parse' '--is-inside-work-tree'
19:26:15.315788 ./git.c:351             trace: built-in: git 'config' 'remote.origin.url' 'ssh://drbruce@ee-jenkins.rtp.raleigh.ibm.com/git/mortgage-repo.git'
19:26:15.610612 ./git.c:351             trace: built-in: git 'version'
19:26:17.055400 ./git.c:351             trace: built-in: git 'fetch' '--tags' '--progress' 'ssh://drbruce@ee-jenkins.rtp.raleigh.ibm.com/git/mortgage-repo.git' '+refs/heads/*:refs/remotes/origin/*'
19:26:17.056541 ./run-command.c:369     trace: run_command: '/tmp/ssh7857973656542272771.sh' 'drbruce@ee-jenkins.rtp.raleigh.ibm.com' 'git-upload-pack '\''/git/mortgage-repo.git'\'''
19:26:17.136736 ./run-command.c:131     trace: execvp: '/tmp/ssh7857973656542272771.sh' 'drbruce@ee-jenkins.rtp.raleigh.ibm.com' 'git-upload-pack '\''/git/mortgage-repo.git'\'''
19:26:20.032178 ./git.c:351             trace: built-in: git 'version'
19:26:20.115092 ./git.c:351             trace: built-in: git 'config' '--system' '--edit'

Additionally here is the Jenkins slave log file which does contain a lot of environment variables. (All the statements except the last warning statement are from the slave startup):

[07/25/17 15:24:55] [SSH] Opening SSH connection to xyz.com:22.
[07/25/17 15:24:55] [SSH] Authentication successful.
[07/25/17 15:24:56] [SSH] The remote users environment is:
@="sh"
ERRNO="0"
HOME="/u/dbruce"
IFS=" 	
"
LINENO="0"
LOGNAME="DBRUCE"
MAIL="/usr/mail/DBRUCE"
MAILCHECK="600"
OPTIND="1"
PATH="/bin"
PPID="67108909"
PS1="\$ "
PS2="> "
PS3="#? "
PS4="+ "
PWD="/u/dbruce"
RANDOM="25130"
SECONDS="0"
SHELL="/bin/sh"
SSH_CLIENT="1.2.3.4 32820 22"
SSH_CONNECTION="1.2.3.4 32820 5.6.7.8 22"
USER="DBRUCE"
[07/25/17 15:24:56] [SSH] Starting sftp client.
[07/25/17 15:24:56] [SSH] Copying latest slave.jar...
[07/25/17 15:24:58] [SSH] Copied 712,800 bytes.
Expanded the channel window size to 4MB
[07/25/17 15:24:58] [SSH] Starting slave process: . /u/dbruce/test/gitenv.sh && export JAVA_HOME=/usr/lpp/java/J7.0_64/ && export IBM_JAVA_ENABLE_ASCII_FILETAG=ON && env && cd "/u/dbruce/test/jenkins" && /usr/lpp/java/J7.0_64/bin/java -Dfile.encoding=ISO8859-1 -Xnoargsconversion -jar slave.jar
MAIL=/usr/mail/DBRUCE
PATH=/var/rocket/bin:/bin
_TAG_REDIR_IN=txt
GIT_EXEC_PATH=/var/rocket/libexec/git-core
GIT_TRACE=/u/dbruce/git_trace.txt
GIT_SHELL=/var/rocket/bin/bash
SSH_CLIENT=1.2.3.4 32820 22
_BPXK_AUTOCVT=ON
SHELL=/bin/sh
IBM_JAVA_ENABLE_ASCII_FILETAG=ON
_CEE_RUNOPTS=FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)
_=/bin/env
LOGNAME=DBRUCE
LIBPATH=:/var/rocket/lib/per15/5.24.0/os390/CORE
GIT_TEMPLATE_DIR=/var/rocket/share/git-core/templates
_TAG_REDIR_OUT=txt
USER=DBRUCE
HOME=/u/dbruce
PERL5LIB=:/var/rocket/lib/perl5
PREFIX=/u/dbruce/prefix
SSH_CONNECTION=1.2.3.4 32820 5.6.7.8 22
JAVA_HOME=/usr/lpp/java/J7.0_64/
_TAG_REDIR_ERR=txt
MANPATH=:/var/rocket/man
<===[JENKINS REMOTING CAPACITY]===>channel started
Slave.jar version: 3.0
This is a Unix agent
Agent successfully connected and online
Jul 25, 2017 7:26:20 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '/var/rocket/bin', return code '128', error message '�atal: Could not switch to '/rsusr/rocket/etc/': EDC5129I No such file or directory.
'

Hi,
Which versions of z/OS are installed on these systems?

Both systems are running z/OS 2.02

Did you try to run Git fetch directly on the system with wrong behaviour (I mean without Jenkins slave). If not, could you try?

Yes, the commands original post above all work successfully when issued directly via the command line.

I’m a bit confused as to what Jenkins is trying to do here. The trace file indicates that something is trying to run git config --system --edit, which should start an interactive editor on the config file, which doesn’t seem right.

Anyway, I have a few suggestions that may shed a little light:

• Please set these variables in gitenv.sh:

  • GIT_TRACE_SETUP=true
  • GIT_FLUSH=1

• Are the login shells the same on the two systems? You can check this from USS by running:

  /bin/tsocmd listuser dbruce omvs | grep PROGRAM

I’ve added the two new variables in gitenv.sh and reran the test but it does not look like any additional information was added to git_trace.txt:

DBRUCE:/u/dbruce>cat git_trace.txt
14:35:26.370994 ./git.c:351             trace: built-in: git 'rev-parse' '--is-inside-work-tree'
14:35:26.602884 ./git.c:351             trace: built-in: git 'config' 'remote.origin.url' 'ssh://drbruce@ee-jenkins.rtp.raleigh.ibm.com/git/mortgage-repo.git'
14:35:26.859714 ./git.c:351             trace: built-in: git 'version'
14:35:28.265430 ./git.c:351             trace: built-in: git 'fetch' '--tags' '--progress' 'ssh://drbruce@ee-jenkins.rtp.raleigh.ibm.com/git/mortgage-repo.git' '+refs/heads/*:refs/remotes/origin/*'
14:35:28.266485 ./run-command.c:369     trace: run_command: '/tmp/ssh5628409048352164143.sh' 'drbruce@ee-jenkins.rtp.raleigh.ibm.com' 'git-upload-pack '\''/git/mortgage-repo.git'\'''
14:35:28.335791 ./run-command.c:131     trace: execvp: '/tmp/ssh5628409048352164143.sh' 'drbruce@ee-jenkins.rtp.raleigh.ibm.com' 'git-upload-pack '\''/git/mortgage-repo.git'\'''
14:35:30.925004 ./git.c:351             trace: built-in: git 'version'
14:35:31.000139 ./git.c:351             trace: built-in: git 'config' '--system' '--edit'

The results of the tsocmd | grep command are the same on both systems: /bin/sh Also in the Jenkins slave startup logs for both systems they are the same as well: SHELL=/bin/sh

*** Rocket internal tracking reference: USSP-830 ***

You mentioned that this works on one system but not another. Have you tried adding the trace variable to the system that works? It would be interesting to see if there are any differences.

I continue to be puzzled by the intent of git config --system --edit. Do you know where that is coming from, and what it’s supposed to do?

Hi,

Could you tell me if there is a ‘/rsusr/rocket/etc/’ directory on both machines?

Neither system had a /rsusr/rocket/etc/ directory originally. In both cases we did not have any references to that directory in our configuration. The only time that directory appeared in the Jenkins log was on the system that was failing during the Git pull. We eventually obtained permission to create that directory and once we did, that error message went away though the Git pull stilled failed. When we looked into the directory after the Git pull failed, there were no additional files created. Our guess is that the directory is hard coded somewhere in your code that gets executed during an error situation. We were hoping some additional diagnostic information was being logged there but in the end nothing was created.

We did eventually figure out that there was a configuration issue on the system that was failing during the Git pull. The problem was that the z/OS TCPIP RESOLVER is not configured correctly causing the domain name in the URL of the Git repository to not resolve.

The Jenkins slave under OMVS uses the RESOLVER task for domain name resolution (this is also true for REXX scripts and C applications). On z/OS there are 2 ways to start the RESOLVER: either let UNIX start it or define a z/OS started procedure. (See https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.2.0/com.ibm.zos.v2r2.halz002/resolver_starting.htm). When UNIX starts the RESOLVER, it starts with a default configuration that doesn’t include the DNS definitions.

A minor complaint we have for Rocket Git is that it seems to be swallowing the original error message and instead just issuing “Seen 0 remote branches” which is technically true but not very helpful when the problem is that a connection could not be created due to DNS failure. We are currently investigating another Jenkins issue where the Git pull is once again failing after we upgraded to the latest version of the Jenkins Git client plugin (and after we fixed the Resolver configuration). In that case as well the only error message we see in the log is “Seen 0 remote branches”.

Hi,

Thanks for explanation and feedback. We are sorry that it was pretty difficult to find the real reason. However, Rocket Git is original git, which works in z/OS. We don’t make changes in tool’s functionality.

I am having this same issue. What was the problem with the Resolver and how did you fix it?

Thanks.

Issue: A Jenkins Git client plugin failure may occur if the z/OS TCPIP RESOLVER is not configured correctly on USS. This condition will cause the Jenkins build to fail with the message “Seen 0 remote branches” appearing in the log.

Cause: The Jenkins slave under OMVS uses the RESOLVER task for domain name resolution (this is also true for REXX scripts and C applications). On z/OS there are 2 ways to start the RESOLVER: either let UNIX start it or define a z/OS started procedure. When UNIX starts the RESOLVER, it starts with a default configuration that doesn’t include the DNS definitions.
(See https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.2.0/com.ibm.zos.v2r2.halz002/resolver_starting.htm).

Resolution: Customize the z/OS RESOLVER to include DNS definitions. Instructions can be found here: http://www-01.ibm.com/support/docview.wss?uid=swg21066874
Alternately as a work around, substituting the domain name in the Git URL for the actual IP address will work as well i.e. https://github.com/IBM -> https://192.30.253.113/IBM

Thanks. I have verified everything with USS resolver and it is working fine. Checking the server side, gethostbyname.

OK, I was incorrect. It is an issue on z/OS. When I run the Jenkins project to the master on Linux, it works fine. When I run it to z/OS, it fails. So, any advice on how to resolve this issue would be helpful. I have deleted and re-installed all the Rocket Open Source software.

Thanks,
Jerry

I have tried using the IP address of the git server, but it still fails with the same error message. It appears the git commands are not working, properly. How do I trace git on z/OS?

Hi, Could you please try with new git: git-2.14.4_b08.181016.tar.gz ?

Tatyana,

I was able to get everything working and with git-2.14.4. Not sure what the problem was, because I was unable get any other messages to display. Tried trace git, but no luck. It wasn’t USS RESOLVER, because using the IP address of the GIT server failed as well. So, I completely removed all the open source and re-installed the following;

• Bash v4.3, git v2.14.4, gzip v1.6, perl v5.24.0, unzip v6.0, and vim v8.0.22
• Very carefully followed all the installation guides for each product
• Update the /etc/profile on z/OS. See below for the updated parts of /etc/profile
• Added the two DBB scripts, supplied by IBM, getevn.sh and git-jenkins.sh
• Updated the z/OS slave
  o . /usr/lpp/IBM/dbb/conf/gitenv.sh && export JAVA_HOME=/usr/lpp/java/J8.0_64/ && export IBM_JAVA_ENABLE_ASCII_FILETAG=ON && env &&
  o Don’t forget to have the ‘. ‘ (dot/space) in the front of the prefix start slave command
• Use ssh to access z/OS

For the Rocket Open Source install, I use /usr/lpp/tools

JAVA_HOME=/usr/lpp/java/J8.0_64
export JAVA_HOME
TOOLS_HOME=/usr/lpp/tools
export TOOLS_HOME

PATH=/bin:.
PATH=$PATH:$JAVA_HOME/bin
PATH=$PATH:$TOOLS_HOME/bin
export PATH

LIBPATH=/lib:/usr/lib
LIBPATH=$LIBPATH:$JAVA_HOME/bin
LIBPATH=$LIBPATH:$JAVA_HOME/lib
LIBPATH=$LIBPATH:$JAVA_HOME/lib/s390x
LIBPATH=$LIBPATH:$JAVA_HOME/lib/s390x/j9vm
LIBPATH=$LIBPATH:$TOOLS_HOME/lib
LIBPATH=$LIBPATH:$TOOLS_HOME/lib/perl5/5.24.0/os390/CORE
export LIBPATH

===========================================

CLASSPATH=$JAVA_HOME/lib
CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/ext

export CLASSPATH

======================================================================

IJO="-Xms1024M -Xmx1024M"
IJO="$IJO -Xshareclasses"
export IBM_JAVA_OPTIONS="$IJO "

MANPATH=/usr/man/%L
MANPATH=$MANPATH:$TOOLS_HOME/man
export MANPATH

======================================================================

export _BPX_SHAREAS=YES

bash environment variables

======================================================================

export _ENCODE_FILE_NEW=IBM-1047
export _ENCODE_FILE_EXISTING=IBM-1047
export _CEE_RUNOPTS=“FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)”
export _BPXK_AUTOCVT=ON
export _TAG_REDIR_ERR=txt
export _TAG_REDIR_IN=txt
export _TAG_REDIR_OUT=txt

git environment variables

======================================================================

export GIT_SHELL=$TOOLS_HOME/bin/bash
export GIT_EXEC_PATH=$TOOLS_HOME/libexec/git-core
export GIT_TEMPLATE_DIR=$TOOLS_HOME/share/git-core/templates
#git config --global core.editor vim

perl environment variables

======================================================================

PERL5LIB=$TOOLS_HOME/lib/perl5
PERL5LIB=$PERL5LIB:$TOOLS_HOME/share/perl/5.24.1
export PERL5LIB

vim environment variables

======================================================================

export VIM=$TOOLS_HOME/share/vim/

I did have an issue with the slave logon account to z/OS. In its home directory /$HOME/.gitconfig, the Jenkins slave was trying to use it, but didn’t have access, even with chmod 777. So, I commented out the “#git config --global core.editor vim”, deleted the .gitconfig, and restart the slave. That problem went away.

I believe the cause of the issue was twofold, but I can’t prove it

• The missing “. “ in the prefix slave start
• Improper installation and configuration of the Rocket Open Source

What would be very helpful, if there a document or some help about how to trace git to get more information about various problems.

Thanks,
Jerry