Open-source Languages & Tools for z/OS

 View Only
Expand all | Collapse all

Running git commands in batch using JCL

  • 1.  Running git commands in batch using JCL

    Posted 09-18-2017 17:02

    It can sometimes be useful to run git commands in batch from JCL. This turns out to be a bit tricky; here’s an approach that works for me. I would be interested in hearing about other approaches.

    The key documentation for this is UNIX System Services User’s Guide: The BPXBATCH utility. There are several issues that make this challenging:

    • There are some unpleasant restrictions on the handling of the standard I/O streams, stdin/stdout/stderr:
      • All three standard I/O streams are, by default, directed to /dev/null.
      • stdin cannot be directed to an MVS dataset. It may be directed to a file in the USS file system.
      • stdout and stderr may be directed to MVS datasets, but you have to be careful about the DCB characteristics.
    • The command to be executed is passed to BPXBATCH via the PARM option on the EXEC statement; this is limited to a maximum length of 100 characters.

    My JCL for running BPXBATCH looks like this:

     //STEP1 EXEC PGM=BPXBATCH
     //STDOUT  DD SYSOUT=*
     //STDPARM DD *
     SH command_to_run ...
     /*
    

    The STDPARM DD statement replaces the PARM= option on the EXEC statement and increases the length of the parameter string to abount 32K. By removing the command line from the EXEC statement, it also makes the JCL easier to read.

    If only STDOUT is allocated, both of the stdout and stderr streams will be sent to the same MVS dataset. This is handy because it blends the two streams together in order, much as they are when running a shell from the command line. If you want the two streams separated, you can provide a DD statement for STDERR as well.

    So that’s the actual JCL, but what should the STDPARM dataset look like? Here’s an example:

     //STEP1 EXEC PGM=BPXBATCH
     //STDOUT  DD SYSOUT=*
     //STDPARM DD *
     SH /rsusr/ported/bin/bash -c '
     
     export USER=TSJLC ;
     export _BPXK_AUTOCVT=ON ;
     export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)" ;
     export _TAG_REDIR_ERR=txt ;
     export _TAG_REDIR_IN=txt ;
     export _TAG_REDIR_OUT=txt ;
     export _BPXK_JOBLOG=STDERR ;
     export _EDC_ADD_ERRNO2=1 ;
     export PATH=/rsusr/ported/bin:/bin ;
     export LIBPATH=/rsusr/ported/lib:/usr/lib ;
     
     set -x ;
    
     git clone git@github.com:zorts/hello_world.git ;
     cd hello_world ;
     git status ;
     git remote -v ;
     '
     /*
    

    I’ll explain this section by section.

     SH /rsusr/ported/bin/bash -c '
    

    This executes the bash shell; the path to bash will depend on where you installed it. The -c command line option causes bash to execute the argument of -c as a command. Note that the last character on that line is a single quote; this starts the command to be run, and the closing quote is at the very end of the input stream.

    Following this first line are the commands to be executed. Note that, as far as bash is concerned, these commands are all on a single “line” (because they are just the argument to the -c option). This means that:

    • each command must be terminated with a semicolon, and
    • you must not try to use comments in this stream! The first hash character (#) will cause all the remaining input to be treated as comment.

    The next chunk of lines sets up the environment for git:

     export USER=TSJLC ;
     export _BPXK_AUTOCVT=ON ;
     export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)" ;
     export _TAG_REDIR_ERR=txt ;
     export _TAG_REDIR_IN=txt ;
     export _TAG_REDIR_OUT=txt ;
     export _BPXK_JOBLOG=STDERR ;
     export _EDC_ADD_ERRNO2=1 ;
     export PATH=/rsusr/ported/bin:/bin ;
     export LIBPATH=/rsusr/ported/lib:/usr/lib ;
    

    There’s no reason these couldn’t be put into a USS file and then sourced (using the . command), which will make this job stream less verbose. However, providing the setup in the job stream makes it very obvious what’s being done.

    This command:

     set -x ;
    

    turns on bash's debugging mode, causing it to display every command it’s about to execute. This isn’t strictly necessary, but it make debugging a bit easier.

    Finally, we get to the actual git commands:

     git clone git@github.com:zorts/hello_world.git ;
     cd hello_world ;
     git status ;
     git remote -v ;
     '
    

    Note that final closing single quote!

    Assuming you have already installed git for z/OS and set up GitHub to be used from z/OS, you could run the JCL above unmodified and it should work.



  • 2.  RE: Running git commands in batch using JCL

    Posted 10-19-2017 05:18

    Let’s face it, BPXBATCH sucks! I would advise anybody wanting to run shell scripts in batch to check out COZBATCH from Dovetailed Technologies. It’s free to use. It can run bash just fine.

    I love bash and have it set as my login shell in the OMVS segment. One issue is that it doesn’t handle local spawns. Local spawns are pretty easy to do and it would be great if Rockets bash port had support.



  • 3.  RE: Running git commands in batch using JCL

    Posted 10-19-2017 09:49

    I’ve created a Rocket internal enhancement request for support of local spawn: PYAN-564. I can’t say anything about a possible implementation date, though.



  • 4.  RE: Running git commands in batch using JCL

    Posted 10-19-2017 10:02

    Thanks Jerry! As long as it’s on the radar that’s cool.



  • 5.  RE: Running git commands in batch using JCL

    Posted 09-13-2018 12:19

    Dave, do you have a jcl that does a clone using COZBATCH. I tried with the one below. The clone happens but the only output i get is in STDERR. I am able to use putty and view the repo. I am a novice so any help would be greatly appreciated. Thanks
    //CLONEST EXEC PGM=COZBATCH
    //STEPLIB DD DSN=SYS1.COZSSH.LOADLIB,DISP=SHR
    //SYSOUT DD SYSOUT=*
    //STDENV DD *
    GIT_SHELL=/usr/local/bin/rocket/bin/bash
    GIT_EXEC_PATH=/usr/local/bin/rocket/libexec/git-core
    GIT_TEMPLATE_DIR=/usr/local/bin/rocket/share/git-core/templates
    PATH=$PATH:/usr/local/bin/rocket/bin
    MANPATH=$MANPATH:/usr/local/bin/rocket/man
    LIBPATH=/usr_local/bin/rocket/lib/perl5/5.24.0/os390/CORE:$LIBPATH
    PERL5LIB=$PERL5LIB:/usr/local/bin/rocket/lib/perl5
    _BPXK_AUTOCVT=ON
    _CEE_RUNOPTS=“FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)”
    _TAG_REDIR_ERR=txt
    _TAG_REDIR_IN=txt
    _TAG_REDIR_OUT=txt
    _BPXK_JOBLOG=STDERR
    _EDC_ADD_ERRNO2=1
    //STDOUT DD SYSOUT=*
    //STDERR DD SYSOUT=*
    //STDIN DD *
    git clone git@xxxgitlab.xxxx/Samples.git
    /*

    stderr output

    Ý: /etc/profile 52: FSUM7351 not found
    Cloning into ‘Samples’…


    • Systems require authorization; unauthorized access is illegal. *

    Note - If you are having trouble please contact your normal support channels

    Also it doesnt like the quotes around POSIX
    CEE3611I The run-time option “FILETAG was an invalid run-time option or is not supported
    Environment.
    CEE3605I The string '”’ was found where a delimiter was expected following the s
    POSIX.



  • 6.  RE: Running git commands in batch using JCL

    Posted 09-14-2018 16:38

    Greg
    Ignore my question. That was just me being new. Apparently the output of clone goes to STDERR. But it was writing only the first couple of lines saying cloning into. I found out by googling that you had to add --progress. Now it produces a whole bunch of output. :slight_smile: Not ideal but this is way better that nothing. Thanks



  • 7.  RE: Running git commands in batch using JCL

    Posted 10-19-2017 10:12

    This is probably a bit controversial for a mainframe ISV but I think it would be great if Rocket open sourced their tools on Github. Lets face it, there’s no competition in the market and willing participants could contribute to improve the tools saving Rocket valuable developer resources. I know I would be willing. Now we have Git on z/OS it’s easy to clone repositories and use push requests for review. My 2 cents anyway.



  • 8.  RE: Running git commands in batch using JCL

    Posted 10-19-2017 11:18

    I don’t have access to your Jira system. For some reason my user/password on this forum doesn’t work. I tried to reset my password but didn’t get an e-mail. Can you direct me to somebody that can solve this issue? Cheers.



  • 9.  RE: Running git commands in batch using JCL

    Posted 10-19-2017 11:21

    Forget that I can now see that Jira is not accessible outside of Rocket staff. I was confused by the link :wink:



  • 10.  RE: Running git commands in batch using JCL

    Posted 10-19-2017 11:43

    Sorry, David, for the confusion - I put the link there to make it easy for the Rocket developer to get to the associate Jira issue.



  • 11.  RE: Running git commands in batch using JCL

    Posted 10-27-2017 08:05

    Kirk Wolf pointed me to converting fork() and exec() Usage to spawn(). Kirk thinks it’s a heavy lift and I’m inclined to believe him as he is an SME in this space.



  • 12.  RE: Running git commands in batch using JCL

    Posted 03-20-2018 17:59

    Hi
    I ran this piece of jcl after modifying it to match our paths. I ran with the commands given below. The output of the list files looks great. The output from the pwd and git status looks garbled. Guessing some encoding issue. Has anybody ran into this ? Please let me know. Thanks.

    set -x ;
    pwd ;
    cd mainframe-test ;
    ls -la ;
    git status ;



  • 13.  RE: Running git commands in batch using JCL

    Posted 03-27-2018 03:11

    Hi,

    Please, look at this comment: Output from GIT always coming back in ASCII