z/OS Tools & Language

Expand all | Collapse all

Output from GIT always coming back in ASCII

Sergey Rezepin03-06-2018 03:06

  • 1.  Output from GIT always coming back in ASCII

    Posted 02-09-2018 02:55

    I have written a REXX exec to inface to GIT.

    Specifically I use BPXWUNIX to issue my commands. The issue I have is the output from the command is always in ASCII (ISO-8859-1). Forcing me the perform a conversion before I can read the results.

    This doesn’t happen if I issue the command in OMVS.

    I’ve tried using BPXWUNIX using redirection

    eg    x=BPXWUNIX('git status >output.txt',,,,,1)  
    

    and coding the stdout in the function call

    eg   x=BPXWUNIX('git status',,'output.',,,1) 
    

    I have my profile setup with

    export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
    export _BPXK_AUTOCVT=ON
    export _TAG_REDIR_ERR=TXT
    export _TAG_REDIR_IN=TXT
    export _TAG_REDIR_OUT=TXT

    Does anyone know how I can get the system/git to perform the conversion to EBCDIC before I get the output ?

    Regards,
    Gary



  • 2.  RE: Output from GIT always coming back in ASCII

    Posted 02-09-2018 08:24

    I am facing the same kind of issues. Not even when just interfacing from REXX, but also straight from the command line. The failing command on my end is “git pull”.

    It returns the data in ASCII and not EBCDIC, causing havoc in parsing output from commands like

    git rev-parse --is-bare-repository

    I made some changes in ‘git-sh-setup’ that make my git pull work, yet it does not feel like a ‘solid’ fix to me…

    Changes are:

    cd_to_toplevel () {                                                        
     cdup=$(git rev-parse --show-toplevel | iconv -f iso8859-1 -t ibm1047) &&  
     cd "$cdup" || {                                                           
      echo >&2 "Cannot chdir to $cdup, the toplevel of the working tree"       
      exit 1                                                                   
     }                                                                         
    }                                                                          
    
    require_work_tree_exists () {                                                                
     if test "z$(git rev-parse --is-bare-repository | iconv -f iso8859-1 -t ibm1047)" != zfalse  
     then                                                                                        
      die "fatal: $0 cannot be used without a working tree."                                     
     fi                                                                                          
    }                                                                                            
    
    git_dir_init () {                                                                
     GIT_DIR=$(git rev-parse --git-dir | iconv -f iso8859-1 -t ibm1047) || exit      
     if [ -z "$SUBDIRECTORY_OK" ]                                                    
     then                                                                            
      test -z "$(git rev-parse --show-cdup)" || {                                    
       exit=$?                                                                       
       echo >&2 "You need to run this command from the toplevel of the working tree."
       exit $exit                                                                    
      }                                                                              
     fi                                                                              
    

    i works, but makes me feel kidna stupid :)…

    Any other ideas>??



  • 3.  RE: Output from GIT always coming back in ASCII

    Posted 02-09-2018 09:07

    :facepalm:

    Just make sure you use the correect bash and live will be good (without the iconvs)



  • 4.  RE: Output from GIT always coming back in ASCII

    Posted 02-09-2018 19:54

    How does one “use the correct bash” ? Could it be that GIT hasn’t been installed correctly ?



  • 5.  RE: Output from GIT always coming back in ASCII

    Posted 02-11-2018 15:41

    I’ve been having the same issue. I don’t know what “the correct bash” means either. I have PROGRAM=/rsusr/ported/bin/bash in my OMVS segment, and that’s what shows up when I do echo $SHELL on a command line.



  • 6.  RE: Output from GIT always coming back in ASCII

    Posted 02-11-2018 21:57

    I spent a lot of time trying to understand this stuff over the weekend. I still have no solution but I have a little better understanding.

    It seems rocket has supplied two versions of bash. The “production” version is bash 4.2. I’ve found the following:

      /_PRDS/PTZ1/bash/share/doc/bash/README.ZOS
    
        Tool - bash-4.2                      
        Works only with the encoding EBCDIC  
    

    Plus there is a version delivered in the git directories

     /_PRDS/PTZ1/git/bash/share/doc/bash/README.ZOS
    
        Tool - bash-4.3  
        Works with the encoding ISO-8859-1 and IBM-1047 
    

    My conclusion the BASH delivered with git should work with both ASCII and EBCDIC files while the “production”/no-git version only support EBCDIC.

    If I type “bash --version” in OMVS I get

    $ bash --version
    bash: FSUM7351 not found
    $

    This is because neither bash is in my path.



  • 7.  RE: Output from GIT always coming back in ASCII

    Posted 02-11-2018 22:31

    Here is another example: if I type the following on OMVS

    echo 'Start GIT Status';  cd ~/zGIT-Repository/EBCDIC;  git status;  echo 'End GIT Status'; 
    

    I receive the following:

    Start GIT Status                                                                
    On branch master                                                                
                                                                                
    Initial commit                                                                  
                                                                                
    Untracked files:                                                                
      (use "git add <file>..." to include in what will be committed)                
                                                                                
            .gitattributes                                                          
            EXEC/                                                                   
                                                                                
    nothing added to commit but untracked files present (use "git add" to track)    
    End GIT Status                                                                  
    

    If I run the following REXX

    command="echo 'Start GIT Status';cd :/zGIT-Repository/EBCDIC;git status;echo 'End GIT Status';"
    rcode=bpxwunix(command,,'stdout.','stderr.','env.',1) 
    do i=1 to stdout.0                                    
       say stdout.i                                       
    end                                                   
    

    I receive the following:

    Start GIT Status                                                                                                                                               
    |>:::/>:::_/:::::::>:::/%::?__:::::>::/:,:::::%::::::::::::::::/::::::%::::::::?::>:%:::::>:::/::::%%:::::?__:::::::::::::
    /::::::::::::::::::>?:::>::/::::::?::?__::::::::>::/:,:::::%::::::::>::::::::::::/:::::?:::/:,::End GIT Status      
    

    The start and end messages are in EBCDIC the output from GIT is in ASCII. Which makes it impossible to do anything with.

    Regards,
    Gary



  • 8.  RE: Output from GIT always coming back in ASCII

    Posted 02-12-2018 05:16

    Hi,
    There is a file “Release notes for Git 2.3.5 for zOS.pdf” on our customer portal. It includes information about git dependecies:
    “Git for z/OS requires the latest Rocket z/OS ports of: bash 4.3…”.
    You CAN’T use git 2.3.5 with bash 4.2 or /bin/sh, because git 2.3.5 has enhanced ASCII support and bash 4.2 doesn’t.
    So right steps should be:

    1. Install bash-4.3 according to README.ZOS
    2. Install git-2.3.5 according to README.ZOS
    3. Be sure, that next settings are in your config (~/.profile or ~/.bashrc):
      export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
      export _BPXK_AUTOCVT=ON
      export _TAG_REDIR_ERR=TXT
      export _TAG_REDIR_IN=TXT
      export _TAG_REDIR_OUT=TXT
    4. Run bash-4.3
    5. Run git-2.3.5 over bash-4.3


  • 9.  RE: Output from GIT always coming back in ASCII

    Posted 02-12-2018 05:43

    yep. All done. Although I believe the “TXT” should be in lower case.

    Its using bash 4.3 as supplied in the git directories



  • 10.  RE: Output from GIT always coming back in ASCII

    Posted 02-12-2018 05:47

    We have a wrapper for git that looks like this

    #!/bin/sh

    name=$( basename $0 )
    dir=$( dirname $0 )

    ptz_dir="$dir/…/…"

    #export _BPXK_AUTOCVT=ON

    . $dir/_gitenv.sh $ptz_dir

    if [[ -x $GIT_BIN/$name ]]; then
    ."$GIT_BIN/$name" "$@"
    elif [[ -x $GIT_EXEC/$name ]]; then
    .$GIT_EXEC/$name "$@"
    else
    echo "unsupported call to $name"
    fi

    and _gitenv.sh looks like this

    #! /bin/sh

    if [[ $# > 0 ]]; then
    TGTDIR="$1"
    else
    TGTDIR="/_PRDS/PTZ"
    fi

    ZOS_GIT_DIR=$TGTDIR/git
    GIT_BIN=$ZOS_GIT_DIR/bin

    #for git use: source in bash, perl and unzip environment
    . $TGTDIR/etc/profile.d/unzip.sh $TGTDIR

    #. $TGTDIR/etc/profile.d/perl.sh $TGTDIR
    #. $TGTDIR/etc/profile.d/bash.sh $TGTDIR
    #NOTE: for the time being (i.e. as of April 2017) we deliver bash and perl inside git
    #as both are not yet ready for general use
    export PATH="$ZOS_GIT_DIR/bash/bin:$ZOS_GIT_DIR/perl/bin:$PATH"
    export LIBPATH="${TGTDIR}/perl/lib/perl5/5.22.0/os390/CORE:$LIBPATH"
    export PERL5LIB="${TGTDIR}/perl/lib/perl5:$PERL5LIB"

    #. $TGTDIR/etc/profile.d/bash.sh
    #for the time being bash 4.3 isn’t usable outside of git
    #there we cannot use the usual sourcing mechanism
    export PATH=$ZOS_GIT_DIR/bash/bin:$PATH

    export GIT_SHELL=$TGTDIR/bash/bin/bash

    #-z returns true if the string is empty
    if [[ -z ${GIT_TEMPLATE_DIR:-} ]]; then
    export GIT_TEMPLATE_DIR=$ZOS_GIT_DIR/share/git-core/templates
    fi

    export GIT_EXEC_PATH=$ZOS_GIT_DIR/libexec/git-core

    #make our wrapper scripts available
    export PATH=$PATH:$ZOS_GIT_DIR

    #MANPATH=$MANPATH:$ZOS_GIT_DIR/man
    #perl related
    export PERL5LIB=$PERL5LIB:$ZOS_GIT_DIR/lib/perl5

    #ASCII support
    export _BPXK_AUTOCVT=ON
    export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
    export _TAG_REDIR_ERR=txt
    export _TAG_REDIR_IN=txt
    export _TAG_REDIR_OUT=txt

    Regards,
    Gary



  • 11.  RE: Output from GIT always coming back in ASCII

    Posted 02-14-2018 08:21

    I have created a sample program that I’m hoping someone will run on their system and publish the results here.

    /* REXX */
    env.0=1
    env.1='HOME=/u/u700215’
    command=“echo ‘Start GIT Status’;”,
    “cd ~/zGIT-Repository/COPYX;”,
    “git status;”,
    "echo ‘End GIT Status’;"
    rcode=bpxwunix(command,‘stdout.’,‘stderr.’,‘env.’,1)
    if rcode>0 then say 'Rcode from bpxwunix='rcode
    say
    say copies(‘STDOUT–’,5)
    do i=1 to stdout.0
    say stdout.i
    end
    say copies(‘STDERR–’,5)
    do i=1 to stderr.0
    say stderr.i
    end

    Line 3 will need to be undated with your home directory,
    Line 5 should be changed so the “cd” points to a git repository.

    My results are:

    STDOUT–STDOUT–STDOUT–STDOUT–STDOUT–
    Start GIT Status
    |>:::/>:::_/:::::>?:::>:::?::?__:::::?:,:>::::::::?::::%:/>:End GIT Status
    STDERR–STDERR–STDERR–STDERR–STDERR–

    What do you get ???

    Regards,
    Gary



  • 12.  RE: Output from GIT always coming back in ASCII

    Posted 02-15-2018 05:32

    Hi Gary,

    git works over bash-4.3. When you run git via REXX this way, bash-4.3 is not started and these variables are not exported:

    export _BPXK_AUTOCVT=ON
    export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
    export _TAG_REDIR_ERR=txt
    export _TAG_REDIR_IN=txt
    export _TAG_REDIR_OUT=txt



  • 13.  RE: Output from GIT always coming back in ASCII

    Posted 02-15-2018 09:35

    Hi Tatyana,

    Thanks for the feedback. So what do you suggest ?

    While I’m no Unix expert, I believe our wrapper code does setup bash 4.3. How can I tell ?

    If bash 4.3 is not being used via this REXX, what do I need to do get it working ?

    Regards,
    Gary



  • 14.  RE: Output from GIT always coming back in ASCII

    Posted 02-16-2018 12:24

    I tried to figure out what’s going wrong.

    So I created ~/zGIT-Repository/COPYX and there I did a git init.

    Then I wrote a script setting all the variables as given in git/share/doc/git/2.3.5/README.ZOS

    #! /bin/sh
    ...
    git status
    

    When calling it from UNIX shell output is:

    On branch master

    Initial commit

    nothing to commit (create/copy files and use “git add” to track)

    So all seems good.

    Now I called above script using Gary’s REXX, and I get ASCII garbage as output from the git status command.

    Now comes the even stranger part:

    When I insert a curl command like this

    curl -s http://www.yahoo.com

    just before the git command then the REXX runs fine, i.e. no garbage at all.

    I have no idea why something like this happens.



  • 15.  RE: Output from GIT always coming back in ASCII

    Posted 02-20-2018 03:10

    Hi Manfred,

    So what you are saying is if you pass the following string to to BPXWUNIX

    echo ‘Start GIT Status’ ; cd ~/zGIT-Repository/COPYX ; git status ; echo ‘End GIT Status’

    The output from the git command comes back in ASCII, but if you pass the string

    echo ‘Start GIT Status’ ; cd ~/zGIT-Repository/COPYX ;curl -s http://www.yahoo.com ; git status ; echo ‘End GIT Status’

    The output from the git command comes back in EBCDIC

    Is that correct ?

    Gary



  • 16.  RE: Output from GIT always coming back in ASCII

    Posted 02-21-2018 16:08

    Hi Gary,
    Admittedly confusing but this is what I observed. Actually, I have no consistent theory what’s going on here.

    Manfred



  • 17.  RE: Output from GIT always coming back in ASCII

    Posted 02-22-2018 12:22

    Hi Tatyana,
    You could test like follows:

    1. Have git, bash, perl, unzip and curl all in a certain directory tree. In my case it is /u/manfred/rocket.

    2. In /u/manfred/rocket I have a script git_env which contains:

    ROCKET=/u/manfred/rocket
    export PATH=$ROCKET/bin:$PATH
    export _BPXK_AUTOCVT=ON
    export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
    export _TAG_REDIR_ERR=txt
    export _TAG_REDIR_IN=txt
    export _TAG_REDIR_OUT=txt

    and use this to source in the environment, i.e. . /u/manfred/rocket/rocket_env

    1. Go to a directory with a git repository and run
      git status

    This should worke fine.

    1. In a batch job run this REXX:

    /* REXX */
    env.0=1
    env.1=‘HOME=/u/manfred’

    /* This is a test case to show Rocket that something is wrong
    */

    BASE = “/u/manfred"
    ROCKET_BASE = BASE||”/rocket"

    command=". “ROCKET_BASE”/rocket_env;",
    ‘curl’,
    http://www.yahoo.com;’,
    “cd /u/manfred/zGIT-Repository/COPYX;”,
    “git status;”,
    “echo ‘End GIT Status’;”

    rcode=bpxwunix(command,‘stdout.’,‘stderr.’,‘env.’,1)
    if rcode>0 then say 'Rcode from bpxwunix='rcode
    say
    say copies(‘STDOUT-’,5)
    do i=1 to stdout.0
    say stdout.i
    end
    say copies(‘STDERR-’,5)
    do i=1 to stderr.0
    say stderr.i
    end

    This should run fine as well. Note that there is an invocation of curl before doing git status

    1. Delete the two curl related lines and run the job again.
      Now git output is ASCII gibberish.

    If you need a sample of a complete batch job just tell me. I could post it here or send it do you via mail. Whatever you like.

    Manfred



  • 18.  RE: Output from GIT always coming back in ASCII

    Posted 03-05-2018 15:29

    Just WOW on the activity and good replies here :slight_smile:

    Still a bit ‘untangible’ as to where it fails and why. I have no different results with bash 4.2 or 4.3. So for now the iconv-fix is making it work…

    If you want a ‘working’ experience too, I’ve git-gisted my solution here : https://gist.github.com/wizardofzos/c588e203aa14b80576712d664406df11



  • 19.  RE: Output from GIT always coming back in ASCII

    Posted 03-06-2018 03:06

    The internal issue is opened for this case.



  • 20.  RE: Output from GIT always coming back in ASCII

    Posted 03-23-2018 10:47

    A colleague of mine had a very good idea. He said: Check what curl and git do to STDOUT and STDERR in terms of tagging.

    I did and found that curl changes tagging to

    STDOUT: 1047 8000
    STDERR: 1047 8000

    This is the reason that git produces readable output when calling curl first.

    When calling git without prior invocation of curl no tagging takes place that is before and after calling git we have:

    STDOUT: 0000 0000
    STDERR: 0000 0000



  • 21.  RE: Output from GIT always coming back in ASCII

    Posted 03-25-2018 20:14

    Excellent Manfred.

    Is there anything I can do to change the tagging for GIT or is this an internal thing that Rocket has to do to the GIT product internally?

    Regards,
    Gary



  • 22.  RE: Output from GIT always coming back in ASCII

    Posted 03-27-2018 03:11

    Hi Gary,

    This problem occurs in git version, which was released more than in 2016. cURL uses new internal library, so this problem doesn’t exist in cURL. We are working on porting new git version (2.14), it will be released soon. The best way is to wait for new version.

    Thanks,
    Tatyana



  • 23.  RE: Output from GIT always coming back in ASCII

    Posted 03-27-2018 04:55

    For the time being, as a workaround I told Gary to do a

    curl --version

    before invoking git. This seems to be sufficient to get a readable output from git.

    I should add that simply tagging stdout and stderr as 1047 and text before invoking git isn’t enough to get readable output from git. Therefore, the workaround above.



  • 24.  RE: Output from GIT always coming back in ASCII

    Posted 03-27-2018 03:02

    Hi Manfred,

    Thanks for your participation on the forum =)



  • 25.  RE: Output from GIT always coming back in ASCII

    Posted 07-09-2018 00:16

    Any update on this “internal issue” ?



  • 26.  RE: Output from GIT always coming back in ASCII

    Posted 07-09-2018 02:34

    Hi Gary,

    The fix will be available for 2.14 git version.

    Thanks,
    Sergey