I have Conda installed and am using /etc/profile for the environment variables for Conda. I'm trying to execute curl in batch but not having much luck. I'm using the following JCL.
//CURL EXEC PGM=BPXBATCH
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//STDENV DD PATH='/etc/profile',PATHOPTS=ORDONLY
//STDIN DD DUMMY
//STDPARM DD *
sh 'curl --version'
/*
I keep getting "-bash: curl --version: command not found"
It's not finding the environment variables (PATH, MANPATH, etc).
This works:
//CURL EXEC PGM=BPXBATCH
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//STDENV DD PATH='/etc/profile',PATHOPTS=ORDONLY
//STDIN DD DUMMY
//STDPARM DD *
sh '/rsusr/conda/envs/default/bin/curl' --version
/*
What's the magic secret handshake/decoder ring to get curl to execute in batch? No, I don't want to issue the "conda activate default" command, I want it to use the /etc/profile environment variables I already have set up.
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
Hello Michael,
What's in your /etc/profile? That file usually contains shell commands, like '
export PATH=something', or '
echo something'. The STDENV DD has a different format, as described here:
https://www.ibm.com/docs/en/zos/2.1.0?topic=bpxbatch-guidelines-defining-stdenvRegards,
Vladimir
------------------------------
Vladimir Ein
------------------------------
Hello Michael,
What's in your /etc/profile? That file usually contains shell commands, like '
export PATH=something', or '
echo something'. The STDENV DD has a different format, as described here:
https://www.ibm.com/docs/en/zos/2.1.0?topic=bpxbatch-guidelines-defining-stdenvRegards,
Vladimir
------------------------------
Vladimir Ein
------------------------------
You are correct, it contains the standard /etc/profile with several export statements. When I do sh 'env', I get this:
MANPATH=$CONDA_PREFIX/share/man:$CONDA_PREFIX/man:/usr/man/%L:.
export CXXFLAGS="-q64 -fexec-charset=ISO8859-1"
SHELL=/bin/bash
CATALINA_HOME=/usr/local/tomcat/apache-tomcat-8.5.0
export PERL5LIB=$CONDA_PREFIX/lib/perl5/site_perl:$CONDA_PREFIX/lib/perl5
# export ${_CMP}_INCDIRS="/usr/include /usr/lpp/cbclib/include"
# export ${_CMP}_PLIB_PREFIX="CEE"
export _BPXK_AUTOCVT=ON
export SSL_CERT_DIR=$CONDA_PREFIX/ssl/certs
# export ${_CMP}_CLIB_PREFIX="CBC"
export LIBPATH=$LIBPATH:$CONDA_PREFIX/lib/perl5/5.24.4/os390/CORE
export SSL_CERT_FILE=$CONDA_PREFIX/ssl/cert.pem
export JAVA_HOME=/usr/lpp/java/current_64
LIBPATH=$CONDA_PREFIX/lib:/lib:/usr/lib:/usr/lpp/db2c10/jdbc/lib:/usr/lpp/IBM/dbb/lib:.
# STEPLIB DD statements, specify "STEPLIB=none" .
export PYTHON_VERSION=python37
*rlogin*) DISPLAY="$8";;
export PYV=py37
# VOL=SER= paramater is not supported by c89/cc/c++, all named data
# PATH=/bin:.
export ASCII_TERMINFO=$CONDA_PREFIX/share/terminfo
MAIL=/usr/mail/$LOGNAME
PATH=/bin
export CFLAGS="-q64 -fexec-charset=ISO8859-1"
export CC="njsc"
CONDA_PREFIX=/rsusr/conda/envs/default
PWD=/u/xx35
export ANT_HOME=$CONDA_PREFIX
export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
LANG=C
export STEPLIB=none
export _TAG_REDIR_ERR=txt
TZ=UTC0
export GIT_EXEC_PATH=$CONDA_PREFIX/libexec/git-core
# export ${_CMP}_WORK_UNIT="SYSDA"
export GIT_SHELL=$CONDA_PREFIX/bin/bash
export CXX_host=$CXX
export CURL_CA_BUNDLE=$CONDA_PREFIX/etc/ssl/cacert.pem
export PYTHON_DVERSION=python3.7
export CHARSETALIASDIR=$CONDA_PREFIX/lib
export _TAG_REDIR_OUT=txt
export _TAG_REDIR_IN=txt
SHLVL=1
HOME=/u/xx35
# For Japanese: LANG=Ja_JP
export LINK=$CXX
export GIT_EDITOR=oedit
export OPENSSL_CONF=$CONDA_PREFIX/ssl/openssl.cnf
LOGNAME=XX35
*telnet*) DISPLAY="$5";;
if Ý "$DISPLAY" = "None:0" ¨
# export ${_CMP}_LIBDIRS="/lib /usr/lib"
CLASSPATH=/usr/lpp/db2c10/jdbc/classes/db2jcc4.jar:\\
export LDFLAGS="-q64"
export GIT_TEMPLATE_DIR=$CONDA_PREFIX/share/git-core/templates
export CXX="njsc++"
INFOPATH=$CONDA_PREFIX/share/info/:.
# ======================================================================
DISPLAY="$DISPLAY:0"
*) DISPLAY="None";;
# export ${_CMP}_SLIB_PREFIX="SYS1"
export FFI_LIB=$CONDA_PREFIX/lib/ffi
_=/bin/env
What I find a little weird is that MANPATH is set just fine, but PATH isn't.
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
You are correct, it contains the standard /etc/profile with several export statements. When I do sh 'env', I get this:
MANPATH=$CONDA_PREFIX/share/man:$CONDA_PREFIX/man:/usr/man/%L:.
export CXXFLAGS="-q64 -fexec-charset=ISO8859-1"
SHELL=/bin/bash
CATALINA_HOME=/usr/local/tomcat/apache-tomcat-8.5.0
export PERL5LIB=$CONDA_PREFIX/lib/perl5/site_perl:$CONDA_PREFIX/lib/perl5
# export ${_CMP}_INCDIRS="/usr/include /usr/lpp/cbclib/include"
# export ${_CMP}_PLIB_PREFIX="CEE"
export _BPXK_AUTOCVT=ON
export SSL_CERT_DIR=$CONDA_PREFIX/ssl/certs
# export ${_CMP}_CLIB_PREFIX="CBC"
export LIBPATH=$LIBPATH:$CONDA_PREFIX/lib/perl5/5.24.4/os390/CORE
export SSL_CERT_FILE=$CONDA_PREFIX/ssl/cert.pem
export JAVA_HOME=/usr/lpp/java/current_64
LIBPATH=$CONDA_PREFIX/lib:/lib:/usr/lib:/usr/lpp/db2c10/jdbc/lib:/usr/lpp/IBM/dbb/lib:.
# STEPLIB DD statements, specify "STEPLIB=none" .
export PYTHON_VERSION=python37
*rlogin*) DISPLAY="$8";;
export PYV=py37
# VOL=SER= paramater is not supported by c89/cc/c++, all named data
# PATH=/bin:.
export ASCII_TERMINFO=$CONDA_PREFIX/share/terminfo
MAIL=/usr/mail/$LOGNAME
PATH=/bin
export CFLAGS="-q64 -fexec-charset=ISO8859-1"
export CC="njsc"
CONDA_PREFIX=/rsusr/conda/envs/default
PWD=/u/xx35
export ANT_HOME=$CONDA_PREFIX
export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
LANG=C
export STEPLIB=none
export _TAG_REDIR_ERR=txt
TZ=UTC0
export GIT_EXEC_PATH=$CONDA_PREFIX/libexec/git-core
# export ${_CMP}_WORK_UNIT="SYSDA"
export GIT_SHELL=$CONDA_PREFIX/bin/bash
export CXX_host=$CXX
export CURL_CA_BUNDLE=$CONDA_PREFIX/etc/ssl/cacert.pem
export PYTHON_DVERSION=python3.7
export CHARSETALIASDIR=$CONDA_PREFIX/lib
export _TAG_REDIR_OUT=txt
export _TAG_REDIR_IN=txt
SHLVL=1
HOME=/u/xx35
# For Japanese: LANG=Ja_JP
export LINK=$CXX
export GIT_EDITOR=oedit
export OPENSSL_CONF=$CONDA_PREFIX/ssl/openssl.cnf
LOGNAME=XX35
*telnet*) DISPLAY="$5";;
if Ý "$DISPLAY" = "None:0" ¨
# export ${_CMP}_LIBDIRS="/lib /usr/lib"
CLASSPATH=/usr/lpp/db2c10/jdbc/classes/db2jcc4.jar:\\
export LDFLAGS="-q64"
export GIT_TEMPLATE_DIR=$CONDA_PREFIX/share/git-core/templates
export CXX="njsc++"
INFOPATH=$CONDA_PREFIX/share/info/:.
# ======================================================================
DISPLAY="$DISPLAY:0"
*) DISPLAY="None";;
# export ${_CMP}_SLIB_PREFIX="SYS1"
export FFI_LIB=$CONDA_PREFIX/lib/ffi
_=/bin/env
What I find a little weird is that MANPATH is set just fine, but PATH isn't.
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
Well, I do not think MANPATH is set just fine. Upon a closer look, it contains unexpanded $CONDA_PREFIX. If you run '
man' with these variables, man will try to open the directory named
$CONDA_PREFIX/share/man - literally, with a dollar sign in the name. It's quite unlikely that you have a directory named '$CONDA_PREFIX' in your home directory; so this is gonna fail, but since MANPATH also contains /usr/man/%L, it's going to work as usual (except it won't find manuals from the miniconda directory).
A bottomline, you can't use /etc/profile in STDENV. Look at all those lines - Perl is not expecting a variable named 'export PERL5LIB' (it expects just PERL5LIB), XL C compiler won't know what the variable named '# export ${_CMP}_PLIB_PREFIX' is; and yes, there is a variable named '# PATH' in addition to the default PATH provided by the shell itself.
Regards,
Vladimir
------------------------------
Vladimir Ein
------------------------------
I have Conda installed and am using /etc/profile for the environment variables for Conda. I'm trying to execute curl in batch but not having much luck. I'm using the following JCL.
//CURL EXEC PGM=BPXBATCH
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//STDENV DD PATH='/etc/profile',PATHOPTS=ORDONLY
//STDIN DD DUMMY
//STDPARM DD *
sh 'curl --version'
/*
I keep getting "-bash: curl --version: command not found"
It's not finding the environment variables (PATH, MANPATH, etc).
This works:
//CURL EXEC PGM=BPXBATCH
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//STDENV DD PATH='/etc/profile',PATHOPTS=ORDONLY
//STDIN DD DUMMY
//STDPARM DD *
sh '/rsusr/conda/envs/default/bin/curl' --version
/*
What's the magic secret handshake/decoder ring to get curl to execute in batch? No, I don't want to issue the "conda activate default" command, I want it to use the /etc/profile environment variables I already have set up.
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
Try
//STDPARM DD *
SH curl --version(without the single quotes)
------------------------------
Jorn Thyssen
Solutions Advisor
Rocket Internal - All Brands
Waltham MA US
------------------------------
Well, I do not think MANPATH is set just fine. Upon a closer look, it contains unexpanded $CONDA_PREFIX. If you run 'man' with these variables, man will try to open the directory named $CONDA_PREFIX/share/man - literally, with a dollar sign in the name. It's quite unlikely that you have a directory named '$CONDA_PREFIX' in your home directory; so this is gonna fail, but since MANPATH also contains /usr/man/%L, it's going to work as usual (except it won't find manuals from the miniconda directory).
A bottomline, you can't use /etc/profile in STDENV. Look at all those lines - Perl is not expecting a variable named 'export PERL5LIB' (it expects just PERL5LIB), XL C compiler won't know what the variable named '# export ${_CMP}_PLIB_PREFIX' is; and yes, there is a variable named '# PATH' in addition to the default PATH provided by the shell itself.
Regards,
Vladimir
------------------------------
Vladimir Ein
------------------------------
Here's what's causing my misunderstanding from the USS User's Guide:
Note: When using BPXBATCH with the SH option (SH is the default), environment variables specified in
the STDENV DD are overridden by those specified in /etc/profile and .profile (which
overrides /etc/profile). This is because SH causes BPXBATCH to execute a login shell that runs
the /etc/profile script and runs the user's .profile.
I would have thought, when running BPXBATCH in JCL, it would read /etc/profile, but I guess that's not the case? My testing does not bear that out.
When I remove the //STDENV, here's what I get:
SHELL=/bin/bash
PATH=/bin
PWD=/u/xx35
TZ=UTC0
SHLVL=1
HOME=/u/xx35
LOGNAME=XX35
_=/bin/env
When I add STDENV, here's what I get. As you can see PATH is not set to what it is in STDENV (I have the /rsusr/conda stuff in it).
LDFLAGS="-q64"
MANPATH=/rsusr/conda/envs/default/share/man:/rsusr/conda/envs/default/man:/usr/man/%L:.
ASCII_TERMINFO=/rsusr/conda/envs/default/share/terminfo
GIT_SHELL=/rsusr/conda/envs/default/bin/bash
SSL_CERT_FILE=/rsusr/conda/envs/default/ssl/cert.pem
FFI_LIB=/rsusr/conda/envs/default/lib/ffi
SHELL=/bin/bash
CATALINA_HOME=/usr/local/tomcat/apache-tomcat-8.5.0
CURL_CA_BUNDLE=/rsusr/conda/envs/default/etc/ssl/cacert.pem
PERL5LIB=/rsusr/conda/envs/default/lib/perl5/site_perl:/rsusr/conda/envs/default/lib/perl5
_TAG_REDIR_IN=txt
ANT_HOME=/rsusr/conda/envs/default
LINK=$CXX
GIT_EDITOR=oedit
_TAG_REDIR_ERR=txt
CXXFLAGS="-q64 -fexec-charset=ISO8859-1"
LIBPATH=/rsusr/conda/envs/default/lib:/lib:/usr/lib:/usr/lpp/db2c10/jdbc/lib:/usr/lpp/IBM/dbb/lib:/rsusr/conda/envs/default/lib/perl
*rlogin*) DISPLAY="$8";;
PYTHON_VERSION=python37
MAIL=/usr/mail/$LOGNAME
PATH=/bin
PYV=py37
_BPXK_AUTOCVT=ON
PWD=/u/xx35
JAVA_HOME=/usr/lpp/java/current_64
LANG=C
TZ=UTC0
_CEE_RUNOPTS=FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)
PYTHON_DVERSION=python3.7
GIT_TEMPLATE_DIR=/rsusr/conda/envs/default/share/git-core/templates
CXX="njsc++"
SSL_CERT_DIR=/rsusr/conda/envs/default/ssl/certs
SHLVL=1
HOME=/u/xx35
# For Japanese: LANG=Ja_JP
OPENSSL_CONF=/rsusr/conda/envs/default/ssl/openssl.cnf
CFLAGS="-q64 -fexec-charset=ISO8859-1"
_TAG_REDIR_OUT=txt
LOGNAME=XX35
*telnet*) DISPLAY="$5";;
if Ý "$DISPLAY" = "None:0" ¨
CLASSPATH=/usr/lpp/db2c10/jdbc/classes/db2jcc4.jar:/usr/lpp/db2c10/jdbc/classes/db2jcc_javax.jar:/usr/lpp/db2c10/jdbc/classes/sqlj4.
INFOPATH=/rsusr/conda/envs/default/share/info/:.
# ======================================================================
CC="njsc"
DISPLAY="$DISPLAY:0"
CXX_host=$CXX
CHARSETALIASDIR=/rsusr/conda/envs/default/lib
*) DISPLAY="None";;
GIT_EXEC_PATH=/rsusr/conda/envs/default/libexec/git-core
_=/bin/env
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
Try
//STDPARM DD *
SH curl --version
(without the single quotes)
------------------------------
Jorn Thyssen
Solutions Advisor
Rocket Internal - All Brands
Waltham MA US
------------------------------
I get curl: command not found. The PATH environment variable is not being set correctly for some odd reason.
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
Well, I do not think MANPATH is set just fine. Upon a closer look, it contains unexpanded $CONDA_PREFIX. If you run 'man' with these variables, man will try to open the directory named $CONDA_PREFIX/share/man - literally, with a dollar sign in the name. It's quite unlikely that you have a directory named '$CONDA_PREFIX' in your home directory; so this is gonna fail, but since MANPATH also contains /usr/man/%L, it's going to work as usual (except it won't find manuals from the miniconda directory).
A bottomline, you can't use /etc/profile in STDENV. Look at all those lines - Perl is not expecting a variable named 'export PERL5LIB' (it expects just PERL5LIB), XL C compiler won't know what the variable named '# export ${_CMP}_PLIB_PREFIX' is; and yes, there is a variable named '# PATH' in addition to the default PATH provided by the shell itself.
Regards,
Vladimir
------------------------------
Vladimir Ein
------------------------------
I think I have found my issue. In my OMVS segment, my shell program is set to /bin/bash. /bin/bash is an alias to /rsusr/conda/envs/default/bin/bash. If I change my OMVS segment to use /bin/sh it works as advertised.
How can I use /bin/bash and still get the environment variables set correctly?
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
I get curl: command not found. The PATH environment variable is not being set correctly for some odd reason.
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
Hi Michael,
I think bash reads other files for the profile like ~/.bashrc or ~/.bash_profile.
I've included a
echo "Sourcing <whatever this script is named>"
in my .profile, .bashrc, .bash_profile, so I can figure out which one is read at what point.
I do not have great experiences using BPXBATCH with bash being the default shell... :/
------------------------------
Jorn Thyssen
Solutions Advisor
Rocket Internal - All Brands
Waltham MA US
------------------------------
Hi Michael,
I think bash reads other files for the profile like ~/.bashrc or ~/.bash_profile.
I've included a
echo "Sourcing <whatever this script is named>"
in my .profile, .bashrc, .bash_profile, so I can figure out which one is read at what point.
I do not have great experiences using BPXBATCH with bash being the default shell... :/
------------------------------
Jorn Thyssen
Solutions Advisor
Rocket Internal - All Brands
Waltham MA US
------------------------------
I can get around the issue by executing a shell script with this content:
#!/bin/bash -l
env
curl --version
This works just fine.
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
I can get around the issue by executing a shell script with this content:
#!/bin/bash -l
env
curl --version
This works just fine.
------------------------------
Michael Babcock
z/OS Systems Programmer, Sr
OneMain Financial
Evansville IN US
------------------------------
When bash is run under BPXBATCH, it starts in non-interactive mode. Per the official doc, it only executes /etc/profile when run in
interactive mode or when the
--login (
-l) key is specified. The latter probably explains why the script from Michael's last message works fine.
Here's the doc:
https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.htmlTo cut it short: when 'Program' in OMVS is set to
bash, the file
/etc/profile is
not executed; when it's
sh, both
/etc/profile and
$HOME/.profile are executed. And this is
normal, documented behavior of bash,
not something specific to z/OS.
There are multiple ways to work around this. For example, you can use the
BASH_ENV variable (described at the above page) to make it run /etc/profile. Something like this:
//STDENV DD *
BASH_ENV=/etc/profile
/*
This worked fine for me.
Another option would be to set OMVS shell to
sh, but then execute the script explicitly via bash. This should work with
sh:
//STDPARM DD *
sh /bin/bash -c "curl --version"
/*
Or you could specify bash in the shebang, if you're running a script and not just curl.
#!/bin/bash
curl --version
Which way to choose, much depends on exactly what you need.
Regards,
Vladimir
------------------------------
Vladimir Ein
------------------------------
When bash is run under BPXBATCH, it starts in non-interactive mode. Per the official doc, it only executes /etc/profile when run in
interactive mode or when the
--login (
-l) key is specified. The latter probably explains why the script from Michael's last message works fine.
Here's the doc:
https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.htmlTo cut it short: when 'Program' in OMVS is set to
bash, the file
/etc/profile is
not executed; when it's
sh, both
/etc/profile and
$HOME/.profile are executed. And this is
normal, documented behavior of bash,
not something specific to z/OS.
There are multiple ways to work around this. For example, you can use the
BASH_ENV variable (described at the above page) to make it run /etc/profile. Something like this:
//STDENV DD *
BASH_ENV=/etc/profile
/*
This worked fine for me.
Another option would be to set OMVS shell to
sh, but then execute the script explicitly via bash. This should work with
sh:
//STDPARM DD *
sh /bin/bash -c "curl --version"
/*
Or you could specify bash in the shebang, if you're running a script and not just curl.
#!/bin/bash
curl --version
Which way to choose, much depends on exactly what you need.
Regards,
Vladimir
------------------------------
Vladimir Ein
------------------------------
And then there is the sledgehammer approach.
In the early days, before getting a slightly better understanding of shells, I used a batch TSO step to build a shell script in /tmp, specifying ALL the environment variables with a subsequent BPXBATCH step to run curl.
Still works for all I know but I've since got a C program using the curl API's to deal with not having so much data in clear text.
shell.1 = 'export PATH=/bin:/usr/sbin:' || ,
'/apc/portedtools/bash/bin:/apc/portedtools/curl/bin:.'
shell.2 = 'export _BPXK_AUTOCVT="ON"'
shell.etc
shell.last = 'bash /somedir/runcurl.sh'
shell.0 = number of lines
x = syscalls('ON')
fname = '/tmp/'USERID()'.stdin';say 'Writing stdin to 'fname
Address syscall "writefile (fname) 600 shell."
lrc = rc
Say 'writefile finished with rc='lrc
x = syscalls('OFF')
then after running that in a TSO step
//UNIX EXEC PGM=BPXBATCH
//STDIN DD PATH='/tmp/&SYSUID..stdin'
The runcurl.sh has some bash style setup and then
curl --cert-type PEM --cacert $CACERT --netrc --silent -O -J -D /tmp/$LOGNAME.head --url $URL > /tmp/$LOGNAME.tmsg
I got the thing to run well enough without having to worry a lot about what was doing what to the env variables since this approach
specified everything......and more sophisticated approaches can develop later as time and reading threads like permit.
------------------------------
hank oerlemans
Senior Technical Specialist
Self Registered
North Sydney NSW AU
------------------------------
