Skip to main content

We have curl installed from Ported Tools from around 2018 I believe. 
When using SSL (i.e. without "-k") it fails on most LPARS as follows

curl -vvv --cacert /u/userid/certs/cacert.crt https://registry.hub.docker.com/v1/repositories/ibmcom/etcd/tags

 curl: (77) error setting certificate verify locations:
 CAfile: /u/userid/certs/cacert.crt
 CApath: none


Setting CURL_CA_BUNDLE yields the same result, as does setting --cafile. (export CURL_CA_BUNDLE="/u/userid/certs/cacert.crt")
I reviewed my .profile and the only other related setting looks fine: export OPENSSL_CONF=/apps/4g00/Ported_tools/curl/ssl/openssl.cnf 
The system /etc/profile also has nothing obvious.

It does work on one LPAR, which conveniently has a shared /u/userid/.profile with a system that fails. 
The same curl command below is used on both systems.
The system /etc/profile between the two systems look the same.
The working host happens to be z/OS 2.5 but according to my notes it also worked when it was z/OS 2.4. 
Different locations for the ca file yield the same results. 
Not coding --cacert and CURL_CA_BUNDLE seems to "work" in that curl  goes with the default ca-certs which doesn't have our required company issued CAs.

  * SSL certificate problem: unable to get local issuer certificate
  * Curl_http_done: called premature == 1

=> Any idea of what to look at next?  I'm thinking it's some kind of USS security access issue to the specified CAfile (/u/userid/certs/cacert.crt), but the permissions are fine.
Below is the relevant info that "-vvv" provides

Bruce


Versions:
- curl --version => curl 7.52.1 (i370-ibm-openedition) libcurl/7.52.1 OpenSSL/1.0.2l zlib/1.2.11 libssh2/1.8.0 nghttp2/1.18.1 
- openssl version => OpenSSL 1.0.2k 26 Jan 2017

Detailed error flow: (
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* error setting certificate verify locations:
CAfile: /u/q3fjj4g/certs/cacert.crt
CApath: none
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (77) error setting certificate verify locations:
CAfile: /u/userid/certs/cacert.crt
CApath: none



------------------------------
Bruce Marshall
Architect, Infrastructure
Royal Bank of Canada
Toronto ON CA
------------------------------

We have curl installed from Ported Tools from around 2018 I believe. 
When using SSL (i.e. without "-k") it fails on most LPARS as follows

curl -vvv --cacert /u/userid/certs/cacert.crt https://registry.hub.docker.com/v1/repositories/ibmcom/etcd/tags

 curl: (77) error setting certificate verify locations:
 CAfile: /u/userid/certs/cacert.crt
 CApath: none


Setting CURL_CA_BUNDLE yields the same result, as does setting --cafile. (export CURL_CA_BUNDLE="/u/userid/certs/cacert.crt")
I reviewed my .profile and the only other related setting looks fine: export OPENSSL_CONF=/apps/4g00/Ported_tools/curl/ssl/openssl.cnf 
The system /etc/profile also has nothing obvious.

It does work on one LPAR, which conveniently has a shared /u/userid/.profile with a system that fails. 
The same curl command below is used on both systems.
The system /etc/profile between the two systems look the same.
The working host happens to be z/OS 2.5 but according to my notes it also worked when it was z/OS 2.4. 
Different locations for the ca file yield the same results. 
Not coding --cacert and CURL_CA_BUNDLE seems to "work" in that curl  goes with the default ca-certs which doesn't have our required company issued CAs.

  * SSL certificate problem: unable to get local issuer certificate
  * Curl_http_done: called premature == 1

=> Any idea of what to look at next?  I'm thinking it's some kind of USS security access issue to the specified CAfile (/u/userid/certs/cacert.crt), but the permissions are fine.
Below is the relevant info that "-vvv" provides

Bruce


Versions:
- curl --version => curl 7.52.1 (i370-ibm-openedition) libcurl/7.52.1 OpenSSL/1.0.2l zlib/1.2.11 libssh2/1.8.0 nghttp2/1.18.1 
- openssl version => OpenSSL 1.0.2k 26 Jan 2017

Detailed error flow: (
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* error setting certificate verify locations:
CAfile: /u/q3fjj4g/certs/cacert.crt
CApath: none
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (77) error setting certificate verify locations:
CAfile: /u/userid/certs/cacert.crt
CApath: none



------------------------------
Bruce Marshall
Architect, Infrastructure
Royal Bank of Canada
Toronto ON CA
------------------------------

Hi Bruce,

Could you check the tag of your CA file?

ls -T /u/userid/certs/cacert.crt

If it's untagged, you will see the following output:

- untagged T=off cacert.crt

Please try to change the tag to IBM-1047:

chtag -tc IBM-1047 /u/userid/certs/cacert.crt

The tag should match with encoding of the file content. You can check the file is readable by cat tool, for example:

cat /u/userid/certs/cacert.crt

If the file is readable,  try to run your curl command.

curl -vvv --cacert /u/userid/certs/cacert.crt <url>

More likely, it should be tagged as IBM-1047, but there is a chance that ISO8859-1 tag should be set.

If the file is not readable after IBM-1047 tag, try to tag your file as ISO8859-1: 

chtag -tc ISO8859-1 /u/userid/certs/cacert.crt

Check if it's readable, and try to run your curl command again.

cat /u/userid/certs/

curl -vvv --cacert /u/userid/certs/cacert.crt <url>

Thanks,

Sergei



------------------------------
Sergei Rezepin
Rocket Software
------------------------------

Hi Bruce,

Could you check the tag of your CA file?

ls -T /u/userid/certs/cacert.crt

If it's untagged, you will see the following output:

- untagged T=off cacert.crt

Please try to change the tag to IBM-1047:

chtag -tc IBM-1047 /u/userid/certs/cacert.crt

The tag should match with encoding of the file content. You can check the file is readable by cat tool, for example:

cat /u/userid/certs/cacert.crt

If the file is readable,  try to run your curl command.

curl -vvv --cacert /u/userid/certs/cacert.crt <url>

More likely, it should be tagged as IBM-1047, but there is a chance that ISO8859-1 tag should be set.

If the file is not readable after IBM-1047 tag, try to tag your file as ISO8859-1: 

chtag -tc ISO8859-1 /u/userid/certs/cacert.crt

Check if it's readable, and try to run your curl command again.

cat /u/userid/certs/

curl -vvv --cacert /u/userid/certs/cacert.crt <url>

Thanks,

Sergei



------------------------------
Sergei Rezepin
Rocket Software
------------------------------

Thanks Segei!  Your line of reasoning helped get this working, though I'm not sure exactly why. 
I needed to enable  AutoConvert (AutoCVT)  with the _BPXK_AUTOCVT envar (=ON). 
To elaborate my test and further question:
 SP1 = host where it already worked; autoconvert was enabled in /etc/profile
 SP2=host where it failed unless I set _BPXK_AUTOCVT =ON  (not YES).
 I actually have two ca files which was convenient for tagged (EBCDIC) and untagged cases.
I don't understand why curl needs AutoConvert enabled when  both ca files are actually EBCDIC. 

Any ideas? For developed access I'm fine with just doing "Export _BPXK_AUTOCVT=ON" before the curl but wonder if something in USS isn't setup right?
Best guess is related to port of curl.  BTW I also tested with the bash (vs native SH) shell with the same results. 

Bruce

Shared file system SP1/SP2:
/u/userid/certs>ls -T cacert.crt
t IBM-1047 T=on cacert.crt
/u/userid/certs>ls -T rbc-ca-bundle.cer
- untagged T=off rbc-ca-bundle.cer
Both files are actually EBCDIC (confirmed using TSO iShell browse)

Tests on SP2:
  curl -vvv --cacert /u/userid/certs/cacert.crt <url>    [tagged file]
  curl -vvv --cacert /u/userid/certs/rbc-ca-bundle.cer <url>    [untagged]

1) By default AutoCVT is off (unset) : both tests  fail  (tagged and untagged)
2) Export _BPXK_AUTOCVT=ON > both files work with curl (EBCDIC or untagged)
3) Set tag to ascii (chtag -tc ISO8859-1), AutoCVT on or off=> fails

System setup
SP1/2: SYS1.PARMLIB(CEEPRM00): FILETAG(NOAUTOCVT,NOAUTOTAG)
SP1: /etc/profile: _BPXK_AUTOCVT=ON
SP2: /etc/profile: _BPXK_AUTOCVT not set



------------------------------
Bruce Marshall
Architect, Infrastructure
Royal Bank of Canada
Toronto ON CA
------------------------------

Thanks Segei!  Your line of reasoning helped get this working, though I'm not sure exactly why. 
I needed to enable  AutoConvert (AutoCVT)  with the _BPXK_AUTOCVT envar (=ON). 
To elaborate my test and further question:
 SP1 = host where it already worked; autoconvert was enabled in /etc/profile
 SP2=host where it failed unless I set _BPXK_AUTOCVT =ON  (not YES).
 I actually have two ca files which was convenient for tagged (EBCDIC) and untagged cases.
I don't understand why curl needs AutoConvert enabled when  both ca files are actually EBCDIC. 

Any ideas? For developed access I'm fine with just doing "Export _BPXK_AUTOCVT=ON" before the curl but wonder if something in USS isn't setup right?
Best guess is related to port of curl.  BTW I also tested with the bash (vs native SH) shell with the same results. 

Bruce

Shared file system SP1/SP2:
/u/userid/certs>ls -T cacert.crt
t IBM-1047 T=on cacert.crt
/u/userid/certs>ls -T rbc-ca-bundle.cer
- untagged T=off rbc-ca-bundle.cer
Both files are actually EBCDIC (confirmed using TSO iShell browse)

Tests on SP2:
  curl -vvv --cacert /u/userid/certs/cacert.crt <url>    [tagged file]
  curl -vvv --cacert /u/userid/certs/rbc-ca-bundle.cer <url>    [untagged]

1) By default AutoCVT is off (unset) : both tests  fail  (tagged and untagged)
2) Export _BPXK_AUTOCVT=ON > both files work with curl (EBCDIC or untagged)
3) Set tag to ascii (chtag -tc ISO8859-1), AutoCVT on or off=> fails

System setup
SP1/2: SYS1.PARMLIB(CEEPRM00): FILETAG(NOAUTOCVT,NOAUTOTAG)
SP1: /etc/profile: _BPXK_AUTOCVT=ON
SP2: /etc/profile: _BPXK_AUTOCVT not set



------------------------------
Bruce Marshall
Architect, Infrastructure
Royal Bank of Canada
Toronto ON CA
------------------------------

There was a bug in the build of cURL when cURL expects ASCII/Latin-1 content in untagged files. It looks, _BPXK_AUTOCVT works as a workaround, but initially it is an incorrect behavior in this build.

Usually, we recommend to set _BPXK_AUTOCVT=ON and some other variables for proper work with tagged files:

export _BPXK_AUTOCVT=ON
export _CEE_RUNOPTS='FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)'
export _TAG_REDIR_IN=txt
export _TAG_REDIR_OUT=txt
export _TAG_REDIR_ERR=txt

Also, the setting of _BPXK_AUTOCVT=ON and export _CEE_RUNOPTS='FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)' are the Miniconda installation requirements.



------------------------------
Sergei Rezepin
Rocket Software
------------------------------

There was a bug in the build of cURL when cURL expects ASCII/Latin-1 content in untagged files. It looks, _BPXK_AUTOCVT works as a workaround, but initially it is an incorrect behavior in this build.

Usually, we recommend to set _BPXK_AUTOCVT=ON and some other variables for proper work with tagged files:

export _BPXK_AUTOCVT=ON
export _CEE_RUNOPTS='FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)'
export _TAG_REDIR_IN=txt
export _TAG_REDIR_OUT=txt
export _TAG_REDIR_ERR=txt

Also, the setting of _BPXK_AUTOCVT=ON and export _CEE_RUNOPTS='FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)' are the Miniconda installation requirements.



------------------------------
Sergei Rezepin
Rocket Software
------------------------------
Thanks again Sergei! I now understand why the one test host where curl worked has the above envars you list set - it's where the Ported Tools were installed and prev testing was done with file tagging.  We have all the envar set except POSIX(ON), which looking at the doc would default to POSIX(OFF). Will correct that in the future.
I found an installed version of curl on the same host of the newer curl v7.73.0 (release date 2020-10-14), and it works without having to set autoconvert :).  Some details below.  
Will probably wait for the AppDev 1.2 update and then update again  before sending to the confused developer trying to curl. 

Bruce 

/u/userid2/miniconda/envs/SYSPROG/bin>printenv _BPXK_AUTOCVT
OFF
/u/userid2/miniconda/envs/SYSPROG/bin>curl --version
curl 7.73.0 (i370-ibm-openedition) libcurl/7.73.0 OpenSSL/1.1.1h zlib/1.2.11 libssh2/1.8.0 nghttp2/1.33.0
Release-Date: 2020-10-14
Protocols: dict file ftp ftps gopher http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz Metalink NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

curl -vvv --cacert /u/userid/certs/cacert.crt https://registry.hub.docker.com/v1/repositories/ibmcom/etcd/tags

< HTTP/1.1 200 Connection established
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /u/userid/certs/rbc-ca-bundle.cer
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
....

------------------------------
Bruce Marshall
Architect, Infrastructure
Royal Bank of Canada
Toronto ON CA
------------------------------