Open-source Languages & Tools for z/OS

 View Only
  • 1.  cURL 7.52.1 stdout encoding

    Posted 07-18-2018 04:58

    Hi, I have some trouble moving curl for z/OS from version 7.16.1 to 7.52.1.
    We call curl from a program written in REXX (using bpxbatch routine) to call a websevice on a remote (non z/OS) server; this webservice returns data already encoded in EBCDIC.
    Using the old version the data is retrieved with the correct encoding, while the new version shows bad encoded data; I obtain the same behaviour even if I call curl from ISHELL or BPXBATCH.
    Please note that we need to get the data from the stdout without writing it to a unix file (If I use the -o option to write to a unix file the data is written with the correct encoding).

    This is what happens with curl for z/OS version 7.16.1:

    curl -v --header “Xml-Stylesheet:/xsl/tws-ws2zos.xsl” --user “***:***” “http://***”

    • HPUC0330 About to connect() to *** port 80 (#0)
    • HPUC0016 Trying ...… HPUC0017 connected
    • HPUC0331 Connected to *** (...) port 80
    • HPUC0143 Server auth using Basic with user ‘***’

    GET *** HTTP/1.1.
    Authorization: Basic ***
    User-Agent: curl/7.16.1.
    Host: ***
    Accept: /.
    Xml-Stylesheet:/xsl/tws-ws2zos.xsl.
    .

    • HPUC0302 Pre readwrite setting chunky header values to default
      % Total % Received % Xferd Average Speed Time Time Time Current
      Dload Upload Total Spent Left Speed
      . 0 0 0 0 0 0 0 0 --:–:-- 0:00:01 --:–:-- 0.
      < Date: Wed, 18 Jul 2018 08:14:58 GMT.
      < Server: Apache.
      < Cache-Control: no-cache.
      < Content-Length: 31.
      < Keep-Alive: timeout=2, max=100.
      < Set-Cookie: JSESSIONID=***
      < Set-Cookie: JSESSIONID=***
      < Content-Type: application/xml.
      { [data not shown]
      . 0 31 0 31 0 0 3 0 --:–:-- 0:00:09 --:–:-- 6.

    • HPUC0320 Closing connection #0
      ORCS0, 3, X, BATCH, $42A, FP01

    The last line shows the output data displayed with the correct encoding.

    This is what happens with curl for z/OS version 7.52.1:

    curl -v --header “Xml-Stylesheet:/xsl/tws-ws2zos.xsl” --user “***:***” “http://***”

    • Trying ...
    • TCP_NODELAY set
      % Total % Received % Xferd Average Speed Time Time Time Current
      Dload Upload Total Spent Left Speed
      . 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0*
    • Server auth using Basic with user ‘***’

    GET *** HTTP/1.1.
    Host: ***
    Authorization: Basic ***
    User-Agent: curl/7.52.1.
    Accept: /.
    Xml-Stylesheet:/xsl/tws-ws2zos.xsl.
    .
    . 0 0 0 0 0 0 0 0 --:–:-- 0:00:01 --:–:-- 0.
    < Date: Wed, 18 Jul 2018 08:19:14 GMT.
    < Server: Apache.
    < Cache-Control: no-cache.
    < Content-Length: 31.
    < Keep-Alive: timeout=2, max=100.
    < Set-Cookie: JSESSIONID=***
    < Set-Cookie: JSESSIONID=***
    < Content-Type: application/xml.
    < .
    { [31 bytes data]

    • Curl_http_done: called premature == 0
      .100 31 100 31 0 0 5 0 0:00:06 0:00:05 0:00:01 7
    • Connection #0 to host *** left intact
      ÖÙÃâðk@ók@çk@ÂÁãÃÈk@[ôòÁk@Æ×ðñ.

    The data is shown with the bad encoding.

    I have tried several combinations of curl parameters (–raw, -B, --zos-output, and some env vars) without success.
    The only ways that I have found to get the correct output are:

    1. using the -o parameter to write the output to a unix file;
    2. pipe the stdout to the command “iconv -f IBM-1047 -t ISO8859-1”

    Is there any way to get curl work as the previous version?

    As additional information, this is the same command called from my Windows pc:

    curl -v --header “Xml-Stylesheet:/xsl/tws-ws2zos.xsl” --user “***:***” “http://***”

    • Trying ...
    • Connected to *** (...) port 80 (#0)
    • Server auth using Basic with user ‘***’

    GET *** HTTP/1.1
    Host: ***
    Authorization: Basic ***
    User-Agent: curl/7.50.0
    Accept: /
    Xml-Stylesheet:/xsl/tws-ws2zos.xsl

    < HTTP/1.1 200 OK
    < Date: Wed, 18 Jul 2018 08:00:23 GMT
    < Server: Apache
    < Cache-Control: no-cache
    < Content-Length: 31
    < Keep-Alive: timeout=2, max=100
    < Set-Cookie: JSESSIONID=***; path=/; HttpOnly
    < Set-Cookie: JSESSIONID=**; path=/; HttpOnly
    < Content-Type: application/xml
    <
    Í┘├Ô­k@¾k@þk@┬┴Ò├╚k@[¶‗┴k@ãέ±
    Connection #0 to host *** left intact

    Thanks in advance.
    Daniele



  • 2.  RE: cURL 7.52.1 stdout encoding

    Posted 07-18-2018 06:23

    Hi Daniele,

    Thanks for using our ported tools.
    Curl 7.16 supports only EBCDIC encoding, Curl 7.52 has Enhancement ASCII support. According to our current strategy we do only porting with ASCII support. So the current version will not work the same as the old version.
    When you use tools with ASCII support, you have to environmental variables:

    export _BPXK_AUTOCVT=ON ;
    export _CEE_RUNOPTS=“FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)” ;
    export _TAG_REDIR_ERR=txt ;
    export _TAG_REDIR_IN=txt ;
    export _TAG_REDIR_OUT=txt ;

    Please, add them into your REXX script. (See here)

    Regards,
    Tatyana



  • 3.  RE: cURL 7.52.1 stdout encoding

    Posted 07-18-2018 08:27

    Hi Tatyana,
    thank you for the reply. I have added the variables but nothing changed. I have also tried the code posted by srezepin replacing the yahoo url with the one of our webservice but the behaviour is always the same. The only difference is that I haven’t called curl through bash because it isn’t available in our system, but I think this wouldn’t solve the problem since RichardW stated that what solved the problem has been moving to the build 6, unfortunately we are already using the build 6 :frowning: I will ask my colleagues if we can install bash to make a second try. I will post updates asap.



  • 4.  RE: cURL 7.52.1 stdout encoding

    Posted 03-21-2019 16:22

    Hi,

    I can confirm that the problem with rockets current cURL build still exists.
    I updated the curl (and even all alvailabe opensource tools by rocket soft) to thre current level (bash-4.3_b018.170518 + curl-7.52.1_b006.170926.tar) and push the following exports via /etc/profile.

    export _BPXK_AUTOCVT=ON ;
    export _CEE_RUNOPTS=“FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)” ;
    export _TAG_REDIR_ERR=txt ;
    export _TAG_REDIR_IN=txt ;
    export _TAG_REDIR_OUT=txt ;
    

    I checked the result with…
    echo $_BPXK_AUTOCVT
    ON

    AUTOCVT is on, realy. I double checked with other tools supporting enhanced ascii. it works!

    When I download a zip file via curl https://testserver/test.zip unzip says allways the zip is broken.

    unzip test.zip
    Archive:  test.zip
      End-of-central-directory signature not found.  Either this file is not
      a zipfile, or it constitutes one disk of a multi-part archive.  In the
      latter case the central directory and zipfile comment will be found on
      the last disk(s) of this archive.
    unzip:  cannot find zipfile directory in one of test.zip or
            test.zip, and cannot find test.ZIP, period.
    

    Only if I use the special atrribute --zos-output=ASCII in my curl execution I get an processable zip file.
    In my opinion the build of curl from rocket soft is broken because it doesn’t honor the envars
    _BPXK_AUTOCVT and _CEE_RUNOPTS correctly. Although curl tags the downloaded zip file as binary the output isn’t untouched by autoconversation.

    May you please describe what --zos-output=ASCII realy does?
    Even when I use it the result ist tagged as binary.

    Kind regards,
    Björn Schanz



  • 5.  RE: cURL 7.52.1 stdout encoding

    Posted 07-18-2018 08:54

    I have found this version of bash in our system:

    GNU bash, version 2.03.0(1)-release (i370-ibm-mvs)
    Copyright 1998 Free Software Foundation, Inc.

    I have tried again passing the curl command as parameter to the above bash, but the result is the same.
    I hope we could find a better solution, in the meantime I will use iconv to get the right output.

    Thanks
    Daniele



  • 6.  RE: cURL 7.52.1 stdout encoding

    Posted 07-23-2018 09:02

    Hi Daniele,

    You should use only Bash 4.3, ported by Rocket. Because only Bash 4.3 has ASCII support.

    Could you tell me, where do you run the REXX script from? Did you try to run if from USS?



  • 7.  RE: cURL 7.52.1 stdout encoding

    Posted 07-23-2018 10:03

    Hi Tatyana,
    I run the rexx script from a batch job using the program IKJEFT01.
    Now I have pasted the script in a unix file and I have executed it from USS, the output is still wrong:

    BPXWUNIX RC=0

    STDOUT >

    ÖÙÃâðk@ók@çk@ÂÁãÃÈk@[ôòÁk@Æ×ðñ.

    I will ask to install the required Bash distribution and I’ll do another try; this may take a while since the required people are on vacation, I’ll let you know the results.

    Thank you
    Daniele



  • 8.  RE: cURL 7.52.1 stdout encoding

    Posted 07-24-2018 04:23

    Ok, I have installed the latest version of bash from Rocket Software:

    GNU bash, version 4.3.46(51)-release (i370-ibm-openedition)
    Copyright © 2013 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

    The output is wrong, whether I call the script from a batch job or the uss shell.