z/OS Tools & Language

Expand all | Collapse all

cURL returns EBCDIC response

  • 1.  cURL returns EBCDIC response

    Posted 05-02-2017 07:46

    I’m calling REST API services with cURL and get a JSON payload back. Using cURL 7.16.1 the response is in ASCII, however using cURL Version: 7.42.1 Build Number: 0013 the response comes back in EBCDIC. Additionally to the EBCDIC output there is error message curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding.This all could be seen in the trace files.
    As we run all that in z/OS 2.2 I would prefer the EBCDIC response, but how to react to the error message?
    Any help is appreciated.
    Thanks, Werner



  • 2.  RE: cURL returns EBCDIC response

    Posted 05-02-2017 07:50

    by the way, I tried the --use-ascii option mentioned in another thread to no avail.
    Werner



  • 3.  RE: cURL returns EBCDIC response

    Posted 05-03-2017 16:05

    We are also using the 7.42.1 build of curl and are experiencing a similar issue with JSON REST results. About half the time the results are returned in EBCDIC, and the other half it comes back in ASCII. We’ve controlled for this by running the output through the iconv utility when it comes back in ASCII.

    We are having an issue in certain cases where the codepage switches in the middle of the output, and half the string is in ASCII, half in EBCDIC. If anyone from rocket is monitoring this post - is there any explanation for why this would occur?

    –use-ascii had no effect for us as well



  • 4.  RE: cURL returns EBCDIC response

    Posted 05-04-2017 02:24

    Thanks Matt for sharing your experience. Oh no, your case is much worse than ours, if the coding changes within a string …
    Anyway, good to know I’m not alone :slight_smile:



  • 5.  RE: cURL returns EBCDIC response

    Posted 05-05-2017 13:14

    To clarify, we are also on build 13. I tried re-extracting the package over again, in case I had missed anything and still had the same results. For now, a retry on the failure message is a simpler coding challenge than trying to handle mixed output, but if this is something that can be addressed with a new build, I’d be happy to provide samples if it will help.



  • 6.  RE: cURL returns EBCDIC response

    Posted 05-12-2017 03:34

    cURL 7.52.1 was released.


    Could you please to try the build of this version?
    If you will have the problems too, please let us know.

    Thanks,
    Sergey



  • 7.  RE: cURL returns EBCDIC response

    Posted 05-12-2017 12:25

    Sergey,

    Thanks, I did see that announcement, and I have begun testing with the new version and it does consistently return an ASCII response, at least so far. With autoconvert, I do see it translated correctly to EBCDIC when I execute the query.

    I do have a related problem, but moreso with Enchanced ASCII in general, in that when cURL is executed from within a Perl script, autoconversion does not occur even though the environment variables appear to be set up correctly.

    Thanks,
    Matt



  • 8.  RE: cURL returns EBCDIC response

    Posted 05-16-2017 03:29

    Matt,
    where do you specify autoconvert to get EBCDIC?

    Thanks,
    Werner



  • 9.  RE: cURL returns EBCDIC response

    Posted 05-16-2017 03:11

    Sergey.
    thanks a lot for the new build. Using it shows some changes:

    • the cURL trace file is now ASCII (7.42.1 was EBCDIC)
    • the cURL response is now ASCII (7.42.1 was EBCDIC)
    • the JSON input file for the REST services have to be in ASCII now. As I call cURL from REXX (z/OS) the JSON file is created in EBCDIC. Now I have to convert it to ASCII before calling cURL. This is different to build 7.42.1, where the input file could be EBCDIC. Any chance/parm to change that?
    • the error message (56) Illegal or missing hexadecimal sequence in chunked-encoding disappeared. It ends now without any error messages.

    What you mean by “be sure your files are tagged”???
    Thanks,
    Werner



  • 10.  RE: cURL returns EBCDIC response

    Posted 05-16-2017 06:41

    Werner,

    What you mean by “be sure your files are tagged”???

    To see the current tags:

    ls -T

    To tag EBCDIC file:

    chtag -tc 1047 file

    To tag ASCII file:

    chtag -tc 819 file

    It may be helpful to avoid the behaviour when untagged files reads as ASCII.

    It’s recommended to tag files according to their encoding.
    And, in that case, cURL must be working correctly with the input tagged files independent of their encoding.

    All new files created by cURL have ASCII encoding and tagged as ASCII - it’s normal behaviour.

    Thanks,
    Sergey



  • 11.  RE: cURL returns EBCDIC response

    Posted 05-16-2017 09:14

    Werner,

    To set autoconvert on, there are a few USS environment variables to set. See the following post regarding unzip with Enhanced ASCII:

    And:
    https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ceea300/filtag.htm

    Regards,
    Matt



  • 12.  RE: cURL returns EBCDIC response

    Posted 05-17-2017 02:22

    Thanks, Sergey, the chtag command was unknown to me.
    So I tagged the input JSON file as EBCDIC, but I still get the error message “unexpected character”, because the data are required to be in ASCII (HTTP annd REST) and no conversion is done internally (at least my interpretation). I also tried the autoconvert options Matt supposed, The only effect was that the trace file returned is EBCDIC.
    Sorry to bother you, but I’m not very skilled with UNIX.
    Thanks,
    Werner



  • 13.  RE: cURL returns EBCDIC response

    Posted 05-22-2017 08:55

    Hi Werner,

    The following values specified for environment variables for the latest versions of bash in README.ZOS:

    export _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
    export _BPXK_AUTOCVT=ON
    export _TAG_REDIR_ERR=txt
    export _TAG_REDIR_IN=txt
    export _TAG_REDIR_OUT=txt
    

    But they are used not only for bash. Probably, we need to add this information to README.ZOS for cURL too.
    In that case, all new files create in ASCII encoding and tag as ISO8859-1 by cURL.
    The existent files read according to their tags.

    Could you please to write some examples of commands you are using for work with JSON and REST, especially what key/options are using for work with the files?

    Thanks,
    Sergey



  • 14.  RE: cURL returns EBCDIC response

    Posted 06-07-2017 01:51

    Hi Sergey,
    I’d like to send you data about my tests, content of JSON file, REST commands, comparison of test results with cURL 7.42.1 and 7.52.1, because they behave differently. However, this is too much for this forum. Is there a way to send this to you directly (if you are interested, of course)?
    Thanks,
    Werner



  • 15.  RE: cURL returns EBCDIC response

    Posted 06-07-2017 03:57

    Hi Werner,

    it is possible to attach a file(s) to your reply here. The 7th button on replying form is “Upload”. Could you try this way?

    Thanks,
    Tatyana



  • 16.  RE: cURL returns EBCDIC response

    Posted 06-08-2017 09:48

    Ok, here’s the first cURL_test_results.doc (28.5 KB)



  • 17.  RE: cURL returns EBCDIC response

    Posted 06-08-2017 09:49

    And the second… cURL cmd_JSONfile.doc (22.5 KB)



  • 18.  RE: cURL returns EBCDIC response

    Posted 07-05-2017 10:46

    Please note there will be a delay in delivering a new build that supports new environment variables previously mentioned. The engineering team has been diverted to a more urgent task.
    Regards,
    -Peter



  • 19.  RE: cURL returns EBCDIC response

    Posted 06-09-2017 08:43

    Hi Werner,

    Now, cURL is supporting only IBM-1047 and ISO8859-1 encodings for existent tagged files.
    And all new files created by cURL is being saved as ASCII and tagged as ISO8859-1 (the trace files are included).

    In next build, there is a planning to add the following environment variables:
    _ENCODE_FILE_NEW - to set encoding and tag for the new files.
    It’s allow to choose output encoding for files.
    _ENCODE_FILE_EXISTING - to control reading/writing untagged files properly.
    They could be set to the following values: ISO8859-1, IBM-1047, UNTAGGED, BINARY.

    Unfortunately, if you have files in another encoding (for example, IBM-1141) you need to convert them to ISO8859-1 or IBM-1047 and tag them in according with their encoding before using by cURL.
    iconv -f IBM-1141 -t IBM-1047 file1 > file2
    chtag -tc IBM-1047 file2
    or
    iconv -f IBM-1141 -t ISO8859-1 file1 > file2
    chtag -tc ISO8859-1 file2
    I’m studying your case and will discuss with team is there a possibility to use another EBCDIC code pages.

    Thanks,
    Sergey