Skip to main content

Webclient Questions

  • December 3, 2015
  • 11 replies
  • 0 views
D

I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

Couple questions:

1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

Any guidance would be appreciate.

    11 replies

    Stephen Hjerpe
    • Stephen Hjerpe
    • Participating Frequently
    • December 4, 2015

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    Can you tell me what WebClient .NET is?

    Are yo working with Silk ... Silk > Silk Performer > Silk Performer 16.5 > Silk Performer Workbench > Load Testing Specific Application Types > .NET Support > Silk Performer .NET Explorer > Using the .NET Message Sample > Working With the .NET Message Sample > Exploring the .NET Message Sample

    or something else?

    Stephen Hjerpe
    • Stephen Hjerpe
    • Participating Frequently
    • December 4, 2015

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    (407) Proxy Authentication Required error is the access to the URL resource requires the prior use of a proxy server ... see www.checkupdown.com/.../E407.html ... somehow your login /  authentication is getting lost or perhaps was cached to begin with .. I would think that the US Treasury Dept is very concerned about authentication.

    D

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    Throwing this out there based on the feedback - could load balancing be happening?   So you're going out one server ten times in a row and suddenly going out a different server where the previous access is not established??  Dunno.  We use IE to obtain the SDN files from Treasury manually.

    D

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    The files we are downloading from the US Treasury website are generally available public files, they are available to anyone to download (they contain lists of suspected 'bad guys' from around the world).  The entire program is copied below - it is quite simple except that it is using WebClient functions to do the actual file downloads.  Our biggest struggle is that when the WebClient function has a problem it crashes the entire AcuGT runtime instead of letting us capture the error and handle it gracefully.

          IDENTIFICATION DIVISION.

          PROGRAM-ID. WEBCLIENT-CBL.

          DATE-WRITTEN. 10/09/2015.

          AUTHOR.

          REMARKS.

         *

         * .NET Web Client research program

         *

          ENVIRONMENT DIVISION.

          CONFIGURATION SECTION.

          SPECIAL-NAMES.

               COPY "WebClient".

               .

         *INPUT-OUTPUT SECTION.

          FILE-CONTROL.

         *

         *

          DATA DIVISION.

          FILE SECTION.

         *

         *

           WORKING-STORAGE SECTION.

          01  PROGRAM-NAME  pic x(20) value "WEBCLIENT.CBLV.23.00".

         *

         * Handles:

         *    Window handles:

          77  WStatus-Handle             handle of window.

          77  WCall-Handle               handle.

          77  WCancel-Handle             handle of thread.

          77  WWebClient-Handle

                 usage is handle of "@System.System.Net.WebClient".

          77  WS-ADDR-SDN         pic x(89) value                          

              "www.treasury.gov/.../sdn.csv".          

          77  WS-FileName-SDN     pic x(18) value                        

              "C:\\TMP\\SDN".                                        

          77  WS-ADDR-ALT         pic x(89) value                          

              "www.treasury.gov/.../alt.csv".          

          77  WS-FileName-ALT     pic x(18) value                        

              "C:\\TMP\\ALT".                                        

          77  WS-ADDR-SDNCMTS     pic x(89) value                          

              "www.treasury.gov/.../sdn_comments.csv".

          77  WS-FileName-SDNCMTS pic x(18) value                        

              "C:\\TMP\\SDNCMTS".                                        

          77  WS-ADDR-CON         pic x(89) value                          

              "www.treasury.gov/.../cons_pr

         -    "im.csv".          

          77  WS-FileName-CON     pic x(18) value                        

              "C:\\TMP\\CONPRM".                                        

          77  WS-ADDR-CONALT         pic x(89) value                          

              "www.treasury.gov/.../cons_al

         -    "t.csv".          

          77  WS-FileName-CONALT     pic x(18) value                        

              "C:\\TMP\\CONAKA".                                        

          77  WS-ADDR-CONCMTS     pic x(89) value                          

              "www.treasury.gov/.../cons_co

         -    "mments.csv".

          77  WS-FileName-CONCMTS pic x(18) value                        

              "C:\\TMP\\CONCMTS".                                        

          PROCEDURE DIVISION.

         *

         * CREATE  - instantiate a NON-GUI CLASS *

              CREATE "@System"

              NAMESPACE is "System.Net"

              CLASS-NAME is "WebClient"

              HANDLE is WWebClient-Handle.

         *    * Set the Default Credentials *

              Modify WWebClient-Handle

                 UseDefaultCredentials is 1.

         *    * Invoke the Download file method *

              PERFORM 10 TIMES

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-SDN, WS-FILENAME-SDN)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-ALT, WS-FILENAME-ALT)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-SDNCMTS, WS-FILENAME-SDNCMTS)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-CON, WS-FILENAME-CON)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-CONALT, WS-FILENAME-CONALT)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-CONCMTS, WS-FILENAME-CONCMTS)

              END-PERFORM.

              Exit Program.

    Thomas Morrison
    • Thomas Morrison
    • Participating Frequently
    • December 7, 2015

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    Since the HTTPS protocol is being used, I would think this could be done quite simply using RMnet (which I think was available in extend 9).

    Tom Morrison, Consultant, South Carolina, USA
    D

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    Tom beat me to the punch in suggesting RMNet. However, we'd need to be sure the files are pure text files, as RMNet has a limitation that it doesn't handle binary payloads.

    Thomas Morrison
    • Thomas Morrison
    • Participating Frequently
    • December 7, 2015

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    The files referenced in the example code are text - CSV.

    Tom Morrison, Consultant, South Carolina, USA
    D

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    you can use BITSAdmin.exe  which is part of Windows 7.

    But for later Windows Versions the Windows PowerShell cmdlets should be better.

    technet.microsoft.com/.../ff382721.aspx

    Thomas Morrison
    • Thomas Morrison
    • Participating Frequently
    • December 8, 2015

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    The following program, in RM/COBOL dialect (because that's what I have installed), uses RMNet to do an HTTP GET on one of the files, and store the result in "results.txt".  The data that are stored are identical to that fetched by the browser(!).  The example uses XML Extensions for the convenience of storing the resulting document (returned in memory) in a single instruction.  No other XML Extensions methods are used.

    RMNet will return the HTTP status and even the result HTTP headers (optional) so that you can have a lot of control over the behavior.

           identification division.
           program-id.  BadGuy.
           data division.
           working-storage section.
           01  response-payload            usage pointer.
           01  response-error              usage pointer.
           01  response-status             pic 9(3) value zero.

           78  Get-Address value
                 "www.treasury.gov/.../alt.csv".
               Copy "lixmlall.cpy".
           linkage section.
           01 http-response pic x(500).
           01 http-error pic x.

           procedure division.
           a.
               XML INITIALIZE.
               if not xml-ok go to z.

               call "NetInit" giving response-status.
               call "NetSetSSLCA" using "ca-bundle.crt".
               call "HttpGet"
                 using
                   Get-Address
                   response-payload
                 giving
                   response-status.

               if response-status not = 0
                   call "NetGetError" using response-error
                   set address of http-response to response-error
                   display http-response(1:)
                   set address of http-response to null
                   call "NetFree" using response-error
               else
                   if response-payload not equal null
                       XML PUT TEXT
                           response-payload
                           "response.txt"
                       if not xml-ok go z end-if
                   end-if
               end-if.

               call "NetCleanup".

               stop run.

           z.
               Perform Display-Error-Status
               XML TERMINATE.
               Stop Run.

           Display-Error-Status.
               If Not XML-OK
                   Display "!!!!!!!!XML Error Occurred!!!!!!!!"
                   Display "XML Status: " XML-Status
                   Perform With Test After Until XML-NoMore
                       XML GET STATUS-TEXT
                       Display XML-StatusText
                   End-Perform
               End-If.

    Tom Morrison, Consultant, South Carolina, USA
    Stephen Hjerpe
    • Stephen Hjerpe
    • Participating Frequently
    • December 8, 2015

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    You might consider using the DECLARATIVES to catch the WebClient .Net error ... USE AFTER STANDARD {EXCEPTION} PROCEDURE ON OBJECT

    {ERROR    }

    D

    I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

    Couple questions:

    1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

    2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

    Any guidance would be appreciate.

    Thanks so much for the direction on capturing the errors through declaratives, now we can at least capture the error cleanly without crashing the runtime.  I appreciate the help.

    Recent badge winners

    Explore all badges
    Terms & ConditionsAccessibility statement