We run Windows D3, and utilise MVS Toolkit extensively. It's a very stable and useful product, but there is always a three-second timespan between the request and response within the server (shown in the log).
Does anyone know of a way to reduce this time, as it can make things quite slow when numerous requests are made? It doesn't seem to be a network-related issue.
Many thanks in advance.

------------------------------
Simon Hawkins
General Manager
Queanbeyan Leagues Club
Queanbeyan NSW AU
------------------------------

Connection Pooling. It's a separate license type in D3. Not only makes the Toolkit faster, but also allows limiting the number of concurrent connections to D3, thus making sure you do not create a MAXUSERS condition. Who's your sales rep? They can get you a quote. You can test this using an D3 evaluation system ID to see how this works. I can help you with that if you're interested.

------------------------------
Brian S. Cram
Principal Technical Support Engineer
Rocket Software
------------------------------

Thank you for this information Brian.
A little further information: The way we use these API's is for internal (to our organisation/domains) updating purposes - not public/external access. So typically there would be hundreds of calls consecutively, but from the same user/machine/VS programme.
Just checking whether this Connection Pooling would benefit our scenario? It sounds to me as though it would.
In which case I would love to test it via a D3 evaluation system ID. Do I get in touch with the MBS boys in Melbourne to do so?

------------------------------
Simon Hawkins
General Manager
Queanbeyan Leagues Club
Queanbeyan NSW AU
------------------------------

Yes. And if you get Mike Raffaele, tell him I said "hello". Thanks.

------------------------------
Brian S. Cram
Principal Technical Support Engineer
Rocket Software
------------------------------

Thank you for this information Brian.
A little further information: The way we use these API's is for internal (to our organisation/domains) updating purposes - not public/external access. So typically there would be hundreds of calls consecutively, but from the same user/machine/VS programme.
Just checking whether this Connection Pooling would benefit our scenario? It sounds to me as though it would.
In which case I would love to test it via a D3 evaluation system ID. Do I get in touch with the MBS boys in Melbourne to do so?

------------------------------
Simon Hawkins
General Manager
Queanbeyan Leagues Club
Queanbeyan NSW AU
------------------------------

The technical answer for the delay is that you have to open a connection to the database for every call. I found in Universe/Unidata that the delay was about 500 to 700 milliseconds. Rocket's connection pooling keeps the connection open so that access is about 10 to 50 milliseconds. However, since there is no traffic cop to make sure the connection is not dropped you might lose any time during the day or night which means you will be back up to your 2 or 3 seconds of delay.

Also, remember whether you are calling a subroutine or doing a query you might have a delay in that routine as well. It is best to test the routine by itself so you know how long it takes it to do its thing.

Posted 07-14-2022 19:56

------------------------------
Doug Averch
------------------------------

We run Windows D3, and utilise MVS Toolkit extensively. It's a very stable and useful product, but there is always a three-second timespan between the request and response within the server (shown in the log).
Does anyone know of a way to reduce this time, as it can make things quite slow when numerous requests are made? It doesn't seem to be a network-related issue.
Many thanks in advance.

------------------------------
Simon Hawkins
General Manager
Queanbeyan Leagues Club
Queanbeyan NSW AU
------------------------------

We have trialled Connection Pooling and it made a huge difference. Where we used to be able to see the load bar move across the screen, it became instantaneous.

I would recommend it.
--
Alex Polglaze
The Book-Keeping Network
(08) 9349 9189
+61 419 776 348
apolglaze@book-keepingnetwork.com.au
https://www.book-keepingnetwork.com.au

Yes. And if you get Mike Raffaele, tell him I said "hello". Thanks.

------------------------------
Brian S. Cram
Principal Technical Support Engineer
Rocket Software
------------------------------

Is the http error 500 a symptom of the same problem? We send a couple requests from the same device and it seems only one works and the rest gets error 500.

------------------------------
Chris Wolcz
Senior Software Developer
Execontrol Global Solutions
Clifton Park NY US
------------------------------

The technical answer for the delay is that you have to open a connection to the database for every call. I found in Universe/Unidata that the delay was about 500 to 700 milliseconds. Rocket's connection pooling keeps the connection open so that access is about 10 to 50 milliseconds. However, since there is no traffic cop to make sure the connection is not dropped you might lose any time during the day or night which means you will be back up to your 2 or 3 seconds of delay.

Also, remember whether you are calling a subroutine or doing a query you might have a delay in that routine as well. It is best to test the routine by itself so you know how long it takes it to do its thing.

Posted 07-14-2022 19:56

------------------------------
Doug Averch
------------------------------

Doug, thank you, that's a really good answer. It should be noted that the MVS Toolkit does in fact have a "traffic cop" to monitor the connection pool connection and reconnect if dropped.

------------------------------
Brian S. Cram
Principal Technical Support Engineer
Rocket Software
------------------------------

Is the http error 500 a symptom of the same problem? We send a couple requests from the same device and it seems only one works and the rest gets error 500.

------------------------------
Chris Wolcz
Senior Software Developer
Execontrol Global Solutions
Clifton Park NY US
------------------------------

Sorry Chris - been a bit busy.
No - I don't receive any errors at all. The problem for me is purely the amount of time each call takes.

------------------------------
Simon Hawkins
Group General Manager
Queanbeyan Leagues Club
Queanbeyan NSW AU
------------------------------

The technical answer for the delay is that you have to open a connection to the database for every call. I found in Universe/Unidata that the delay was about 500 to 700 milliseconds. Rocket's connection pooling keeps the connection open so that access is about 10 to 50 milliseconds. However, since there is no traffic cop to make sure the connection is not dropped you might lose any time during the day or night which means you will be back up to your 2 or 3 seconds of delay.

Also, remember whether you are calling a subroutine or doing a query you might have a delay in that routine as well. It is best to test the routine by itself so you know how long it takes it to do its thing.

Posted 07-14-2022 19:56

------------------------------
Doug Averch
------------------------------

Thanks Doug
The subroutines I use are just momentary file updates, so I'm pretty sure it's just the current necessity to open a new connection each time that's causing the delay. Also for my purposes, an occasional re-connection wouldn't be a problem - it's just the cumulation of hundreds of calls at three seconds each that slows things down.
So I will test connection pooling as per Brian's advice and Alex's recommendation.

------------------------------
Simon Hawkins
Group General Manager
Queanbeyan Leagues Club
Queanbeyan NSW AU
------------------------------

We run Windows D3, and utilise MVS Toolkit extensively. It's a very stable and useful product, but there is always a three-second timespan between the request and response within the server (shown in the log).
Does anyone know of a way to reduce this time, as it can make things quite slow when numerous requests are made? It doesn't seem to be a network-related issue.
Many thanks in advance.

------------------------------
Simon Hawkins
General Manager
Queanbeyan Leagues Club
Queanbeyan NSW AU
------------------------------

If using Connection Pooling it is important to make the server code transactional and stateless as it will not have any 'memory' of prior usage. The rules are:

Do not have any expectations on first invocation - the connection pool server process will be a 'tabula rasa' - i.e. blank.
Leave the connection pool server process in the state in which anyone else would want to find it, leaving nothing behind - e.g. record locks.
If you need to have commonly re-used shared objects such as file handles that are opened once and once only and the kept open then put these in specific COMMON block variables and use program logic and matching boolean flags in COMMON so that they are opened once and once only. These will persist on the connection pooling server process and can be re-used by subsequent invocations. You can use one of two approaches here:
- Open all files on first invocation in case needed - this simplifies logic a little but at the expense of a potentially significant delay if a lot of files are opened in one fell swoop.
- Only open a file as it is needed by the application - this is generally a better approach if not all calls to a connection pooling process require all the files.

Connection Pooling is a useful feature, though be aware that the client-server relationship is no longer based on a dedicated server process and a one-to-one relationship with a specific client process, but on a many-to-many freely shareable process basis.

Regards

JJ

------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

If using Connection Pooling it is important to make the server code transactional and stateless as it will not have any 'memory' of prior usage. The rules are:

Do not have any expectations on first invocation - the connection pool server process will be a 'tabula rasa' - i.e. blank.
Leave the connection pool server process in the state in which anyone else would want to find it, leaving nothing behind - e.g. record locks.
If you need to have commonly re-used shared objects such as file handles that are opened once and once only and the kept open then put these in specific COMMON block variables and use program logic and matching boolean flags in COMMON so that they are opened once and once only. These will persist on the connection pooling server process and can be re-used by subsequent invocations. You can use one of two approaches here:
- Open all files on first invocation in case needed - this simplifies logic a little but at the expense of a potentially significant delay if a lot of files are opened in one fell swoop.
- Only open a file as it is needed by the application - this is generally a better approach if not all calls to a connection pooling process require all the files.

Connection Pooling is a useful feature, though be aware that the client-server relationship is no longer based on a dedicated server process and a one-to-one relationship with a specific client process, but on a many-to-many freely shareable process basis.

Regards

JJ

------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

Just by way of closure, I did test using a Connection Pooling temporary license (which took a while, as I ended up having to whitelist Rocket's DNS hostname to enable Deactivation/Activation).
Testing a batch of 110 consecutive calls, it took just a few seconds (after waiting three seconds for the initial connection). This would have taken more than five minutes with the original non-CP license. So I was very pleased with the result.
I had anticipated upgrading all three of our sites (D3 Servers) if I was happy with the results, but these CP licenses aren't cheap. So I've purchased for just one site, and am now using Super Q Pointers for communication between servers, which for our purposes is a perfectly good workaround for a more-economical outcome.
Thank you everybody for your assistance on this topic - all of your comments and suggestions were very helpful.

------------------------------
Simon Hawkins
Group General Manager
Canberra Raiders Group
Queanbeyan NSW AU
------------------------------

Be careful with the Super-Q-pointer thing. If the process using the connection pool trips on a network issue while trying to read a file on another server, the process could stall and the traffic will back up on that pipe. Remember, Super Q pointers are a good thing, but only as stable as the network connection between two machines.

------------------------------
Brian S. Cram
Principal Technical Support Engineer
Rocket Software
------------------------------

It's a good point Brian, thank you. Pick BASIC is the only language I use which (as far as I know) doesn't have an "on error ...." or equivalent option.
Our three D3 servers are geographically spread out, but fortunately the communications within our domain are quite stable.
I will test the outcome by physically unplugging one of the remote D3 server's ethernet connection to see what happens.

------------------------------
[Simon] [Hawkins]
[Group General Manager]
[Canberra Raiders Group]
[Canberra] [ACT] [Australia]
------------------------------

Actually, it does. Here's part of a document I gave to someone doing OSFI stuff between servers:

Recommendations

OSFI programming:

As far as adding resiliency to the existing OSFI programming techniques, only a few changes need to be made:

Any program (foreground or phantom) that opens OSFI files needs to have the following statements executed before opening, reading or writing OSFI files:
- EXECUTE \\NET-ERRORS (SL\\ CAPTURING JUNK
  - The L options tells the system "log RFSE's, don't stop and prompt the user for intervention, don't drop to debugger".
  - The S option suppresses echoing of the NET-ERRORS state, probably redundant with CAPTURING JUNK clause.
- EXECUTE \\SET-REMOTE-CLOSE (N\\ CAPTURING JUNK
  - Normally when remote files are opened, the file handle is cached by D3 and never explicitly closed, even by a CLOSE statement. This was done to enhance throughput. Setting remote close on allows explicit closing of remote files so that cached handles for no-longer-available resources do not linger to cause problems, as in a temporary network problem.
- Any program accessing OSFI files should keep them open for as narrow a window as practically dictated by throughput need. Ideally when a single record is going to be read and then written, the flow should be:
  - Open
  - Read and/or Write
  - Close

within the inner loop, not opened at the top of the program and closed at program termination. Exceptions would be when processing a significant number of records in a short period of time where the preferred flow should be:

Open
Read and/or Write all records
Close

within the inner loop.

In programs that open more than one OSFI file, it's a good idea for any OPEN/ELSE logic to close all OSFI files opened in the program to make sure that none get missed. Example:
- OPEN RMTFLE1 TO FILE1 ELSE GOTO EXIT
- OPEN RMTFLE2 TO FILE2 ELSE GOTO EXIT

Then at exit:

CLOSE FILE1
CLOSE FILE2

In the event that the first OPEN was successful and the second OPEN failed, the first file needs to be closed on exit, and it doesn't hurt anything to close a file that was never successfully opened.

All WRITEs performed on OSFI files should use the ON ERROR clause to recover from a failed write.
Any ELSE clause on a failed OSFI OPEN or ON ERROR clause on a failed OSFI WRITE should handle a graceful close of all OSFI files and exit from the program, and optionally a wait and retry cycle. To the degree that an ELSE clause on a failed read can be certainly attributed to a network error (and not merely a record that genuinely does not exist) should follow the same cleanup and exit path and optional wait and retry cycle.

------------------------------
Brian S. Cram
Principal Technical Support Engineer
Rocket Software
------------------------------

Actually, it does. Here's part of a document I gave to someone doing OSFI stuff between servers:

Recommendations

OSFI programming:

As far as adding resiliency to the existing OSFI programming techniques, only a few changes need to be made:

Any program (foreground or phantom) that opens OSFI files needs to have the following statements executed before opening, reading or writing OSFI files:
- EXECUTE \\NET-ERRORS (SL\\ CAPTURING JUNK
  - The L options tells the system "log RFSE's, don't stop and prompt the user for intervention, don't drop to debugger".
  - The S option suppresses echoing of the NET-ERRORS state, probably redundant with CAPTURING JUNK clause.
- EXECUTE \\SET-REMOTE-CLOSE (N\\ CAPTURING JUNK
  - Normally when remote files are opened, the file handle is cached by D3 and never explicitly closed, even by a CLOSE statement. This was done to enhance throughput. Setting remote close on allows explicit closing of remote files so that cached handles for no-longer-available resources do not linger to cause problems, as in a temporary network problem.
- Any program accessing OSFI files should keep them open for as narrow a window as practically dictated by throughput need. Ideally when a single record is going to be read and then written, the flow should be:
  - Open
  - Read and/or Write
  - Close

within the inner loop, not opened at the top of the program and closed at program termination. Exceptions would be when processing a significant number of records in a short period of time where the preferred flow should be:

Open
Read and/or Write all records
Close

within the inner loop.

In programs that open more than one OSFI file, it's a good idea for any OPEN/ELSE logic to close all OSFI files opened in the program to make sure that none get missed. Example:
- OPEN RMTFLE1 TO FILE1 ELSE GOTO EXIT
- OPEN RMTFLE2 TO FILE2 ELSE GOTO EXIT

Then at exit:

CLOSE FILE1
CLOSE FILE2

In the event that the first OPEN was successful and the second OPEN failed, the first file needs to be closed on exit, and it doesn't hurt anything to close a file that was never successfully opened.

All WRITEs performed on OSFI files should use the ON ERROR clause to recover from a failed write.
Any ELSE clause on a failed OSFI OPEN or ON ERROR clause on a failed OSFI WRITE should handle a graceful close of all OSFI files and exit from the program, and optionally a wait and retry cycle. To the degree that an ELSE clause on a failed read can be certainly attributed to a network error (and not merely a record that genuinely does not exist) should follow the same cleanup and exit path and optional wait and retry cycle.

------------------------------
Brian S. Cram
Principal Technical Support Engineer
Rocket Software
------------------------------

Again - great information - thank you Brian.
My previous post bravely asserted "fortunately the communications within our domain are quite stable", which turned out to be a rookie error, as links dropped out on Saturday afternoon, and with the Super Q Pointers still trying to tick away, things blew up in a big way.
So I will incorporate your OSFI suggestions into my programming and then give it a good test.

------------------------------
Simon Hawkins
Group General Manager
Canberra Raiders Group
Canberra ACT Australia
------------------------------

OSFI programming:

OSFI programming:

Sign up

Please log in or register:

Welcome to the Rocket Forum!

Please log in or register:

Scanning file for viruses.

This file cannot be downloaded