How to determine when connection pools are overly busy

We utilize 4 connection pools used by our Web Service calls.  Occasionally we have had web services timing out and not even reaching our U2 programs.  I was hoping for some ideas on how we might  check the connection pools to determine when they are overly busy, or calls are backing up/timing out.  We have not found a way to determine this.  Our web team contacts us when their calls are failing and we must dig around looking for a problem. 

Hoping there might be some "tools" that would specifically target the health of the connection pools.

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

Hi Nelson,

Did you get any responses to this post? This is also something that we would be interest in.

Regards,
Barry

------------------------------
Barry Green
Director
CAPITOL BUSINESS SERVICES
Mawson Lakes SA AU
------------------------------

Hi Nelson,

Did you get any responses to this post? This is also something that we would be interest in.

Regards,
Barry

------------------------------
Barry Green
Director
CAPITOL BUSINESS SERVICES
Mawson Lakes SA AU
------------------------------

Sorry Barry - nothing but crickets!

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

Sorry Barry - nothing but crickets!

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

Hi Nelson,

That's disappointing.

@Rocket Software have you no comments or suggestions for this?



​

------------------------------
Barry Green
Director
CAPITOL BUSINESS SERVICES
Mawson Lakes SA AU
------------------------------

We utilize 4 connection pools used by our Web Service calls.  Occasionally we have had web services timing out and not even reaching our U2 programs.  I was hoping for some ideas on how we might  check the connection pools to determine when they are overly busy, or calls are backing up/timing out.  We have not found a way to determine this.  Our web team contacts us when their calls are failing and we must dig around looking for a problem. 

Hoping there might be some "tools" that would specifically target the health of the connection pools.

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

Hi Nelson,

I like your "tools" suggestion and think it'd be a fantastic addition.

In a *nix environment there are tools such as topas, nmon and vmstat utilities that indicate how much resources a process is using and could be an assist in determining "load". So in short, your system admin team may be able to assist here?

We also accasionally have similar issues and find that restarting unirpcd and killing off hung child processes usually fixes it. We're running AIX 7.2 with an old unsupported uv10.3.4 therefore no chance of us getting some additional UV "toolkit". So we resort to operating system monitoring tools and utilities to get us through until we can (one day) upgrade to a supported UV version.

Regards,
Peter

------------------------------
Peter Cheney
Developer
Rocket Forum Shared Account
BRISBANE QLD AU
------------------------------

We utilize 4 connection pools used by our Web Service calls.  Occasionally we have had web services timing out and not even reaching our U2 programs.  I was hoping for some ideas on how we might  check the connection pools to determine when they are overly busy, or calls are backing up/timing out.  We have not found a way to determine this.  Our web team contacts us when their calls are failing and we must dig around looking for a problem. 

Hoping there might be some "tools" that would specifically target the health of the connection pools.

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

Hi Nelson.

Something which could really benefit you in this scenario, would be the MultiValue Integration Server (MVIS) from Rocket:

https://www.rocketsoftware.com/products/rocket-multivalue-integration-server

This is specifically designed to give you greater transparency on what your Connection Pools are doing, and help diagnose issues associated with incoming connections; slow response, unavailable connections, poor UX, license issues etc.  It also improves flexibility in using the connections, better APIs, Swagger definitions, and a whole lot more.

Definitely worth investigating :-)

@Barry - we will reach out to you to discuss this separately from the forum :-)

Neil​​

------------------------------
Neil Tiver
BDM
Meier Business Systems PTY LTD
Carnegie VIC AU
------------------------------

Hi Nelson,

That's disappointing.

@Rocket Software have you no comments or suggestions for this?



​

------------------------------
Barry Green
Director
CAPITOL BUSINESS SERVICES
Mawson Lakes SA AU
------------------------------

Sorry for dropping the ball on this one guys, but I thank @Neil Tiver for jumping in. We have a great customer story about how MVIS  really allowed them to scale effectively: https://www.rocketsoftware.com/customer-case-studies/mainstream-new-zealand

Let me know if I can provide further info.

​All the best,
Dave

------------------------------
David Andrews
Head of Customer Advocacy
Rocket Software
South Salem NY US
------------------------------

Sorry for dropping the ball on this one guys, but I thank @Neil Tiver for jumping in. We have a great customer story about how MVIS  really allowed them to scale effectively: https://www.rocketsoftware.com/customer-case-studies/mainstream-new-zealand

Let me know if I can provide further info.

​All the best,
Dave

------------------------------
David Andrews
Head of Customer Advocacy
Rocket Software
South Salem NY US
------------------------------

So, the bottom line is that there is nothing in U2 to provide any insight into CPs.  Upgrading to MVIS seems like a lot of changes to get to something that I had hoped would be available with a lot less effort.
Disappointing.

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

So, the bottom line is that there is nothing in U2 to provide any insight into CPs.  Upgrading to MVIS seems like a lot of changes to get to something that I had hoped would be available with a lot less effort.
Disappointing.

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

If you're not using it already you may want to look into F5 load balancing.   It's something you'll need if heavy traffic is anticipated.

Karl

------------------------------
Karl Pozmann
Sr. Analyst, Developer, LTC Systems
Omnicare Inc
Dublin OH US
------------------------------

So, the bottom line is that there is nothing in U2 to provide any insight into CPs.  Upgrading to MVIS seems like a lot of changes to get to something that I had hoped would be available with a lot less effort.
Disappointing.

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

Hi Dave and Rocket,

My MVIS $0.02 worth.

I think MVIS is a fantastic tool but in our environment, we need end to end logging of online customer activity.

Maybe I've missed something here but I have yet to find a suitable MVIS logging solution for our needs. To get anything close to useful we need to log everything (trace level) set but then there is so much information it becomes nigh impossible to find the information required. While this is sort of OK in a development environment, it is far from acceptable in a production environment where much lower "info" level of logging is desirable to both reduce the sheer volume of log data and also optimise server performance.

This lower-level logging does not provide sufficient resource information such as the account/end point names and token information that can be used to identify activity from customer browser through to the UniVerse account being accessed. This may not be as big an issue in the white paper FMS example but I believe it is critical in financial transactions, particularly when investigating historical issues, enquiries, and complaints that can extend back in time for months and even years in some cases.

It would be really good I think to be able to nominate certain key fields per end point, and whether these fields can be clear text or masked. Perhaps in a "log group" and be able to allocate the "group" to a particular level of MVIS logging such as "info" but able to choose error, warn, info, debug, trace as required.

Any thoughts or recommendations?

Cheers,
Peter
------------------------------------------------------------------------------- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600)

Hi Dave and Rocket,

My MVIS $0.02 worth.

I think MVIS is a fantastic tool but in our environment, we need end to end logging of online customer activity.

Maybe I've missed something here but I have yet to find a suitable MVIS logging solution for our needs. To get anything close to useful we need to log everything (trace level) set but then there is so much information it becomes nigh impossible to find the information required. While this is sort of OK in a development environment, it is far from acceptable in a production environment where much lower "info" level of logging is desirable to both reduce the sheer volume of log data and also optimise server performance.

This lower-level logging does not provide sufficient resource information such as the account/end point names and token information that can be used to identify activity from customer browser through to the UniVerse account being accessed. This may not be as big an issue in the white paper FMS example but I believe it is critical in financial transactions, particularly when investigating historical issues, enquiries, and complaints that can extend back in time for months and even years in some cases.

It would be really good I think to be able to nominate certain key fields per end point, and whether these fields can be clear text or masked. Perhaps in a "log group" and be able to allocate the "group" to a particular level of MVIS logging such as "info" but able to choose error, warn, info, debug, trace as required.

Any thoughts or recommendations?

Cheers,
Peter
------------------------------------------------------------------------------- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600)

Peter, I agree with you that the logging in MVIS is very lacking at the moment. The good news is that the next maintenance release 1.3.x, expected at the end of June, will include updated logs with the account name. See attached.

There's also development request MVIS-3348 to include a requestId/transactionId in each log so you'll be able to link mvis logs with the rest of the logs generated by your stack.

------------------------------
Kenneth Thompson
Application Developer
Storis Inc
Mount Arlington NJ US
------------------------------

Peter, I agree with you that the logging in MVIS is very lacking at the moment. The good news is that the next maintenance release 1.3.x, expected at the end of June, will include updated logs with the account name. See attached.

There's also development request MVIS-3348 to include a requestId/transactionId in each log so you'll be able to link mvis logs with the rest of the logs generated by your stack.

------------------------------
Kenneth Thompson
Application Developer
Storis Inc
Mount Arlington NJ US
------------------------------

Just to clarify MVIS-3348 is not part of the MVIS 1.3.1 release

------------------------------
Stuart MacKenzie
Rocket Software Inc
CO US
------------------------------

We utilize 4 connection pools used by our Web Service calls.  Occasionally we have had web services timing out and not even reaching our U2 programs.  I was hoping for some ideas on how we might  check the connection pools to determine when they are overly busy, or calls are backing up/timing out.  We have not found a way to determine this.  Our web team contacts us when their calls are failing and we must dig around looking for a problem. 

Hoping there might be some "tools" that would specifically target the health of the connection pools.

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

A couple of things on this thread:
1. MVIS-3348 to include a requestId/transactionId in each log. - Has been accepted for inclusion in MVIS-1.3.2 towards the end of the year
2. The engineering team are currently starting an investigation on how we can best monitor connection pools and what information would be most valuable to customers. The results of this investigation will be included in a future quarterly release of Rocket's MV Experiences product.
MV Experiences is a free product for customers on maintenance and is currently available for UniVerse 11.3.4.
It should be available for UniVerse 12 later this year followed by UniData.

------------------------------
Stuart MacKenzie
Rocket Software Inc
CO US
------------------------------

We utilize 4 connection pools used by our Web Service calls.  Occasionally we have had web services timing out and not even reaching our U2 programs.  I was hoping for some ideas on how we might  check the connection pools to determine when they are overly busy, or calls are backing up/timing out.  We have not found a way to determine this.  Our web team contacts us when their calls are failing and we must dig around looking for a problem. 

Hoping there might be some "tools" that would specifically target the health of the connection pools.

------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

Hi Nelson, I've only just started playing with UniVerse (via uopy) and found the following strategy helpful.  Not using connecting pooling but the principles would be the same:

When I construct a session, there are a couple of attributes on it that are helpful:  there is a "health_check()" function on the session which tries to determine whether it's useful or not, an "is_active" flag to indicate whether it's capable of doing something.  I would assume these are also in the java/.NET versions of the product.

I've also been using a function on the session "start_idle()".  I found in my web service that the connection would drop silently overnight, so most of the time I look at the idle time on the session and if it's been idle more than an hour, I reconstruct it.  The session construction is done via a kind of lazy initialisation so getting a working session does all the health checks and whatnot before passing a working session back to the bit of the program that needs it.

From my very limited experience with this so far, the strategy of assuming the session is dead unless proven otherwise makes for a more reliable web service in my case.

cheers,
dave.

------------------------------
David Rubie
Integration Specialist
Rocket Forum Shared Account
------------------------------