Skip to main content

We are running UV11.3.1 ( Pick Format ) on AIX 7200-05-08-2420 

Yesterday, we started getting reports from the call centers about user's sessions freezing up. Normally, our first thought when this happens is a record lock of some kind. 

If we tried to log into the application, we got to the login prompt and were able to authenticate. But once we successfully entered our userid/password to get into the application the session would hang. 

We create thousands of uniobjects connections to the DB throughout the day from our web sites and IVR, and these were timing out.

Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.[IBM U2][UODOTNET - UNIRPC][ErrorCode=81009] The RPC failed

UniVerse itself was running fine. I could start new sessions and as long as I stayed on the UV account ( bypassing the application login process ), I could move around the system, do TCL queries, etc… 

When I did a LIST.READU EVERY, the system showed no waiters. So it didn't seem to be a situation where everyone was backed up waiting for a specific record to be released. 

We went to the UV account ( logged in as root ) and did an UNLOCK ALL, an UNLOCK GROUPLOCK ALL, pretty much every UNLOCK combination that we could think of. This did not release the hung sessions. 

I'm not sure if this is related, but I also saw record locks on device 0, inode 0. 

Ultimately, we resorted to stopping and restarting UniVerse to free things up and allow the users to resume their normal work.

We encountered a similar, if not the exact same scenario, once previously.

I'm wondering if anyone else has ever run into this? Also wondering if there are other things either in UniVerse or at the Unix level that we might have tried before resorting to the restart.



------------------------------
Anthony Pontecorvo

------------------------------

We are running UV11.3.1 ( Pick Format ) on AIX 7200-05-08-2420 

Yesterday, we started getting reports from the call centers about user's sessions freezing up. Normally, our first thought when this happens is a record lock of some kind. 

If we tried to log into the application, we got to the login prompt and were able to authenticate. But once we successfully entered our userid/password to get into the application the session would hang. 

We create thousands of uniobjects connections to the DB throughout the day from our web sites and IVR, and these were timing out.

Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.[IBM U2][UODOTNET - UNIRPC][ErrorCode=81009] The RPC failed

UniVerse itself was running fine. I could start new sessions and as long as I stayed on the UV account ( bypassing the application login process ), I could move around the system, do TCL queries, etc… 

When I did a LIST.READU EVERY, the system showed no waiters. So it didn't seem to be a situation where everyone was backed up waiting for a specific record to be released. 

We went to the UV account ( logged in as root ) and did an UNLOCK ALL, an UNLOCK GROUPLOCK ALL, pretty much every UNLOCK combination that we could think of. This did not release the hung sessions. 

I'm not sure if this is related, but I also saw record locks on device 0, inode 0. 

Ultimately, we resorted to stopping and restarting UniVerse to free things up and allow the users to resume their normal work.

We encountered a similar, if not the exact same scenario, once previously.

I'm wondering if anyone else has ever run into this? Also wondering if there are other things either in UniVerse or at the Unix level that we might have tried before resorting to the restart.



------------------------------
Anthony Pontecorvo

------------------------------

That should not be possible.  An inode of 0 means a unallocated block on disk, not a legitimate file.  It sounds like something is returning NULL to indicate failure, and it's not being checked properly.



------------------------------
Tilghman Lesher
Senior Developer
Self Registered
WHITES CREEK TN US
------------------------------

We are running UV11.3.1 ( Pick Format ) on AIX 7200-05-08-2420 

Yesterday, we started getting reports from the call centers about user's sessions freezing up. Normally, our first thought when this happens is a record lock of some kind. 

If we tried to log into the application, we got to the login prompt and were able to authenticate. But once we successfully entered our userid/password to get into the application the session would hang. 

We create thousands of uniobjects connections to the DB throughout the day from our web sites and IVR, and these were timing out.

Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.[IBM U2][UODOTNET - UNIRPC][ErrorCode=81009] The RPC failed

UniVerse itself was running fine. I could start new sessions and as long as I stayed on the UV account ( bypassing the application login process ), I could move around the system, do TCL queries, etc… 

When I did a LIST.READU EVERY, the system showed no waiters. So it didn't seem to be a situation where everyone was backed up waiting for a specific record to be released. 

We went to the UV account ( logged in as root ) and did an UNLOCK ALL, an UNLOCK GROUPLOCK ALL, pretty much every UNLOCK combination that we could think of. This did not release the hung sessions. 

I'm not sure if this is related, but I also saw record locks on device 0, inode 0. 

Ultimately, we resorted to stopping and restarting UniVerse to free things up and allow the users to resume their normal work.

We encountered a similar, if not the exact same scenario, once previously.

I'm wondering if anyone else has ever run into this? Also wondering if there are other things either in UniVerse or at the Unix level that we might have tried before resorting to the restart.



------------------------------
Anthony Pontecorvo

------------------------------

Based on your description, one possible cause would be a held Filelock or Grouplock semaphore. The semaphores can be displayed via the 'smat -s' command at the OS level. If a persistent State of '1' shows for a particular semaphore, it is most likely stuck. In normal processing, they should be held for a very short period of time and released. If you find one that is being held, the unlock command could be used to unlock. For example, to clear grouplock semaphore 88, the following command could be issued. 

# unlock GROUPLOCK SEMAPHORE 88
Please wait, semaphore clear may take as long as 15 seconds.
Semaphore cleared.
#

Note, you would only want to run the above command if the semaphore was stuck and users were hanging. When this situation occurs, you can typically see the Collisions and Retries for the stuck semaphore increasing rapidly.

Thanks,

Neil 



------------------------------
Neil Morris
Universe Advanced Technical Support
Rocket Software
------------------------------

Based on your description, one possible cause would be a held Filelock or Grouplock semaphore. The semaphores can be displayed via the 'smat -s' command at the OS level. If a persistent State of '1' shows for a particular semaphore, it is most likely stuck. In normal processing, they should be held for a very short period of time and released. If you find one that is being held, the unlock command could be used to unlock. For example, to clear grouplock semaphore 88, the following command could be issued. 

# unlock GROUPLOCK SEMAPHORE 88
Please wait, semaphore clear may take as long as 15 seconds.
Semaphore cleared.
#

Note, you would only want to run the above command if the semaphore was stuck and users were hanging. When this situation occurs, you can typically see the Collisions and Retries for the stuck semaphore increasing rapidly.

Thanks,

Neil 



------------------------------
Neil Morris
Universe Advanced Technical Support
Rocket Software
------------------------------
Neil,

Thanks very much for your suggestions. If we observe this behavior again we will follow this procedure.

One follow up, does adding the keyword SEMAPHORE cause a difference in the behavior of the UNLOCK verb (As in UNLOCK ALL). Or is specifically targeting the GROUPLOCK significant.

Thank you.


Anthony Pontecorvo
Special Projects Administrator
Fees & Fines Department
Linebarger Goggan Blair & Sampson, LLP
[https://testing.publicans.com/images/Logos/Linebarger_logo_410x99px.png] [https://testing.publicans.com/images/Logos/Phone_35x35.png] Main: (210) 403-8600
[https://testing.publicans.com/images/Logos/Phone_35x35.png] Direct: (210) 403.8716
[https://testing.publicans.com/images/Logos/Email_35x35.png] Anthony.Pontecorvo@lgbs.com
[https://testing.publicans.com/images/Logos/Address_35x35.png] 2915 W. Bitters Rd, Suite 400
San Antonio, TX 78248
[https://testing.publicans.com/images/Logos/Website%201.5_35x35.png] www.lgbs.com


CONFIDENTIALITY STATEMENT
This transmission may be: (1) subject to the Attorney-Client Privilege, (2) an attorney work product, or (3) strictly confidential. If you are not the intended recipient of this message, you may not disclose, print, copy or disseminate this information. If you have received this in error, please reply and notify the sender (only) and delete the message. Unauthorized interception of this e-mail is a violation of federal criminal law.
Neil,

Thanks very much for your suggestions. If we observe this behavior again we will follow this procedure.

One follow up, does adding the keyword SEMAPHORE cause a difference in the behavior of the UNLOCK verb (As in UNLOCK ALL). Or is specifically targeting the GROUPLOCK significant.

Thank you.


Anthony Pontecorvo
Special Projects Administrator
Fees & Fines Department
Linebarger Goggan Blair & Sampson, LLP
[https://testing.publicans.com/images/Logos/Linebarger_logo_410x99px.png] [https://testing.publicans.com/images/Logos/Phone_35x35.png] Main: (210) 403-8600
[https://testing.publicans.com/images/Logos/Phone_35x35.png] Direct: (210) 403.8716
[https://testing.publicans.com/images/Logos/Email_35x35.png] Anthony.Pontecorvo@lgbs.com
[https://testing.publicans.com/images/Logos/Address_35x35.png] 2915 W. Bitters Rd, Suite 400
San Antonio, TX 78248
[https://testing.publicans.com/images/Logos/Website%201.5_35x35.png] www.lgbs.com


CONFIDENTIALITY STATEMENT
This transmission may be: (1) subject to the Attorney-Client Privilege, (2) an attorney work product, or (3) strictly confidential. If you are not the intended recipient of this message, you may not disclose, print, copy or disseminate this information. If you have received this in error, please reply and notify the sender (only) and delete the message. Unauthorized interception of this e-mail is a violation of federal criminal law.

Hi Anthony,

The use of the SEMAPHORE keyword will result in the UNLOCK command targeting a specific semaphore. So in the situation described, it is important to use the syntax "UNLOCK GROUPLOCK SEMAPHORE #" if the 'smat -s' command identifies a persistent group lock semaphore. The semaphore lock is different from the record, group, and file locks displayed by LIST.READU. 

Hope that helps.

Thanks,

Neil



------------------------------
Neil Morris
Universe Advanced Technical Support
Rocket Software
------------------------------
Neil,

Thanks very much for your suggestions. If we observe this behavior again we will follow this procedure.

One follow up, does adding the keyword SEMAPHORE cause a difference in the behavior of the UNLOCK verb (As in UNLOCK ALL). Or is specifically targeting the GROUPLOCK significant.

Thank you.


Anthony Pontecorvo
Special Projects Administrator
Fees & Fines Department
Linebarger Goggan Blair & Sampson, LLP
[https://testing.publicans.com/images/Logos/Linebarger_logo_410x99px.png] [https://testing.publicans.com/images/Logos/Phone_35x35.png] Main: (210) 403-8600
[https://testing.publicans.com/images/Logos/Phone_35x35.png] Direct: (210) 403.8716
[https://testing.publicans.com/images/Logos/Email_35x35.png] Anthony.Pontecorvo@lgbs.com
[https://testing.publicans.com/images/Logos/Address_35x35.png] 2915 W. Bitters Rd, Suite 400
San Antonio, TX 78248
[https://testing.publicans.com/images/Logos/Website%201.5_35x35.png] www.lgbs.com


CONFIDENTIALITY STATEMENT
This transmission may be: (1) subject to the Attorney-Client Privilege, (2) an attorney work product, or (3) strictly confidential. If you are not the intended recipient of this message, you may not disclose, print, copy or disseminate this information. If you have received this in error, please reply and notify the sender (only) and delete the message. Unauthorized interception of this e-mail is a violation of federal criminal law.

Anthony,

It's possible you ran out of a key resource - I also suggest:

  • uvlictool report_lic to look at license usage, and then try unlictool clean_lic -a if the pool looks full (as admin/root).
  • Check if socket resources or message queues have been exhaussted at the OS level. There is a delay before released resources become available and a very high re-use rate can result in insufficient resources. If this is the case then the overheads of more resources are tiny and an increase is readily warranted.
  • As this is AIX then you may be encountering periodic disk buffer purge/disk pacing. If so then I recommend setting j2_npagesPerWriteBehindCluster to zero - see https://www.ibm.com/docs/en/aix/7.2?topic=i-ioo-command for details. Please note that unless made permanent with the ioo -p option, any changes made will be lost on a reboot.
    • NOTE - I routinely recommend this setting for UniData/UniVerse random access in any case. While there are other tunables that can have an effect, this one has leapt out in headlines regarding it's effect at times.

Regards

JJ



------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

Anthony,

It's possible you ran out of a key resource - I also suggest:

  • uvlictool report_lic to look at license usage, and then try unlictool clean_lic -a if the pool looks full (as admin/root).
  • Check if socket resources or message queues have been exhaussted at the OS level. There is a delay before released resources become available and a very high re-use rate can result in insufficient resources. If this is the case then the overheads of more resources are tiny and an increase is readily warranted.
  • As this is AIX then you may be encountering periodic disk buffer purge/disk pacing. If so then I recommend setting j2_npagesPerWriteBehindCluster to zero - see https://www.ibm.com/docs/en/aix/7.2?topic=i-ioo-command for details. Please note that unless made permanent with the ioo -p option, any changes made will be lost on a reboot.
    • NOTE - I routinely recommend this setting for UniData/UniVerse random access in any case. While there are other tunables that can have an effect, this one has leapt out in headlines regarding it's effect at times.

Regards

JJ



------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

For what it's worth, we used to run out of licenses a lot before we started migrating away from Universe. I wrote a python program that executes the uvlictool to clean up any unused licenses.

This is the line from crontab to run it every 1/2 hour:

*/30    *      *   *  *  root  /usr/uv/python/bin/python3 /source/programs/PYTHON.PY/clean_uvlic.py

The program keeps track of the total number of licenses used in a uvlictool_total_cleaned.txt file and writes daily logs to a "uvlictool_cleaned_yyyy-mm-dd._log.txt" file.  We have a library called get_host_paths that return our paths. I didn't include that since it doesn't really apply to any of  you.  You can hard code a path to put the those files wherever you want. You can also just comment out the writes.

You might also need to change the path to the uvlictool and python3 since those might be at a different place on your system.

For what it's worth.

clean_uvlic.py

#!/usr/bin/env python3

# 2023-12-14 JEG Wrote program to clean the licenses and keep track of the number
#                of cleaned licenses in the uvlictool_cleaned.txt and uvlictool_total_cleaned.txt
#                files in the purge log.
#                The purpose was to keep track of how many licenses are being abandoned. After
#                writing this program I found another: /etc/cron.daily/uvliccleanup.sh. I moved that
#                file to the /home/jgold folder since it only runs daily and doesn't keep track of
#                anything it does. This routine is currently set to run every 1/2 hour.
#
# 2023-12-26 jeg Put a date on the log file. It's going in PURGE.LOG.7.DAYS but since it's running
#                every half hour, the file is always less than 7 days old and it never gets
#                purged.
#                I also used get_host_paths to get the paths to the log file folder.

import subprocess
from datetime import datetime
import get_host_paths

total_cleaned_path = get_host_paths.get_purge_logs_path() + 'uvlictool_total_cleaned.txt'

current_date = datetime.today()
formatted_date = current_date.strftime("%Y-%m-%d")
log_path = get_host_paths.get_purge_log_path('PURGE.LOG.7.DAYS') + f'uvlictool_cleaned_{formatted_date}_log.txt'

def get_pids():

   in_use = 0
   available = 0
      
   pids = {}
   command = ["/usr/uv/bin/uvlictool"]
   process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
   
   for line in process.stdout:
      line = line.split()
      #print (line)
      
      if len(line) == 2:
         lic_number = line[0]
         pid = line[1]
         pids[pid] = lic_number
      
      # if this is the "xxx license seats are in use." line, get the number
      if len(line) == 6:
         if line[5] == 'use.':
            in_use = line[0]
            
      # if this is the "xxx licnse seats are available." line get that number
      if len(line) == 5:
         if line[4] == 'available.':
            available = line[0]
            
   return (pids, in_use, available)

def get_total_cleaned():

   # Read the current number from the file

   try:
      with open(total_cleaned_path, 'r') as file:
         current_number = int(file.readline().strip())
   except FileNotFoundError:
      current_number = 0  # If the file does not exist, start from 0
   return current_number

def write_total_cleaned(new_number):

   # Overwrite the file with the new number
   with open(total_cleaned_path, 'w') as file:
      file.write(str(new_number))

def append_to_log(line):
   with open(log_path, 'a') as file:
      file.write(f"{line}\\n")

if __name__ == "__main__":

   pids_b4, in_use, available = get_pids()
   #pids_b4['test']='0' # to simulate a difference between the dicts

   command = ["sudo", "/usr/uv/bin/uvlictool","clean_lic","-a"]
   process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)

   # print the output if needed for debugging
   #print ('stdout')
   #for line in process.stdout:
   #   print ('   '+line)   
   #print ('stderr')
   #for line in process.stderr:
   #   print ('   '+line)

   pids_after, in_use, available = get_pids()
   
   unique_keys = set(pids_b4.keys()) - set(pids_after.keys())

   total_cleaned = get_total_cleaned()
   #current_datetime = datetime.now()
   number_cleaned = len(unique_keys)

   #if number_cleaned != 0:
   total_cleaned += number_cleaned
   write_total_cleaned(total_cleaned)

   append_to_log(f"{current_date} Used: {in_use:>4} Available: {available:>4} Cleaned: {number_cleaned:>4} Total Cleaned: {total_cleaned:>4}")



------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US
------------------------------
Terms & ConditionsAccessibility statement