Resource temporarily unavailable in UDT_connect

Thomas VanKirk
Author
Participating Frequently
Forum|Forum|4 years ago
October 25, 2021

One of our applications received this error message and aborted. The remote data was never offline.

UniData Release 8.2 Build: (1000) (c) Copyright Rocket Software, Inc. 1985-2020. All rights reserved. Current UniData home is /usr/ud/. Current working directory is /dock/app1/PLT3. :perform_remexec dp:KMC 870 lblack [1/1] SELECT UNIT.ID SELECT UNIT.ID 1311 records selected to list 0. In /usr/ud/sys/CTLG/p/perform_remexec at line 225, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 225, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 225 Connect dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 225 efs_connect() failed, fname (SAVEDLISTS)In /usr/ud/sys/CTLG/p/perform_remexec at line 228, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 228, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 228 Conne!

ct dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 228 efs_connect() failed, fname (SAVEDLISTS)CAN NOT OPEN U.DP.KMC.SAVEDLISTS remote string - executed by perform_remexec

Anyone ever see this before?

------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------

This Unidata 8.2.2 on AIX 7.1

------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------

Like

T

Thomas VanKirk
Author
Participating Frequently
Forum|Forum|4 years ago
October 25, 2021

One of our applications received this error message and aborted. The remote data was never offline.

UniData Release 8.2 Build: (1000) (c) Copyright Rocket Software, Inc. 1985-2020. All rights reserved. Current UniData home is /usr/ud/. Current working directory is /dock/app1/PLT3. :perform_remexec dp:KMC 870 lblack [1/1] SELECT UNIT.ID SELECT UNIT.ID 1311 records selected to list 0. In /usr/ud/sys/CTLG/p/perform_remexec at line 225, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 225, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 225 Connect dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 225 efs_connect() failed, fname (SAVEDLISTS)In /usr/ud/sys/CTLG/p/perform_remexec at line 228, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 228, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 228 Conne!

ct dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 228 efs_connect() failed, fname (SAVEDLISTS)CAN NOT OPEN U.DP.KMC.SAVEDLISTS remote string - executed by perform_remexec

Anyone ever see this before?

------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------

Also this error when accessing an NFA pointer:

Can not open NFA file - U.DP.KMC.LBDAT. Retrying...

Seems to work fine after a short wait.

------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------

Like

T

Thomas VanKirk
Author
Participating Frequently
Forum|Forum|2 years ago
February 8, 2023

One of our applications received this error message and aborted. The remote data was never offline.

UniData Release 8.2 Build: (1000) (c) Copyright Rocket Software, Inc. 1985-2020. All rights reserved. Current UniData home is /usr/ud/. Current working directory is /dock/app1/PLT3. :perform_remexec dp:KMC 870 lblack [1/1] SELECT UNIT.ID SELECT UNIT.ID 1311 records selected to list 0. In /usr/ud/sys/CTLG/p/perform_remexec at line 225, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 225, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 225 Connect dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 225 efs_connect() failed, fname (SAVEDLISTS)In /usr/ud/sys/CTLG/p/perform_remexec at line 228, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 228, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 228 Conne!

ct dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 228 efs_connect() failed, fname (SAVEDLISTS)CAN NOT OPEN U.DP.KMC.SAVEDLISTS remote string - executed by perform_remexec

Anyone ever see this before?

------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------

Here is a command you can use to check sockets in a CLOSE_WAIT state. It appears that when we have this issue we have a high number of these, over 60 thousand. This is on AIX. Can any of you run this on your systems and see what it returns?

netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT

Here is a simple ksh script that will identify these and track them back to the active process id:

#!/bin/ksh
#!set -x
>netstat.txt
netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT | awk '{print $1}' > netstat.txt

# file name

file=/usr/local/bin/sysutil/netstat.txt

while read netstat_id

do

#print "Next $netstat_id"
#echo $netstat_id

for PID in $(rmsock $netstat_id tcpcb | awk '{print $9}')

do

ps -ef | grep -v grep | grep $PID | awk '{print $1,$2,$10}'

done

done <"$file"
"CLOSE_WAIT.sh" 31 lines, 737 characters

root@mis:/usr/local/bin/sysutil> CLOSE_WAIT.sh
phanproc 19202262 PHANTOM

------------------------------
Tom Vankirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH US
------------------------------

Like

+1

John Jenkins
Participating Frequently
Forum|Forum|2 years ago
February 8, 2023

Here is a command you can use to check sockets in a CLOSE_WAIT state. It appears that when we have this issue we have a high number of these, over 60 thousand. This is on AIX. Can any of you run this on your systems and see what it returns?

netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT

Here is a simple ksh script that will identify these and track them back to the active process id:

#!/bin/ksh
#!set -x
>netstat.txt
netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT | awk '{print $1}' > netstat.txt

# file name

file=/usr/local/bin/sysutil/netstat.txt

while read netstat_id

do

#print "Next $netstat_id"
#echo $netstat_id

for PID in $(rmsock $netstat_id tcpcb | awk '{print $9}')

do

ps -ef | grep -v grep | grep $PID | awk '{print $1,$2,$10}'

done

done <"$file"
"CLOSE_WAIT.sh" 31 lines, 737 characters

root@mis:/usr/local/bin/sysutil> CLOSE_WAIT.sh
phanproc 19202262 PHANTOM

------------------------------
Tom Vankirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH US
------------------------------

Also worth checking on TIME_WAIT, FIN_WAIT and FIN_WAIT2.

The TIME_WAIT timeout can be an issue if a large number of socket opens and closes occur in rapid succession though the cleanup interval can be tuned on the NIC or (for Windows) in the registry. The total number of available socket handles can also be tuned of really needed..

That they exist and are transient is not an issue. If there are a large number and persist is however an issue,

The most frequent issue I see is related to TIME_WAIT.

Regards

------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

JJ

Like

T

Thomas VanKirk
Author
Participating Frequently
Forum|Forum|2 years ago
February 9, 2023

Also worth checking on TIME_WAIT, FIN_WAIT and FIN_WAIT2.

The TIME_WAIT timeout can be an issue if a large number of socket opens and closes occur in rapid succession though the cleanup interval can be tuned on the NIC or (for Windows) in the registry. The total number of available socket handles can also be tuned of really needed..

That they exist and are transient is not an issue. If there are a large number and persist is however an issue,

The most frequent issue I see is related to TIME_WAIT.

Regards

------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

John,

We are on AIX. Are these Windows settings? Where would we see these and set them? How would we know these are an issue exactly?

------------------------------
Tom Vankirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH US
------------------------------

Like

+1

John Jenkins
Participating Frequently
Forum|Forum|2 years ago
February 10, 2023

John,

We are on AIX. Are these Windows settings? Where would we see these and set them? How would we know these are an issue exactly?

------------------------------
Tom Vankirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH US
------------------------------

Tom,

Here are a few useful articles on TIME_WAIT and how to deal with it for LINUX (they are equally applicable to AIX except on specific detail)

and a further article which is AIX specific:

systems-tuning-aix

As a general guide, if TIME_WAIT is an issue then reduce the timeout and consider increasing the number of available TCP connections. It can be equally important to check the TCP client side as well:

planning-tcpip-settings

As a closing point, DNS lookup failures can increase the number of connections going into this state. Please check that the DNS server is not overloaded and/or periodically unavailable for brief intervals.

Regards,

JJ

------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

JJ