One of our applications received this error message and aborted. The remote data was never offline.
UniData Release 8.2 Build: (1000) (c) Copyright Rocket Software, Inc. 1985-2020. All rights reserved. Current UniData home is /usr/ud/. Current working directory is /dock/app1/PLT3. :perform_remexec dp:KMC 870 lblack [1/1] SELECT UNIT.ID SELECT UNIT.ID 1311 records selected to list 0. In /usr/ud/sys/CTLG/p/perform_remexec at line 225, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 225, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 225 Connect dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 225 efs_connect() failed, fname (SAVEDLISTS)In /usr/ud/sys/CTLG/p/perform_remexec at line 228, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 228, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 228 Conne!
ct dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 228 efs_connect() failed, fname (SAVEDLISTS)CAN NOT OPEN U.DP.KMC.SAVEDLISTS remote string - executed by perform_remexec
Anyone ever see this before?
------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------
This Unidata 8.2.2 on AIX 7.1
------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------
One of our applications received this error message and aborted. The remote data was never offline.
UniData Release 8.2 Build: (1000) (c) Copyright Rocket Software, Inc. 1985-2020. All rights reserved. Current UniData home is /usr/ud/. Current working directory is /dock/app1/PLT3. :perform_remexec dp:KMC 870 lblack [1/1] SELECT UNIT.ID SELECT UNIT.ID 1311 records selected to list 0. In /usr/ud/sys/CTLG/p/perform_remexec at line 225, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 225, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 225 Connect dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 225 efs_connect() failed, fname (SAVEDLISTS)In /usr/ud/sys/CTLG/p/perform_remexec at line 228, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 228, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 228 Conne!
ct dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 228 efs_connect() failed, fname (SAVEDLISTS)CAN NOT OPEN U.DP.KMC.SAVEDLISTS remote string - executed by perform_remexec
Anyone ever see this before?
------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------
Also this error when accessing an NFA pointer:
Can not open NFA file - U.DP.KMC.LBDAT. Retrying...Seems to work fine after a short wait.
------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------
One of our applications received this error message and aborted. The remote data was never offline.
UniData Release 8.2 Build: (1000) (c) Copyright Rocket Software, Inc. 1985-2020. All rights reserved. Current UniData home is /usr/ud/. Current working directory is /dock/app1/PLT3. :perform_remexec dp:KMC 870 lblack [1/1] SELECT UNIT.ID SELECT UNIT.ID 1311 records selected to list 0. In /usr/ud/sys/CTLG/p/perform_remexec at line 225, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 225, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 225 Connect dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 225 efs_connect() failed, fname (SAVEDLISTS)In /usr/ud/sys/CTLG/p/perform_remexec at line 228, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 228, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 228 Conne!
ct dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 228 efs_connect() failed, fname (SAVEDLISTS)CAN NOT OPEN U.DP.KMC.SAVEDLISTS remote string - executed by perform_remexec
Anyone ever see this before?
------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------
Here is a command you can use to check sockets in a CLOSE_WAIT state. It appears that when we have this issue we have a high number of these, over 60 thousand. This is on AIX. Can any of you run this on your systems and see what it returns?
netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT
Here is a simple ksh script that will identify these and track them back to the active process id:
#!/bin/ksh
#!set -x
>netstat.txt
netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT | awk '{print $1}' > netstat.txt
# file name
file=/usr/local/bin/sysutil/netstat.txt
while read netstat_id
do
#print "Next $netstat_id"
#echo $netstat_id
for PID in $(rmsock $netstat_id tcpcb | awk '{print $9}')
do
ps -ef | grep -v grep | grep $PID | awk '{print $1,$2,$10}'
done
done <"$file"
"CLOSE_WAIT.sh" 31 lines, 737 characters
root@mis:/usr/local/bin/sysutil> CLOSE_WAIT.sh
phanproc 19202262 PHANTOM
------------------------------
Tom Vankirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH US
------------------------------
Here is a command you can use to check sockets in a CLOSE_WAIT state. It appears that when we have this issue we have a high number of these, over 60 thousand. This is on AIX. Can any of you run this on your systems and see what it returns?
netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT
Here is a simple ksh script that will identify these and track them back to the active process id:
#!/bin/ksh
#!set -x
>netstat.txt
netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT | awk '{print $1}' > netstat.txt
# file name
file=/usr/local/bin/sysutil/netstat.txt
while read netstat_id
do
#print "Next $netstat_id"
#echo $netstat_id
for PID in $(rmsock $netstat_id tcpcb | awk '{print $9}')
do
ps -ef | grep -v grep | grep $PID | awk '{print $1,$2,$10}'
done
done <"$file"
"CLOSE_WAIT.sh" 31 lines, 737 characters
root@mis:/usr/local/bin/sysutil> CLOSE_WAIT.sh
phanproc 19202262 PHANTOM
------------------------------
Tom Vankirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH US
------------------------------
Also worth checking on TIME_WAIT, FIN_WAIT and FIN_WAIT2.
The TIME_WAIT timeout can be an issue if a large number of socket opens and closes occur in rapid succession though the cleanup interval can be tuned on the NIC or (for Windows) in the registry. The total number of available socket handles can also be tuned of really needed..
That they exist and are transient is not an issue. If there are a large number and persist is however an issue,
The most frequent issue I see is related to TIME_WAIT.
Regards
------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------
Also worth checking on TIME_WAIT, FIN_WAIT and FIN_WAIT2.
The TIME_WAIT timeout can be an issue if a large number of socket opens and closes occur in rapid succession though the cleanup interval can be tuned on the NIC or (for Windows) in the registry. The total number of available socket handles can also be tuned of really needed..
That they exist and are transient is not an issue. If there are a large number and persist is however an issue,
The most frequent issue I see is related to TIME_WAIT.
Regards
------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------
John,
We are on AIX. Are these Windows settings? Where would we see these and set them? How would we know these are an issue exactly?
------------------------------
Tom Vankirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH US
------------------------------
John,
We are on AIX. Are these Windows settings? Where would we see these and set them? How would we know these are an issue exactly?
------------------------------
Tom Vankirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH US
------------------------------
Tom,
Here are a few useful articles on TIME_WAIT and how to deal with it for LINUX (they are equally applicable to AIX except on specific detail)
and a further article which is AIX specific:
As a general guide, if TIME_WAIT is an issue then reduce the timeout and consider increasing the number of available TCP connections. It can be equally important to check the TCP client side as well:
As a closing point, DNS lookup failures can increase the number of connections going into this state. Please check that the DNS server is not overloaded and/or periodically unavailable for brief intervals.
Regards,
JJ
------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------
