Here is a command you can use to check sockets in a CLOSE_WAIT state. It appears that when we have this issue we have a high number of these, over 60 thousand. This is on AIX. Can any of you run this on your systems and see what it returns?
netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT
Here is a simple ksh script that will identify these and track them back to the active process id:
#!/bin/ksh
#!set -x
>netstat.txt
netstat -Aan |grep -v 127.0.0.1|grep CLOSE_WAIT | awk '{print $1}' > netstat.txt
# file name
file=/usr/local/bin/sysutil/netstat.txt
while read netstat_id
do
#print "Next $netstat_id"
#echo $netstat_id
for PID in $(rmsock $netstat_id tcpcb | awk '{print $9}')
do
ps -ef | grep -v grep | grep $PID | awk '{print $1,$2,$10}'
done
done <"$file"
"CLOSE_WAIT.sh" 31 lines, 737 characters
root@mis:/usr/local/bin/sysutil> CLOSE_WAIT.sh
phanproc 19202262 PHANTOM
------------------------------
Tom Vankirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH US
------------------------------
Original Message:
Sent: 10-13-2021 12:13
From: Tom Vankirk
Subject: Resource temporarily unavailable in UDT_connect
One of our applications received this error message and aborted. The remote data was never offline.
UniData Release 8.2 Build: (1000) (c) Copyright Rocket Software, Inc. 1985-2020. All rights reserved. Current UniData home is /usr/ud/. Current working directory is /dock/app1/PLT3. :perform_remexec dp:KMC 870 lblack [1/1] SELECT UNIT.ID SELECT UNIT.ID 1311 records selected to list 0. In /usr/ud/sys/CTLG/p/perform_remexec at line 225, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 225, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 225 Connect dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 225 efs_connect() failed, fname (SAVEDLISTS)In /usr/ud/sys/CTLG/p/perform_remexec at line 228, connect() error(11): Resource temporarily unavailable In /usr/ud/sys/CTLG/p/perform_remexec at line 228, Connect() error (11): Resource temporarily unavailable in UDT_connect. In /usr/ud/sys/CTLG/p/perform_remexec at line 228 Conne!
ct dp:/dp/apps/KMC:1155 failed, status 11 In /usr/ud/sys/CTLG/p/perform_remexec at line 228 efs_connect() failed, fname (SAVEDLISTS)CAN NOT OPEN U.DP.KMC.SAVEDLISTS remote string - executed by perform_remexec
Anyone ever see this before?
------------------------------
Tom VanKirk
Unix admin
Cabinetworks Group Michigan, LLC
Garrettsville OH United States
------------------------------