All,

     So we have been battling some extreme system slowness intermittently for years. When this comes up we max out on LCT's, the AIX kernel goes into a panic causing thousands of processes to back up in CPU wait. In  our case it was always nfauser. 

     We were able to eventually track it back to some infrastructure we had implemented for opening files. Some of the code was basically opening up an NFA  file pointer for each record selected in a loop, so if it was thousands you got thousands of opens and thousands of remote nfauser sessions. We found it and changed it to open once and seemed to fix it for a while. Then it was still happening intermittently. With the previously mentioned bug we saw this on 3 of 4 production systems, once fixed only on our heaviest nfa accessed data server.

   What we were able to determine with help from Rocket support was that upon log off UniData spawns a processes that scans /usr/ud/tmp, savedlists, and SAVEDLISTS to cleanup any transient work files left behind. While I was aware that /usr/ud/tmp was cleaned up, we did not know that savedlists and SAVEDLISTS were also. So, we had a SAVEDLISTS file in our main production account that had 350k files in it. So when UniData tried to scan that x hundreds and thousands of nfauser processes running normally, it would back up the processes table and throw the AIX kernel into a panic. We had to bring down UniData to clear it and it would come right back. We have put in an engineering request to have a switch in UniData to stop the scans of savedlists and SAVEDLISTS as there is zero reason for that scan to happen. In the meantime we are purging those areas more aggressively and are going to point SAVEDLISTS to a new directory in the VOC and leave the original SAVEDLISTS empty.  

     If any of you are experiencing similar performance degradation and cannot find out why, check your SAVEDLISTS files.