Rocket U2 | UniVerse & UniData

 View Only
Expand all | Collapse all

Unidata memory usage very high from 7.3.3 to 8.2.1

  • 1.  Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-16-2021 11:55
    Hello. Has anyone upgraded from Unidata 7.3.3 to 8.2.1 and saw a large increase in overall system memory usage? Especially in an AIX environment?

    We upgraded and went from 50 gig of real memory and 16gb of paging, with 1% paging to currently 130gb real memory  and increased paging to 80gb as a precaution after the system ran out of memory the days after the upgrade. We had to add memory and paging space on the fly after the upgrade. 

    We saw smaller increase on 3 other servers, about 50%,  and the server with 990 users, over 850 logged in routinely, we had the extreme memory usage numbers above.

    Here is our top ten processes currently:

    ps aux | head -1; ps aux | sort -rn +3 | head
    USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
    jgoelz 64815780 0.0 3.0 3687024 3687040 pts/352 A Aug 12 16:16 udt
    sfarren 2490806 0.0 1.0 1464620 1464636 pts/278 A Aug 13 6:24 udt
    rloomis 53412190 0.0 1.0 1021108 1021124 pts/1025 A 06:43:46 7:36 udt
    kbarbe 55247314 0.0 1.0 679120 679136 pts/870 A 06:19:08 2:07 udt
    jgraham 63897812 0.0 1.0 717588 717604 pts/381 A 05:08:10 3:07 udt
    jfisher 65339868 0.0 1.0 914892 914908 pts/542 A 05:39:58 3:11 udt
    gwright 15205330 0.0 1.0 760368 760384 pts/473 A Aug 13 4:52 udt
    gstevens 13828612 0.0 1.0 943908 943924 pts/456 A 05:44:04 4:20 udt
    gellis 54198690 0.0 1.0 1959608 1959624 pts/217 A Aug 13 15:13 udt
    emorris 34275578 0.0 1.0 931280 931296 pts/1408 A 07:28:00 4:55 udt

    Nothing changed in our AIX Unix operating system, hardware or applications.

    ------------------------------
    Tom VanKirk
    Unix admin
    Cabinetworks Group Michigan, LLC
    Garrettsville OH United States
    ------------------------------


  • 2.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 09:18
    Edited by Nik Kesic 08-17-2021 09:18
    Hello Tom.

    We upgraded from udt 7.3.7 to 8.2.1 on AIX without seeing a drastic memory usage increase.
    Did you compare the difference between your UniData 7.3.3  "/usr/ud73/include/udtconfig" file to UniData 8.2.1  "/usr/ud82/include/udtconfig" file?


    ------------------------------
    Nik Kesic
    DevOps
    LKQ
    NASHVILLE TN United States
    ------------------------------



  • 3.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 09:28
    Yes, we have gone over all the settings several times with support and no relief. We even downsized some of our shared memory settings to relieve some memory usage and still no better. Noone can tell us why the new heap memory model is so memory intensive for us.




    ------------------------------
    Tom VanKirk
    Unix admin
    Cabinetworks Group Michigan, LLC
    Garrettsville OH United States
    ------------------------------



  • 4.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 09:53
    How many users do you have and do you use System Builder screens?

    ------------------------------
    Tom VanKirk
    Unix admin
    Cabinetworks Group Michigan, LLC
    Garrettsville OH United States
    ------------------------------



  • 5.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 10:00
    Hi Tom

    We have over 5000 SB Client users. 
    Our P980 has 48 dedicated processors with 550GB of memory.

    cheers

    ------------------------------
    Nik Kesic
    DevOps
    LKQ
    NASHVILLE TN United States
    ------------------------------



  • 6.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 10:06
    How many logged into a single server at the same time? We saw this huge increase on a server with about 850 users on at once and to a lesser degree on our other 3 nodes with a few hundred each.

    Did you see any increase at all?

    ------------------------------
    Tom VanKirk
    Unix admin
    Cabinetworks Group Michigan, LLC
    Garrettsville OH United States
    ------------------------------



  • 7.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 10:19
    Tom,

    Currently we have just over 3500 users logged in with a 5412 user license. I have seen 5000 users logged in.

    ------------------------------
    Nik Kesic
    DevOps
    LKQ
    NASHVILLE TN United States
    ------------------------------



  • 8.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-18-2021 09:46
    Nik, can you do me a favor and show your output of " ps aux | head -1; ps aux | sort -rn +3 | head", and what does topas show for memory? Also current user count?

    dock:/ # ps aux | head -1; ps aux | sort -rn +3 | head
    USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
    jharsch 24183506 0.0 2.0 2020884 2016996 pts/136 A Aug 13 13:22 udt
    jfogus 28443070 0.0 2.0 2540340 2540356 pts/885 A Aug 17 5:39 udt
    wmach 42467944 0.0 1.0 1185432 1097896 pts/960 A Aug 13 13:11 udt
    twilson2 48627894 0.0 1.0 955060 955076 pts/1074 A Aug 16 10:08 udt
    thall2 32244462 0.0 1.0 806460 806476 pts/974 A 06:30:26 2:47 udt
    thall2 21365464 0.0 1.0 1169668 1169684 pts/598 A 06:30:58 4:32 udt
    sgardne2 23724732 0.0 1.0 727348 727364 pts/1221 A Aug 17 2:56 udt
    rthomps4 41026260 0.0 1.0 1148660 1148676 pts/668 A Aug 17 10:38 udt
    rmccune 33882278 0.0 1.0 1136100 817460 pts/663 A Aug 13 3:03 udt
    rmccune 24445252 0.0 1.0 886880 870448 pts/674 A Aug 13 2:58 udt

    topas for our server:
    MEMORY
    Real,MB 131072
    % Comp 90
    % Noncomp 9
    % Client 9

    PAGING SPACE
    Size,MB 80896
    % Used 7
    % Free 93

    Licensed(UDT+CP)/Effective Udt Sql iPhtm Pooled Total

    ( 990 + 0 ) / 990 824 0 9 0 833

    ------------------------------
    Tom VanKirk
    Unix admin
    Cabinetworks Group Michigan, LLC
    Garrettsville OH United States
    ------------------------------



  • 9.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-18-2021 10:23
    Hi Tom

    $ ps aux | head -1; ps aux | sort -rn +3 | head
    USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
    zw811438 46138982 0.0 0.0 11584 11600 pts/2040 A 08:41:19 0:00 udt
    zu132398 4394524 0.0 0.0 20200 20216 pts/946 A 07:38:37 0:07 udt
    zr829058 29429664 0.0 0.0 13188 13204 pts/1560 A 09:34:46 0:00 udt
    zm864094 36504056 0.0 0.0 21700 21716 pts/500 A 06:28:38 0:06 udt
    zm130839 8784152 0.0 0.0 26236 26252 pts/2031 A Aug 16 0:09 udt
    zh219422 40109600 0.0 0.0 18116 18132 pts/323 A Aug 17 0:02 udt
    zh219422 30935616 0.0 0.0 17476 17492 pts/1973 A Aug 16 0:04 udt
    zh206290 25626118 0.0 0.0 15940 15956 pts/1851 A 08:55:05 0:00 udt
    ytripone 61539938 0.0 0.0 19912 19928 pts/1688 A 08:09:43 0:06 udt
    ytripone 10161116 0.0 0.0 13472 13488 pts/1552 A 08:03:40 0:02 udt

    MEMORY
    Real,MB 555008
    % Comp 31
    % Noncomp 67
    % Client 67

    PAGING SPACE
    Size,MB 41408
    % Used 2
    % Free 98

    cheers

    ------------------------------
    Nik Kesic
    DevOps
    LKQ
    NASHVILLE TN United States
    ------------------------------



  • 10.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 12:32
    Do process sizes grow over time? If so, you may have a memory leak. Do all process sizes grow? Do they grow at the same rate? What is common among those that grow most rapidly?

    ------------------------------
    Mark A Baldridge
    Principal Consultant
    Thought Mirror, LLC
    Nacogdoches, Texas United States
    ------------------------------



  • 11.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 12:41
    Edited by Nik Kesic 08-17-2021 12:41
    Mark you bring up a good point as I see Tom's are a few days old. In the case of the IBM system I am monitoring, we do not allow a process to run more than 2 days because we does see memory growth. The max growth i see which starts around 8K, will grow to 100K

    ------------------------------
    Nik Kesic
    DevOps
    LKQ
    NASHVILLE TN United States
    ------------------------------



  • 12.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 13:02
    It sounds like a product improvement opportunity. :)

    Mark A. Baldridge
    Thought Mirror, LLC
    467 County Rd 2011, Nacogdoches, TX 75965
    (413) 841 3892 (Think of it as 4138 4138 92)




  • 13.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 12:54
    This system has processes that log in and then constantly refresh data to the screens real time. They are fairly CPU intensive as well. 

    The thing is, these have not changed in years without memory issues and routinely run for days, or weeks. We instruct users to log off after each shift, but  many do not comply. Since we do alot of NFA updates we cannot simply log them off as they may be in the middle of waiting for remote processing to return back or be written out.

    Here is another current memory snapshot, you can see these are even higher, up to 2gig.

    ps aux | head -1; ps aux | sort -rn +3 | head
    USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
    gmarlo 56557854 0.0 2.0 2235192 2235208 pts/391 A 16:19:15 12:48 udt
    dlaurenz 7471484 0.0 2.0 2821008 2821024 pts/1044 A Aug 16 10:29 udt
    wmach 42467944 0.0 1.0 734848 734864 pts/960 A Aug 13 10:14 udt
    thall2 48234606 0.0 1.0 1704984 1705000 pts/307 A 07:00:08 5:59 udt
    thall2 7930750 0.0 1.0 1149828 1149844 pts/1013 A 07:00:39 4:17 udt
    slewis 1246022 0.0 1.0 1528344 1528360 pts/666 A 13:15:34 35:27 udt
    rmccune 24445252 0.0 1.0 886880 886896 pts/674 A Aug 13 2:58 udt
    rloomis 22086012 0.0 1.0 1054324 1054340 pts/1207 A 06:59:39 9:07 udt
    rbaldwin 16842834 0.0 1.0 767640 767656 pts/849 A Aug 12 7:33 udt
    mbush 27722388 0.0 1.0 908840 908856 pts/370 A Aug 12 12:56 udt

    ------------------------------
    Tom VanKirk
    Unix admin
    Cabinetworks Group Michigan, LLC
    Garrettsville OH United States
    ------------------------------



  • 14.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 13:26
    Tom

    As a matter of interest, did you recompile and catalog all your programs when you moved to 8.2.1.? 


    ------------------------------
    Nik Kesic
    DevOps
    LKQ
    NASHVILLE TN United States
    ------------------------------



  • 15.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 14:09
    No we did not. Did you do that for your upgrade? If so, how would you do it for all the programs?

    ------------------------------
    Tom VanKirk
    Unix admin
    Cabinetworks Group Michigan, LLC
    Garrettsville OH United States
    ------------------------------



  • 16.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-17-2021 14:27
    Did you run updatevoc on each of the accounts?

    ------------------------------
    Mark A Baldridge
    Principal Consultant
    Thought Mirror, LLC
    Nacogdoches, Texas United States
    ------------------------------



  • 17.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 08-18-2021 08:53
    Yes, I ran updatevoc on all accounts.

    We did not recompile all code, never did that before for an upgrade.


    ------------------------------
    Tom VanKirk
    Unix admin
    Cabinetworks Group Michigan, LLC
    Garrettsville OH United States
    ------------------------------



  • 18.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    ROCKETEER
    Posted 11-25-2021 08:55
    Tom,

    Unsure if relevant:

    UniData 8.2.1
    • UDT-15102 Prior to this release, the use of the BASIC MQPUT() and MQGET() functions could have leaked memory. This could cause UniData processes to grow in size when these functions were used repeatedly during the same UniData session. The functions MQOPEN() and MQINQ() were also impacted to some degree. All of these memory leak issues have been resolved.

    And included for reference with regard to a change in memory usage.

    UniData 8.2.3
    • UDT-18482 Starting with UniData 8.1.0, UniBasic variable allocation was moved from shared memory to heaped memory and, as such, the amount of heaped memory a process could use was limited by the operating system itself. Prior to this release, if UniData breached this limit on UNIX, it could result in the session terminating without explanation and leaving the UNIX session in a hung or unresponsive state. This issue has been resolved.


    ------------------------------
    John Jenkins
    Principal Technical Support Engineer
    Rocket Software Limited
    U.K.
    ------------------------------



  • 19.  RE: Unidata memory usage very high from 7.3.3 to 8.2.1

    Posted 11-25-2021 10:58
    Does it say what version was fixed? We are on 8.2.2