UniVerse on AIX Recommendations and tuning.

Good day, I'm supporting a IBM Power AIX environment, which I have been working with for about a year. Along with sorting out the general maintenance of the systems and moving to AIX 7.2 there is a large UniVerse environment on the hardware I support. So I've done a little reading, as I didn't build these environments, about recommend tuning. I couldn't find anything specific, so along with what I've found, I wanted to see if there was anything else I could do or look into. We have a number of test environments, and servers that I can build to test these things on.
I have found some mention of tuning JFS, first was check the type of filesystem mount, if CIO enabled disable it. Then set 'ioo -p -o j2_dynamicBufferPreallocation=64'. I've also seen mention that j2_nPagesPerWriteBehindCluster should be changed from the default 32 to zero

I have some ideas of things I might like to try and see if they made a difference, setting the  "/usr/sbin/syncd 60 > /dev/null 2>&1 &" default to 10 in '/sbin/rc.boot'.
Along with setting I/O pacing, as this is normally turned off as default: chdev -l sys0 -a maxpout='33' -a minpout='24'. I was also interested in learning if turning on Asynchronous IO makes a difference, I know other DB's make this recommendation. But I've no idea if it something that others have found useful.

Finally disk/lun and fibre configuration, with the changes of most system having high speed SSD fibre attached volumes. The instances of lots of LUNs added to a environment and spread across to get better write speeds are gone. With that I need to make sure that the disk, fibre and attachment tunings are the best they can be. I've done the usual and checks the disk queue depths, fibre command queues, but nothing really screams out as an issue.  I have just done some test with 'num_io_queues' and 'num_sp_cmd_elem' on the fibre adapters, but so far I've seen no real difference. 
I am inexperienced with the sort of data that UniVerse writes as my background is DB2 or Oracle, but I understand it's mostly 4k read/writes. Is this true?

Thanks very much.

Thanks, is there a reason that you go for the external file logs compared to inline?

I did wonder about the number of disk LUNs, I have done some testing on a lone AIX server with it running 4k sequential writes. There was some improvement for the writes over the reads, that only seemed to be good for a write heavy system.  I was thinking that the increased number of disk paths would make an improvement, but it didn't seem to be a massive amount. This was for a relatively simple 10 minute test, so in a real world example its very really possible I'm going to get something very different. 
The main reason for the tests is to give examples of the performance improvement to show the team before I look to push it out. 
If there is some JFS improvements, I would be keen to try those.

We have a mix of VIO attached and on metal systems. I prefer VIO managed systems in case I need to move stuff about.

------------------------------
Daniel Martin-Corben
Rocket Forum Shared Account
------------------------------

UV database files are just files as far as AIX is concerned. There is no support for AIO.

So anything that helps cache files in memory, and maximises write performance is a help.

We used to run UV 11.1 and 11.2 on AIX 7.1 (we now use RHEL instead).

We did have a number of goes at engaging IBM to understand how to tune AIX to improve UV performance.
Digging out my notes uncovered the following tunings as a result of these efforts.

These might help, but they are AIX 7.1 so assess them against your current settings and current version applicability.

Buffers

lvmo -v uvvg -o pv_pbuf_count=4096
ioo -p -o j2_dynamicBufferPreallocation=128

JFS2 Sequential Write-behind algorithm & Sync Control

ioo -p -o j2_nPagesPerWriteBehindCluster=0 -o j2_syncPageCount=256 -o j2_syncPageLimit=40960

JFS2 Filesystem Logging & mount option 'noatime'

Mount UVTEMP and UV data filesystems with 'noatime' option to avoid WRITING to the filesystem each time you READ a database file.

Storage Protection Keys

/usr/sbin/skctl -V -k'off'

Memory Tracing

raso -r -o mtrc_enabled=0

Component Tracing

ctctrl -P memtraceoff

Good day, I'm supporting a IBM Power AIX environment, which I have been working with for about a year. Along with sorting out the general maintenance of the systems and moving to AIX 7.2 there is a large UniVerse environment on the hardware I support. So I've done a little reading, as I didn't build these environments, about recommend tuning. I couldn't find anything specific, so along with what I've found, I wanted to see if there was anything else I could do or look into. We have a number of test environments, and servers that I can build to test these things on.
I have found some mention of tuning JFS, first was check the type of filesystem mount, if CIO enabled disable it. Then set 'ioo -p -o j2_dynamicBufferPreallocation=64'. I've also seen mention that j2_nPagesPerWriteBehindCluster should be changed from the default 32 to zero

I have some ideas of things I might like to try and see if they made a difference, setting the  "/usr/sbin/syncd 60 > /dev/null 2>&1 &" default to 10 in '/sbin/rc.boot'.
Along with setting I/O pacing, as this is normally turned off as default: chdev -l sys0 -a maxpout='33' -a minpout='24'. I was also interested in learning if turning on Asynchronous IO makes a difference, I know other DB's make this recommendation. But I've no idea if it something that others have found useful.

Finally disk/lun and fibre configuration, with the changes of most system having high speed SSD fibre attached volumes. The instances of lots of LUNs added to a environment and spread across to get better write speeds are gone. With that I need to make sure that the disk, fibre and attachment tunings are the best they can be. I've done the usual and checks the disk queue depths, fibre command queues, but nothing really screams out as an issue.  I have just done some test with 'num_io_queues' and 'num_sp_cmd_elem' on the fibre adapters, but so far I've seen no real difference. 
I am inexperienced with the sort of data that UniVerse writes as my background is DB2 or Oracle, but I understand it's mostly 4k read/writes. Is this true?

Thanks very much.

There are some workloads that j2_nPagesPerWriteBehindCluster impact. The function flushes dirty pages in the previous, next lower address, cluster to disk. Once on the write queue, you cannot access them until they complete the flush and are marked clean.

An example of one workload repeatedly writes records into adjacent groups. The demonstration uses a type 2 file, and repeats writing of records n and n+1, such as record ID pairs of 127 and 128. When record n is in the end of one cluster, and n+1 is in the next cluster, writing record n+1 forces the flush of the page containing record n, and you have to wait for it to become free again. Writing the pair of records that cross the cluster boundary can take significantly longer elapsed time than other record pairs in the file. In addition, the forced writes defeat only writing into the cache, so physical writes can drastically increase. Measurements also indicate additional CPU charged to the process due to the physical disk writes.

With this example to understand the mechanism, where do we see this happen in an application?

• A file containing control records can exceed one cluster. Control records can contain next available numbers, last updated time stamps, and other frequently updated values. The j2_nPagesPerWriteBehindCluster can affect primary groups in the file, as well as blocks in overflow chains.

• UVTEMP contains sorting and selecting work files. These temporary files typically have short life span, less than the default file system sync time of 60 seconds. The ps command displays sync daemon cycle time as part of the executed command. Disabling j2_nPagesPerWriteBehindCluster can avoid sending those temporary blocks to disk completely. Short duration EXECUTE CAPTURING files seldom grow to the default 128K cluster size, so the pain incurred from this workload relates to creating and deleting inodes for the capture file.

• Index records consist of main file ID for an indexed value, separated by field marks. A chain of 8K blocks comprising the record can cross cluster boundaries. While not necessarily in file address sequence, writing can delay rewriting the next time because of actively-flushing dirty blocks. I witnessed one nightly process writing daily activity records into a summary file. The index built on date contained a multiple-megabyte list of ID when complete. Each addition of a main file ID would rewrite the index record each time, one ID longer for each record containing "today". The approximately 2.5 hour run each night dropped to under 40 minutes, with the dirty blocks only flushing at the sync daemon cycle time instead of each write. Combined with other nightly processes interacting with j2_nPagesPerWriteBehindCluster, the physical writes became almost negligible.

• Oversize records in dynamic and static hashed files are comprised of group-size blocks. An oversize write first adds the blocks to the free list, and then allocates from the free list, and possibly more at the end of the overflow area, to rewrite the record. Touching those blocks more than once can affect adjacent clusters and aggravate j2_nPagesPerWriteBehindCluster dirty page flushing.

These are examples of the workloads affected by j2_nPagesPerWriteBehindCluster.

Maybe I need to resume the Hitchhiker's Guide to the UniVerse series?

------------------------------
Mark A Baldridge
Principal Consultant
Thought Mirror
Nacogdoches, Texas United States
------------------------------

Has anyone done any testing with striping the lv? In theory it should allow multiple logical LUN's to be accessed also helping the improvement of performance on the system.

There are some workloads that j2_nPagesPerWriteBehindCluster impact. The function flushes dirty pages in the previous, next lower address, cluster to disk. Once on the write queue, you cannot access them until they complete the flush and are marked clean.

An example of one workload repeatedly writes records into adjacent groups. The demonstration uses a type 2 file, and repeats writing of records n and n+1, such as record ID pairs of 127 and 128. When record n is in the end of one cluster, and n+1 is in the next cluster, writing record n+1 forces the flush of the page containing record n, and you have to wait for it to become free again. Writing the pair of records that cross the cluster boundary can take significantly longer elapsed time than other record pairs in the file. In addition, the forced writes defeat only writing into the cache, so physical writes can drastically increase. Measurements also indicate additional CPU charged to the process due to the physical disk writes.

With this example to understand the mechanism, where do we see this happen in an application?

• A file containing control records can exceed one cluster. Control records can contain next available numbers, last updated time stamps, and other frequently updated values. The j2_nPagesPerWriteBehindCluster can affect primary groups in the file, as well as blocks in overflow chains.

• UVTEMP contains sorting and selecting work files. These temporary files typically have short life span, less than the default file system sync time of 60 seconds. The ps command displays sync daemon cycle time as part of the executed command. Disabling j2_nPagesPerWriteBehindCluster can avoid sending those temporary blocks to disk completely. Short duration EXECUTE CAPTURING files seldom grow to the default 128K cluster size, so the pain incurred from this workload relates to creating and deleting inodes for the capture file.

• Index records consist of main file ID for an indexed value, separated by field marks. A chain of 8K blocks comprising the record can cross cluster boundaries. While not necessarily in file address sequence, writing can delay rewriting the next time because of actively-flushing dirty blocks. I witnessed one nightly process writing daily activity records into a summary file. The index built on date contained a multiple-megabyte list of ID when complete. Each addition of a main file ID would rewrite the index record each time, one ID longer for each record containing "today". The approximately 2.5 hour run each night dropped to under 40 minutes, with the dirty blocks only flushing at the sync daemon cycle time instead of each write. Combined with other nightly processes interacting with j2_nPagesPerWriteBehindCluster, the physical writes became almost negligible.

• Oversize records in dynamic and static hashed files are comprised of group-size blocks. An oversize write first adds the blocks to the free list, and then allocates from the free list, and possibly more at the end of the overflow area, to rewrite the record. Touching those blocks more than once can affect adjacent clusters and aggravate j2_nPagesPerWriteBehindCluster dirty page flushing.

These are examples of the workloads affected by j2_nPagesPerWriteBehindCluster.

Maybe I need to resume the Hitchhiker's Guide to the UniVerse series?

------------------------------
Mark A Baldridge
Principal Consultant
Thought Mirror
Nacogdoches, Texas United States
------------------------------

It's been a  while since I've dipped into JFS2 - and there used to be some useful papers on optimising for random access on various IBM web pages, sadly long gone - at least at the original locations. As Mark commented, j2_nPagesPerWriteBehindCluster can have a major effect and if set inappropriately can have the effect of pausing physical disk access while buffers are flushed at intervals. While on heavy batch loads a delay of 30 seconds every hour or so may overall be more efficient on workload throughput, for an interactive workload a lockup of 30 seconds is far more significant compared to saving the total number of disk IOPS in a day. It's also worth noting that when setting the value with the ioo command, if you wish to have the change continue past the next reboot you need to explicitly say so to make the cha nge permanent.  

As a generalisation which seems to work, setting j2_nPagesPerWriteBehindCluster to zero seems to be the best option. As does putting the log device  on a separate fast disk in the Volume Group.

I also recall a limit somewhere - I think it was to the number of write channels per logical volume or volume group (but please cross-check me on this). Where a very large number of random access (i.e. hashed) files are being updated concurrently then it can be better to have multiple volumes than adopt a  one-big-volume approach. 

I spent quite a lot of time working with  a customer on tuning and optimising their JFS2 file system some years ago, down the level of buffer sizes, policies, queues and load management, but there is no general 'one size fits all' answer unfortunately - and you can get to the point of diminishing returns fairly quickly.

Hoping this helps somewhat, unfortunately I no longer have the original IBM documents or the site-specific report I wrote at the time. 

JJ