The flexibility of iCluster configuration can allow an administrator many different avenues for consideration for tuning. I like to say, "It's not the great word processor that creates a best selling novel". As a result, we have a powerful tool in iCluster, we just need to understand and implement it the best way to meet the demands of the project with the available flexibility in the requirements and the facilities available to us. (Application availability demands, network constraints, system resources potential and resource availability, and replication results review or monitoring.)
There are several additional performance topic posts in the Community Forum and if you don't find what you are looking for here, take a look at the other posts or ask questions to the community. I'll try to respond to all questions directed to me but sometimes the answer may come from a fellow iCluster user in our group. Feel free to add additional expertise and experience to the discussion. (Of course, please keep the discussion positive and helpful.)
Here are some assorted tips on how to make the iCluster apply process perform optimally:
- End journaling on the backup for busy files (or all files if you are rarely going to use the backup for a switchover but primarily for offloading backups and possible DR). Ending journaling on the backup could add some time (anticipated to be very low) to the switch process as these files will be restarted journaling at that time. (You can benchmark the process to see if your changes impact switch times significantly) The benefit would be reduced I/O on the backup server which could improve update performance on the backup node. To accomplish that task, first identify the busiest files in replication. The "Journal analysis reports" menu option under the "iCluster reports menu" option can reveal this detail. We may, once we identify the target files, want to also separate these into a separate replication group as that will allow us to set a group parameter to not journal files on the backup (ICluster would otherwise automatically start journaling for them again). Or if we chose to leave them grouped together we could end journaling for all objects for this group on the backup, set our default journal as desired and allow journaling to be automatically initiated in the event of a switchover.
- If your applications update records randomly over the file (i.e. prominently random vs. sequentially) and are larger than your memory pool, turn on the Optimize apply parm for the group by ending the group and then using the OPTIMZAP(*YES) parameter on the DMCHGGRP command followed by restarting the group normally.
- Separate your most active files into a different journal. Again the journal analysis report can help us identify the most impactful files for this action. When we have a new journal added to a group, iCluster will initiate a new apply process for the objects in the additional journal. Apply throughput is the obvious result and can make significant difference in latency reduction efforts.
- Access Path Rebuild (APR) processing can take a significant amount of time for files that are significantly large and have many logicals and indexes built over them. Usually that is a one-off process and no action is required but large files that perform an APR frequently can impact replication apply performance as the production file is put back in service and pushing updates while the APR is trying to complete for the target files and logicals. Basically, what the Access Path Rebuild Controller service does is automatically cycle the parameter for APR processing from *IMMEDIATE to *DELAYED for the files enrolled so that on-going updates can be processed and keep latency low. For large files with many logicals and a large number of inserts, updates and deletes, you can use the Access Path Rebuild Controller on iCluster 7.1 and above. Use the DMCRTAPLST (Create Access Path List, DMADDPFAPL (Add Physical Files to Access Path List) and DMSTRAPM (Start Access Path Management) commands to make this happen. Check it out in more detail in the iCluster User Guide and online help.
- Consider a separate staging store library for each group at iCluster 7.1+ using the STGSTORLIB parameter on the DMADDGRP/DMCHGGRP command. I said 'consider it' but actually this is highly recommended. Separate staging not only can improve performance but it also provides a level of separation that can protect a replication group from damaged objects (in the event of an abnormal end) so that only one group would effected rather than all groups sharing a Staging Library. Try the command DMGRPSTS and notice the Store Lib column in the resulting report to see if all your groups have a unique definition.
- Add a shared memory pool and set the XDMCLUSTER SBS to use it. The backup node will benefit the most from this tip however I like to set this up in advance so that a switchover can perform well and is less complex regardless of which node is in the BACKUP role. Even if there is nothing running on the backup node besides iCluster and there should be nothing competing with iCluster for memory in SYSBAS, iCluster apply performance is enhanced by making memory directly available to the XDMCLUSTER SBS. How much should I allocate to it? Allocate as much as is available to the shared pool. We are running v7.4 IBM i and QPFRADJ is activated on the backup node. The OS seems to be responsive to memory demands when needed making it easy for administration.
- Allow the IBM i system to determine the paging characteristics for the pool in which iCluster is running using CHGSHRPOOL (<pool id>) PAGING(*CALC)". This is aligned with the previous tip and is the recommended setting for paging for the shared pool.
- Last but not least, upgrade to the latest version of Rocket iCluster. Some of these tips are applicable to versions of iCluster all the way back to V7.1 but I hope everyone would not have stayed on an older release just because it is working 'well enough'. The Rocket development team works hard to identify performance enhancements, usability features and new IBM i OS support to help our users have the best and most reliable experience with iCluster. They also actively review customer enhancement requests and identifying those that would impact the community with the best features in our solution and putting them on the roadmap. Staying on an older release could have you missing features that could assist reliability and performance capability that has already been delivered to the solution. One such example is a significant improvement in 'Commitment Control' apply performance delivered to iCluster in the summer of 2021.
Using these tips as part of your tuning strategy for iCluster should help you think about what factors impact replication performance and how to best mitigate negative impacts. Rocket also offers professional services to help implement these changes if you need assistance. Reach out to your sales executive and they will help you estimate the right amount of assistance needed.
If you have a tip you use in your environment that I missed, please share it with the community or ask questions if the discussion was thought provoking. We have some very highly experienced IBM i expertise in our user community that are willing to help or discuss strategies. Also invite anyone you think would be interested in the discussion to join us.
------------------------------
Mark Watts
Software Engineer
Rocket Software Inc
Waltham MA United States
------------------------------