In pre-production DC-DR systems, we did a test of workload switchover this weekend.  We started with restoring from tape updated app DB libraries from UAT over the existing app DB in DC system while iCluster replication group is stopped (both nodes still active).  When the restore was finished, I ran DMMRKPOS  and started the group back with USEMARKED(*YES)  because  I wanted to test Sync Check function to do the replication in this case.

Then I ran Sync Check *FULL at primary node and saw more than 3,000 out-of-sync objects reported. Then I manually ran OOS Activation to replicate to backup node during the night time.     The next morning, we let a few test users enter some app transactions into DC node and when they finished, I checked that I saw no pending change application and we did iCluster and application workload switchover to DR node and users tested various app functions and surprisingly found that many functions encountered errors.  Then the app team member displayed some key app tables in backup node and surprisingly found these tables had DIFFERENT contents from those in primary node!!  

I consulted system engineer from local Rocket WW partner and was told to run Sync Check again with *CHECKSUM and I did with ACTOOS(*YES) and, again, surprisingly found that 259 more objects report as out of sync and activated to DR node.   I was confused as I expected that FULL is FULL. it's a common sense meaning, right?  

Then I browsed iCluster 9.1.0 User's Guide and found this statement about Sync Check :

*FULL-Performs an *OBJATTR sync check on all objects that are replicated by the group and a *FILEATTR sync check on database file objects. If the selected group is not active, only an *OBJATTR sync check is run and a message is issued. This is the default value. 

You can optionally run a checksum sync check as part of the continuous *FULL sync check by specifying YES for the Check file contents element of the FULLSCOPTS (*FULL sync check options) parameter of the DMSETSVAL command.

Then I found this from the help text :

At this point, I had a confusing impression that User Guide inplied that FULLSCOPTS works only for "continuous SC" but the help text implied it worked for both manual and continuous SC. 

From the information above, I want to check if I have an accurate conclusion on using DMSTRSC *FULL.  Am I correct to understand that  without running a "continuous sync check", DMSTRSC *FULL ACTUALLY does a partial sync check which means that I must follow manual DMSTRSC *FULL with another round of manual DMSTRSC *CHECKSUM in order to be sure that the data from my primary node are FULLY replicated to the backup node?

If my understanding above is correct, I hope that iCluster team makes addition information in both User Guide and help text of DMSTRSC to address this subtle point. 

But if actually FULLSCOPTS *YES works for both manual and continuous SC, then I ask that iCLuster developer team set FULLSCOPTS to *YES as a default value at least for the Check File Content part.  This is because setting it to *NO by default causes a hassle of running DMSTRSC twice to ensure a really full OOS report and replication.

Thanks in advance for any clarification for my confusion.

Hi Satid,

First of all, in order to reduce the amount of OOS, I suggest you execute the steps as below:

• end group,
• run DMMRKPOS,
• perform data update, 
• start group with mark position.

Secondly,  if you want to check file content when perform FULL sync check, parameter RUNCHKSUM(*YES) should be set *YES, for example DMSTRSC GROUP(XNMRKTST) SCTYPE(*FULL) RUNCHKSUM(*YES), so NO need to run DMSTRSC *FULL, then run DMSTRSC *CHECKSUM. Actually, there are two iCluster system values on DMSETSVL

*FULL sync check options:        FULLSCOPTS       

  Check file contents  . . . . .                      *NO   ---  Default value of RUNCHKSUM for DMSTRSC and STRHASC

  Check obsolete objects . . . .                *NO   ---  Default value of RUNCHKOBSL for DMSTRSC and STRHASC

At the same time, it can be used for continue sync check when user want to perform FULL  continue sync check and check file content or obsolete object

Finally, both default value from FULLSCOPTS should keep *NO, because checksum sync check will take more system CPU% and affect system performance.

------------------------------
Jason Sun
Rocket iCluster Development Team

------------------------------

In pre-production DC-DR systems, we did a test of workload switchover this weekend.  We started with restoring from tape updated app DB libraries from UAT over the existing app DB in DC system while iCluster replication group is stopped (both nodes still active).  When the restore was finished, I ran DMMRKPOS  and started the group back with USEMARKED(*YES)  because  I wanted to test Sync Check function to do the replication in this case.

Then I ran Sync Check *FULL at primary node and saw more than 3,000 out-of-sync objects reported. Then I manually ran OOS Activation to replicate to backup node during the night time.     The next morning, we let a few test users enter some app transactions into DC node and when they finished, I checked that I saw no pending change application and we did iCluster and application workload switchover to DR node and users tested various app functions and surprisingly found that many functions encountered errors.  Then the app team member displayed some key app tables in backup node and surprisingly found these tables had DIFFERENT contents from those in primary node!!  

I consulted system engineer from local Rocket WW partner and was told to run Sync Check again with *CHECKSUM and I did with ACTOOS(*YES) and, again, surprisingly found that 259 more objects report as out of sync and activated to DR node.   I was confused as I expected that FULL is FULL. it's a common sense meaning, right?  

Then I browsed iCluster 9.1.0 User's Guide and found this statement about Sync Check :

*FULL-Performs an *OBJATTR sync check on all objects that are replicated by the group and a *FILEATTR sync check on database file objects. If the selected group is not active, only an *OBJATTR sync check is run and a message is issued. This is the default value. 

You can optionally run a checksum sync check as part of the continuous *FULL sync check by specifying YES for the Check file contents element of the FULLSCOPTS (*FULL sync check options) parameter of the DMSETSVAL command.

Then I found this from the help text :

At this point, I had a confusing impression that User Guide inplied that FULLSCOPTS works only for "continuous SC" but the help text implied it worked for both manual and continuous SC. 

From the information above, I want to check if I have an accurate conclusion on using DMSTRSC *FULL.  Am I correct to understand that  without running a "continuous sync check", DMSTRSC *FULL ACTUALLY does a partial sync check which means that I must follow manual DMSTRSC *FULL with another round of manual DMSTRSC *CHECKSUM in order to be sure that the data from my primary node are FULLY replicated to the backup node?

If my understanding above is correct, I hope that iCluster team makes addition information in both User Guide and help text of DMSTRSC to address this subtle point. 

But if actually FULLSCOPTS *YES works for both manual and continuous SC, then I ask that iCLuster developer team set FULLSCOPTS to *YES as a default value at least for the Check File Content part.  This is because setting it to *NO by default causes a hassle of running DMSTRSC twice to ensure a really full OOS report and replication.

Thanks in advance for any clarification for my confusion.