Rocket iCluster

Just wondering how people handle the replication of /qibm/UserData?

------------------------------
Duane Gingerich
SE
Able-One Systems Inc
kitchener ON Canada
------------------------------

Hi Duane, 

Assuming the target system was built with an option 21 save then restore, all the apps and paths are there and ready to be started.  But what to replicate can be a 'per application or service' dependent response. 

For instance, reference this IBM document posted regarding the replication of the 'Content Manager'.  https://www.ibm.com/support/pages/system/files/inline-files/CMOD%20for%20i%20-%20Considerations%20for%20High%20Availability_0.pdf

Most of the items in this path are user entries and can be easily replicated and used on the target system without any additional intervention.  However we see from the document that some apps are concerned that replication may lock some of their objects during replication.     

Some clients walk through the contents carefully to identify specifically what they want to make available on the target node in real time and select those in an IFS replication group.  Entries in the HTTPA and HTTPSVR require more handholding to get new applications mirrored from the primary registered onto the target node(s) when they are changed or added on the primary system.  (Just because the objects are replicated doesn't necessarily mean they will be available to start using the HTTPADMIN without making certain all the requisite objects are also replicated.)  In other words, you can add a HTTPSVR from /qibm/UserData/ that is actually located in /www/some_child_folder.

Another option is to set up periodic refresh groups to snapshot the contents of this folder and dependents to keep contents up to date while avoiding locking objects during critical operations of the production app.  But object locking can also be avoided using IFS journaling to replicate object changes while setting an appropriate object replication delay at the replication group level to avoid application conflicts.

Hope that helps.


------------------------------
Mark Watts
Software Engineer
Rocket Software Inc
Waltham MA United States
------------------------------

Original Message:
Sent: 10-13-2021 12:35
From: Duane Gingerich
Subject: Replicating /qibm/UserData

Just wondering how people handle the replication of /qibm/UserData?

------------------------------
Duane Gingerich
SE
Able-One Systems Inc
kitchener ON Canada
------------------------------

Hi Mark/Duane,

We also encountered a similar issue further down the path /QIBM/USERDATA/ICSS/Cert with replicating certificates and maintaining DCM.
With added focus on security what are the Rocket recommendations for replicating certs and maintaining DCM on source/target LPAR's?
The customer wishes to maintain a single set of certs on the source LPAR's only.

PS. We recently upgraded Rocket to the latest version on all LPAR's per earlier Rocket recommendations. Together with OS V7R4 patched to almost latest levels to benefit from the latest OS & Rocket features.

Thank You,


------------------------------
Charles Charalambous
Rocket Forum Shared Account
------------------------------

Original Message:
Sent: 10-28-2021 17:01
From: Mark Watts
Subject: Replicating /qibm/UserData

Hi Duane, 

Assuming the target system was built with an option 21 save then restore, all the apps and paths are there and ready to be started.  But what to replicate can be a 'per application or service' dependent response. 

For instance, reference this IBM document posted regarding the replication of the 'Content Manager'.  https://www.ibm.com/support/pages/system/files/inline-files/CMOD%20for%20i%20-%20Considerations%20for%20High%20Availability_0.pdf

Most of the items in this path are user entries and can be easily replicated and used on the target system without any additional intervention.  However we see from the document that some apps are concerned that replication may lock some of their objects during replication.     

Some clients walk through the contents carefully to identify specifically what they want to make available on the target node in real time and select those in an IFS replication group.  Entries in the HTTPA and HTTPSVR require more handholding to get new applications mirrored from the primary registered onto the target node(s) when they are changed or added on the primary system.  (Just because the objects are replicated doesn't necessarily mean they will be available to start using the HTTPADMIN without making certain all the requisite objects are also replicated.)  In other words, you can add a HTTPSVR from /qibm/UserData/ that is actually located in /www/some_child_folder.

Another option is to set up periodic refresh groups to snapshot the contents of this folder and dependents to keep contents up to date while avoiding locking objects during critical operations of the production app.  But object locking can also be avoided using IFS journaling to replicate object changes while setting an appropriate object replication delay at the replication group level to avoid application conflicts.

Hope that helps.


------------------------------
Mark Watts
Software Engineer
Rocket Software Inc
Waltham MA United States

Original Message:
Sent: 10-13-2021 12:35
From: Duane Gingerich
Subject: Replicating /qibm/UserData

Just wondering how people handle the replication of /qibm/UserData?

------------------------------
Duane Gingerich
SE
Able-One Systems Inc
kitchener ON Canada
------------------------------

Hi Charles,

I spoke to colleagues and did a bit of research.  What I have found is a very helpful Question & Answer page on the IBM web site that you may find helpful as well.  There are several videos and instructions to help manage digital certificates.  The very last Q&A entry discusses backup/migrate/replicate of the Digital Certificate Management (DCM) environment. 
The reference is from: https://www.ibm.com/support/pages/digital-certificate-manager-dcm-frequently-asked-questions-and-common-tasks

This article highlights the requirement to perform some migration steps to activate the Digital Certificates on the target for migration or replication.

Notice also that the document includes in panel #4  a warning of "IBM i V7.2 and V7.3 Only".  This document referenced was published prior to the release of V7.4 and it may require reaching out to the writer or searching further if the subject OS version is V7.4.

The particular post entry of interest is included below.

• How do I backup/migrate/replicate my Digital Certificate Management (DCM) environment?
  
  There are three main items comprising your DCM environment. When backing up,migrating,or replicating your DCM environment; you should ensure the following items are included and restored/replicated properly to your new IBM i server.
  

  1.	The DCM certificate store (aka key store) files containing your SSL certificates. The certificate store files are all located in the following IFS location: /QIBM/UserData/ICSS/Cert This directory can be backed up/migrated/replicated with the following SAV (save) and RST (restore) CL commands. CRTSAVF FILE(QGPL/DCMIFS) SAV DEV('/QSYS.LIB/QGPL.LIB/DCMIFS.FILE') OBJ(('/QIBM/UserData/ICSS/Cert')) DTACPR(*HIGH) PVTAUT(*YES) RST DEV('/QSYS.LIB/QGPL.LIB/DCMIFS.FILE') OBJ(('/QIBM/UserData/ICSS/Cert')) PVTAUT(*YES)
  2.	The certificate store system password stash. The certificate store system password is used by IBM i OS applications (Telnet,FTP,Host Servers,HTTP,etc.) to authenticate to the DCM certificate stores (*SYSTEM and Local CA). This password stash is needed to keep the passwords in sync between the DCM certificate stores in the IFS and the system password hash managed by the OS. Failure to save and restore the certificate store system password stash along with the DCM certificate stores in the IFS,may result in the loss of your SSL certificates! Please refer to the following URL for more information on this. IBM Knowledge Center - Backup and recovery considerations for DCM data =================================== To save the certificate store system password stash,you can use either the SAVSYS or SAVSECDTA commands. To restore the DCM password stash,use the following restore user profile command: RSTUSRPRF USRPRF(*NONE) SECDTA(*DCM) NOTE: The above command requires your IBM i server to be in restricted state (ENDSBS *ALL).
  3.	The user index containing the SSL certificate to IBM i application assignments. The SSL certificate to IBM i application assignments are stored in the following user index object: QUSRSYS/QYCDCERTI This can be saved with the SAVOBJ command and restored with the RSTOBJ command. CRTSAVF FILE(QGPL/DCMUSRIDX) SAVOBJ OBJ(QYCDCERTI) LIB(QUSRSYS) DEV(*SAVF) SAVF(QGPL/DCMUSRIDX) PVTAUT(*YES) RSTOBJ OBJ(QYCDCERTI) SAVLIB(QUSRSYS) DEV(*SAVF) SAVF(QGPL/DCMUSRIDX) PVTAUT(*YES)
  4.	NOTE: For IBM i 7.2 and 7.3 ONLY,you will need to run the following command to migrate your IBM i Local CA information. Run the following command on the IBM i command line: CALL QICSS/QYCUMIGREX This call will migrate the local CA information to the new format that IBM i 7.2 and 7.3 utilizes (since the OS now supports more than one Local CA).
  5.	(OPTIONAL) - All DCM Application IDs are stored under the exit point,QIBM_QSY_CERT_APPS,in the QUSRSYS/QUSEXRGOBJ object. WARNING!!! The QUSRSYS/QUSEXRGOBJ object contains information for ALL exit points. If this object is migrated,ALL exit point information is also migrated. Use caution when migrating/replication this object since this will migrate/replicate ALL exit point information which might negatively affect your IBM i server if not handled properly. If you wish to migrate/replicate your DCM application ID information,you will need to save and restore/replicate this object. This can be saved with the SAVOBJ command and restored with the RSTOBJ command. CRTSAVF FILE(QGPL/DCMAPPS) SAVOBJ OBJ(QUSEXRGOBJ) LIB(QUSRSYS) DEV(*SAVF) SAVF(QGPL/DCMAPPS) PVTAUT(*YES) RSTOBJ OBJ(QUSEXRGOBJ) SAVLIB(QUSRSYS) DEV(*SAVF) SAVF(QGPL/DCMAPPS) PVTAUT(*YES)

  The following document provides more information regarding the backup and recovery of your DCM environment.
  
  

  IBM Knowledge Center - Backup and recovery considerations for DCM data
  
  =-=-=-=-=-=-=-=-=-=-=-=-=-=-=
  
  

------------------------------
Mark Watts
Sr. Software Engineer
Rocket Software Inc
Waltham MA United States
------------------------------

Original Message:
Sent: 11-03-2021 17:46
From: Charles Charalambous
Subject: Replicating /qibm/UserData

Hi Mark/Duane,

We also encountered a similar issue further down the path /QIBM/USERDATA/ICSS/Cert with replicating certificates and maintaining DCM.
With added focus on security what are the Rocket recommendations for replicating certs and maintaining DCM on source/target LPAR's?
The customer wishes to maintain a single set of certs on the source LPAR's only.

PS. We recently upgraded Rocket to the latest version on all LPAR's per earlier Rocket recommendations. Together with OS V7R4 patched to almost latest levels to benefit from the latest OS & Rocket features.

Thank You,


------------------------------
Charles Charalambous
Rocket Forum Shared Account

Original Message:
Sent: 10-28-2021 17:01
From: Mark Watts
Subject: Replicating /qibm/UserData

Hi Duane, 

Assuming the target system was built with an option 21 save then restore, all the apps and paths are there and ready to be started.  But what to replicate can be a 'per application or service' dependent response. 

For instance, reference this IBM document posted regarding the replication of the 'Content Manager'.  https://www.ibm.com/support/pages/system/files/inline-files/CMOD%20for%20i%20-%20Considerations%20for%20High%20Availability_0.pdf

Most of the items in this path are user entries and can be easily replicated and used on the target system without any additional intervention.  However we see from the document that some apps are concerned that replication may lock some of their objects during replication.     

Some clients walk through the contents carefully to identify specifically what they want to make available on the target node in real time and select those in an IFS replication group.  Entries in the HTTPA and HTTPSVR require more handholding to get new applications mirrored from the primary registered onto the target node(s) when they are changed or added on the primary system.  (Just because the objects are replicated doesn't necessarily mean they will be available to start using the HTTPADMIN without making certain all the requisite objects are also replicated.)  In other words, you can add a HTTPSVR from /qibm/UserData/ that is actually located in /www/some_child_folder.

Another option is to set up periodic refresh groups to snapshot the contents of this folder and dependents to keep contents up to date while avoiding locking objects during critical operations of the production app.  But object locking can also be avoided using IFS journaling to replicate object changes while setting an appropriate object replication delay at the replication group level to avoid application conflicts.

Hope that helps.


------------------------------
Mark Watts
Sr. Software Engineer
Rocket Software Inc
Waltham MA United States

Original Message:
Sent: 10-13-2021 12:35
From: Duane Gingerich
Subject: Replicating /qibm/UserData

Just wondering how people handle the replication of /qibm/UserData?

------------------------------
Duane Gingerich
SE
Able-One Systems Inc
kitchener ON Canada
------------------------------
​

Hi Mark,
This is an excellent DCM link.
Thank You,

------------------------------
Charles Charalambous
Rocket Forum Shared Account
------------------------------

Original Message:
Sent: 11-11-2021 11:08
From: Mark Watts
Subject: Replicating /qibm/UserData

Hi Charles,

I spoke to colleges and did a bit of research.  What I have found is a very helpful Question & Answer page on the IBM web site that you may find helpful.  There are several videos and instructions to help manage digital certificates.  The very last Q&A entry discusses backup/migrate/replicate of the Digital Certificate Management (DCM) environment. 
The reference is from: https://www.ibm.com/support/pages/digital-certificate-manager-dcm-frequently-asked-questions-and-common-tasks

This article highlights the requirement to perform some migration steps to activate the Digital Certificates on the target for migration or replication.


The particular post entry of interest is included below.

• How do I backup/migrate/replicate my Digital Certificate Management (DCM) environment?
  
  There are three main items comprising your DCM environment. When backing up,migrating,or replicating your DCM environment; you should ensure the following items are included and restored/replicated properly to your new IBM i server.
  

  1.	The DCM certificate store (aka key store) files containing your SSL certificates. The certificate store files are all located in the following IFS location: /QIBM/UserData/ICSS/Cert This directory can be backed up/migrated/replicated with the following SAV (save) and RST (restore) CL commands. CRTSAVF FILE(QGPL/DCMIFS) SAV DEV('/QSYS.LIB/QGPL.LIB/DCMIFS.FILE') OBJ(('/QIBM/UserData/ICSS/Cert')) DTACPR(*HIGH) PVTAUT(*YES) RST DEV('/QSYS.LIB/QGPL.LIB/DCMIFS.FILE') OBJ(('/QIBM/UserData/ICSS/Cert')) PVTAUT(*YES)
  2.	The certificate store system password stash. The certificate store system password is used by IBM i OS applications (Telnet,FTP,Host Servers,HTTP,etc.) to authenticate to the DCM certificate stores (*SYSTEM and Local CA). This password stash is needed to keep the passwords in sync between the DCM certificate stores in the IFS and the system password hash managed by the OS. Failure to save and restore the certificate store system password stash along with the DCM certificate stores in the IFS,may result in the loss of your SSL certificates! Please refer to the following URL for more information on this. IBM Knowledge Center - Backup and recovery considerations for DCM data =================================== To save the certificate store system password stash,you can use either the SAVSYS or SAVSECDTA commands. To restore the DCM password stash,use the following restore user profile command: RSTUSRPRF USRPRF(*NONE) SECDTA(*DCM) NOTE: The above command requires your IBM i server to be in restricted state (ENDSBS *ALL).
  3.	The user index containing the SSL certificate to IBM i application assignments. The SSL certificate to IBM i application assignments are stored in the following user index object: QUSRSYS/QYCDCERTI This can be saved with the SAVOBJ command and restored with the RSTOBJ command. CRTSAVF FILE(QGPL/DCMUSRIDX) SAVOBJ OBJ(QYCDCERTI) LIB(QUSRSYS) DEV(*SAVF) SAVF(QGPL/DCMUSRIDX) PVTAUT(*YES) RSTOBJ OBJ(QYCDCERTI) SAVLIB(QUSRSYS) DEV(*SAVF) SAVF(QGPL/DCMUSRIDX) PVTAUT(*YES)
  4.	NOTE: For IBM i 7.2 and 7.3 ONLY,you will need to run the following command to migrate your IBM i Local CA information. Run the following command on the IBM i command line: CALL QICSS/QYCUMIGREX This call will migrate the local CA information to the new format that IBM i 7.2 and 7.3 utilizes (since the OS now supports more than one Local CA).
  5.	(OPTIONAL) - All DCM Application IDs are stored under the exit point,QIBM_QSY_CERT_APPS,in the QUSRSYS/QUSEXRGOBJ object. WARNING!!! The QUSRSYS/QUSEXRGOBJ object contains information for ALL exit points. If this object is migrated,ALL exit point information is also migrated. Use caution when migrating/replication this object since this will migrate/replicate ALL exit point information which might negatively affect your IBM i server if not handled properly. If you wish to migrate/replicate your DCM application ID information,you will need to save and restore/replicate this object. This can be saved with the SAVOBJ command and restored with the RSTOBJ command. CRTSAVF FILE(QGPL/DCMAPPS) SAVOBJ OBJ(QUSEXRGOBJ) LIB(QUSRSYS) DEV(*SAVF) SAVF(QGPL/DCMAPPS) PVTAUT(*YES) RSTOBJ OBJ(QUSEXRGOBJ) SAVLIB(QUSRSYS) DEV(*SAVF) SAVF(QGPL/DCMAPPS) PVTAUT(*YES)

  The following document provides more information regarding the backup and recovery of your DCM environment.
  
  

  IBM Knowledge Center - Backup and recovery considerations for DCM data
  
  =-=-=-=-=-=-=-=-=-=-=-=-=-=-=
  
  

------------------------------
Mark Watts
Software Engineer
Rocket Software Inc
Waltham MA United States

Original Message:
Sent: 11-03-2021 17:46
From: Charles Charalambous
Subject: Replicating /qibm/UserData

Hi Mark/Duane,

We also encountered a similar issue further down the path /QIBM/USERDATA/ICSS/Cert with replicating certificates and maintaining DCM.
With added focus on security what are the Rocket recommendations for replicating certs and maintaining DCM on source/target LPAR's?
The customer wishes to maintain a single set of certs on the source LPAR's only.

PS. We recently upgraded Rocket to the latest version on all LPAR's per earlier Rocket recommendations. Together with OS V7R4 patched to almost latest levels to benefit from the latest OS & Rocket features.

Thank You,


------------------------------
Charles Charalambous
Rocket Forum Shared Account

Original Message:
Sent: 10-28-2021 17:01
From: Mark Watts
Subject: Replicating /qibm/UserData

Hi Duane, 

Assuming the target system was built with an option 21 save then restore, all the apps and paths are there and ready to be started.  But what to replicate can be a 'per application or service' dependent response. 

For instance, reference this IBM document posted regarding the replication of the 'Content Manager'.  https://www.ibm.com/support/pages/system/files/inline-files/CMOD%20for%20i%20-%20Considerations%20for%20High%20Availability_0.pdf

Most of the items in this path are user entries and can be easily replicated and used on the target system without any additional intervention.  However we see from the document that some apps are concerned that replication may lock some of their objects during replication.     

Some clients walk through the contents carefully to identify specifically what they want to make available on the target node in real time and select those in an IFS replication group.  Entries in the HTTPA and HTTPSVR require more handholding to get new applications mirrored from the primary registered onto the target node(s) when they are changed or added on the primary system.  (Just because the objects are replicated doesn't necessarily mean they will be available to start using the HTTPADMIN without making certain all the requisite objects are also replicated.)  In other words, you can add a HTTPSVR from /qibm/UserData/ that is actually located in /www/some_child_folder.

Another option is to set up periodic refresh groups to snapshot the contents of this folder and dependents to keep contents up to date while avoiding locking objects during critical operations of the production app.  But object locking can also be avoided using IFS journaling to replicate object changes while setting an appropriate object replication delay at the replication group level to avoid application conflicts.

Hope that helps.


------------------------------
Mark Watts
Software Engineer
Rocket Software Inc
Waltham MA United States

Original Message:
Sent: 10-13-2021 12:35
From: Duane Gingerich
Subject: Replicating /qibm/UserData

Just wondering how people handle the replication of /qibm/UserData?

------------------------------
Duane Gingerich
SE
Able-One Systems Inc
kitchener ON Canada
------------------------------