Skip to main content
Have you ever stopped to look at the jobs running in the iCluster subsystems and ever wondered what they are doing? Here's a breakdown of the jobs on the source and the target nodes, and a brief description of what they do.

One of the most important jobs to monitor when starting iCluster is the DMCHATL job, it runs on both nodes, it's the TCP/IP listener job that ensures the nodes are communicating with each other. 

If we have any kinds of issues with replication, it's always good to look into the group jobs in the subsystems on both the source and target nodes. Check the data journal scrape job (HADDJS) on source and the data journal scrape receiver job (HADDJSR) on the target. 

Thanks.


On Source node - iCluster jobs
JOB FUNCTION  DESCRIPTION No. OF JOBS
DMCHATL DMCHATL TCP/IP listening job 1
DMM00000MO DMCHAMO Monitor Information out 1
DMSXXXXXCI DMCHACI Control Information in many
DMSXXXXXCO DMCHACO Control Information out many
DMSXXXXXDI DMCHADI Data in many
DMSXXXXXDO DMCHADO Data out many
HAMON OMSM Processes monitor requests on source 1
DM_DEFMGR DM_DEFMGR Definition Manager 1 per active node
DM_MONITOR DM_MONITOR Definition Manager Help job 1 on each node
DM_GRPSTS DM_GRPSTS Group Status 1 on each node
TMONITOR OMTM Monitor   1 on each node
DM_GUIGENT DM_GUIGENT GUI Agent 1 on each node
DM_MONDATA DM_MONDATA Central performance data store 1 on Master node
DMMONHA DM_MONHA Replication monitor 1 on Master node
DMDLTJRCV DMDLTJRCV Journal Management 1 per journal

On Source node - Group jobs 
groupname OMGROUPJOB Scrapes audit journal for object only 1 per group
groupname HADDAS HA Data audit scrape - scrapes audit journal for data information 1 per group
groupname HADDJS HA Data journal scrape - scrapes data journals 1 per journal per group
groupname HADSFP HA Data save file process 1 per journal per group


On Target node - iCluster jobs
JOB FUNCTION  DESCRIPTION No. OF JOBS
DMCHATL DMCHATL TCP/IP listening job 1
DMM00000MO DMCHAMO Monitor Information out 1
DMSXXXXXCI DMCHACI Control Information in many
DMSXXXXXCO DMCHACO Control Information out many
DMSXXXXXDI DMCHADI Data in many
DMSXXXXXDO DMCHADO Data out many
HAMON OMSM Processes monitor requests on source 1
DM_DEFMGR DM_DEFMGR Definition Manager 1 per active node
DM_MONITOR DM_MONITOR Definition Manager Help job 1 on each node
DM_GRPSTS DM_GRPSTS Group Status 1 on each node
TMONITOR OMTM Monitor   1 on each node
DM_GUIGENT DM_GUIGENT GUI Agent 1 on each node
DMDLTJRCV DMDLTJRCV Journal Management 1 on Master node
DMCHGAPMNT DMCHGAPMNT Access path maintenance 1 on Master node


On Target node - Group jobs
groupname OMRCV Receives object only entries & sends to stage store 1 per group
groupname OMTARGET Applies object changes 1 per group
groupname HADDASR HA Data audit scrape receiver - receives audit journal for data information from HADDAS job 1 per journal per group
groupname HADDJSR HA Data journal scrape receiver - receives data journal information from HADDJS job and sends it to stage area 1 per journal per group
groupname HADSFPR HA Data save file process receiver - receives save file from HADSFP and sends it to stage area  
groupname HADTUP HA Transaction Update process - match and merge process  




------------------------------
Don Broadbridge
Senior CSE
Rocket Internal - All Brands
Denver CO United States
------------------------------
Have you ever stopped to look at the jobs running in the iCluster subsystems and ever wondered what they are doing? Here's a breakdown of the jobs on the source and the target nodes, and a brief description of what they do.

One of the most important jobs to monitor when starting iCluster is the DMCHATL job, it runs on both nodes, it's the TCP/IP listener job that ensures the nodes are communicating with each other. 

If we have any kinds of issues with replication, it's always good to look into the group jobs in the subsystems on both the source and target nodes. Check the data journal scrape job (HADDJS) on source and the data journal scrape receiver job (HADDJSR) on the target. 

Thanks.


On Source node - iCluster jobs
JOB FUNCTION  DESCRIPTION No. OF JOBS
DMCHATL DMCHATL TCP/IP listening job 1
DMM00000MO DMCHAMO Monitor Information out 1
DMSXXXXXCI DMCHACI Control Information in many
DMSXXXXXCO DMCHACO Control Information out many
DMSXXXXXDI DMCHADI Data in many
DMSXXXXXDO DMCHADO Data out many
HAMON OMSM Processes monitor requests on source 1
DM_DEFMGR DM_DEFMGR Definition Manager 1 per active node
DM_MONITOR DM_MONITOR Definition Manager Help job 1 on each node
DM_GRPSTS DM_GRPSTS Group Status 1 on each node
TMONITOR OMTM Monitor   1 on each node
DM_GUIGENT DM_GUIGENT GUI Agent 1 on each node
DM_MONDATA DM_MONDATA Central performance data store 1 on Master node
DMMONHA DM_MONHA Replication monitor 1 on Master node
DMDLTJRCV DMDLTJRCV Journal Management 1 per journal

On Source node - Group jobs 
groupname OMGROUPJOB Scrapes audit journal for object only 1 per group
groupname HADDAS HA Data audit scrape - scrapes audit journal for data information 1 per group
groupname HADDJS HA Data journal scrape - scrapes data journals 1 per journal per group
groupname HADSFP HA Data save file process 1 per journal per group


On Target node - iCluster jobs
JOB FUNCTION  DESCRIPTION No. OF JOBS
DMCHATL DMCHATL TCP/IP listening job 1
DMM00000MO DMCHAMO Monitor Information out 1
DMSXXXXXCI DMCHACI Control Information in many
DMSXXXXXCO DMCHACO Control Information out many
DMSXXXXXDI DMCHADI Data in many
DMSXXXXXDO DMCHADO Data out many
HAMON OMSM Processes monitor requests on source 1
DM_DEFMGR DM_DEFMGR Definition Manager 1 per active node
DM_MONITOR DM_MONITOR Definition Manager Help job 1 on each node
DM_GRPSTS DM_GRPSTS Group Status 1 on each node
TMONITOR OMTM Monitor   1 on each node
DM_GUIGENT DM_GUIGENT GUI Agent 1 on each node
DMDLTJRCV DMDLTJRCV Journal Management 1 on Master node
DMCHGAPMNT DMCHGAPMNT Access path maintenance 1 on Master node


On Target node - Group jobs
groupname OMRCV Receives object only entries & sends to stage store 1 per group
groupname OMTARGET Applies object changes 1 per group
groupname HADDASR HA Data audit scrape receiver - receives audit journal for data information from HADDAS job 1 per journal per group
groupname HADDJSR HA Data journal scrape receiver - receives data journal information from HADDJS job and sends it to stage area 1 per journal per group
groupname HADSFPR HA Data save file process receiver - receives save file from HADSFP and sends it to stage area  
groupname HADTUP HA Transaction Update process - match and merge process  




------------------------------
Don Broadbridge
Senior CSE
Rocket Internal - All Brands
Denver CO United States
------------------------------
Hi, 
When the dmchatl loops, would you recommend to 1)end the job and restart it using strhatcp
Or
2)recommend to restart the entire subsystem?

------------------------------
ZQ L
Manager | Shared Infrastructure Services
United Overseas Bank Limited
Singapore Singapore
------------------------------
Hi, 
When the dmchatl loops, would you recommend to 1)end the job and restart it using strhatcp
Or
2)recommend to restart the entire subsystem?

------------------------------
ZQ L
Manager | Shared Infrastructure Services
United Overseas Bank Limited
Singapore Singapore
------------------------------
Hi ZQ,

I'd recommend trying the first command and if you continued to have issues, a recycle on the XDMCLUSTER subsystem. If you kept seeing, issues with the DMCHATL looping or dropping, I'd specifically go looking at running some connectivity tests (HAPNGTCP and NETSTAT to start with).

Apologies for tardy response, had some customer project tasks to respond too.

Regards.

------------------------------
Don Broadbridge
Senior CSE
Rocket Internal - All Brands
Denver CO United States
------------------------------
Hi, 
When the dmchatl loops, would you recommend to 1)end the job and restart it using strhatcp
Or
2)recommend to restart the entire subsystem?

------------------------------
ZQ L
Manager | Shared Infrastructure Services
United Overseas Bank Limited
Singapore Singapore
------------------------------
Hi ZQ,

I'd also suggest you check the current version of iCluster with command DMSYSINF.  The condition you describe (DMCHATL looping or hanging) occurred first in releases a few years ago when customers were advancing OS levels but not keeping iCluster at current available versions and fix levels.  To help keep iCluster running smoothly, upgrade to at least iCluster v8.3 and Fix level 8.3.2.1.  This will give you new features and most reliable environments running on the IBM i V6.1 or higher.

Regards,

------------------------------
Mark Watts
Software Engineer
Rocket Software Inc
Waltham MA United States
------------------------------
Have you ever stopped to look at the jobs running in the iCluster subsystems and ever wondered what they are doing? Here's a breakdown of the jobs on the source and the target nodes, and a brief description of what they do.

One of the most important jobs to monitor when starting iCluster is the DMCHATL job, it runs on both nodes, it's the TCP/IP listener job that ensures the nodes are communicating with each other. 

If we have any kinds of issues with replication, it's always good to look into the group jobs in the subsystems on both the source and target nodes. Check the data journal scrape job (HADDJS) on source and the data journal scrape receiver job (HADDJSR) on the target. 

Thanks.


On Source node - iCluster jobs
JOB FUNCTION  DESCRIPTION No. OF JOBS
DMCHATL DMCHATL TCP/IP listening job 1
DMM00000MO DMCHAMO Monitor Information out 1
DMSXXXXXCI DMCHACI Control Information in many
DMSXXXXXCO DMCHACO Control Information out many
DMSXXXXXDI DMCHADI Data in many
DMSXXXXXDO DMCHADO Data out many
HAMON OMSM Processes monitor requests on source 1
DM_DEFMGR DM_DEFMGR Definition Manager 1 per active node
DM_MONITOR DM_MONITOR Definition Manager Help job 1 on each node
DM_GRPSTS DM_GRPSTS Group Status 1 on each node
TMONITOR OMTM Monitor   1 on each node
DM_GUIGENT DM_GUIGENT GUI Agent 1 on each node
DM_MONDATA DM_MONDATA Central performance data store 1 on Master node
DMMONHA DM_MONHA Replication monitor 1 on Master node
DMDLTJRCV DMDLTJRCV Journal Management 1 per journal

On Source node - Group jobs 
groupname OMGROUPJOB Scrapes audit journal for object only 1 per group
groupname HADDAS HA Data audit scrape - scrapes audit journal for data information 1 per group
groupname HADDJS HA Data journal scrape - scrapes data journals 1 per journal per group
groupname HADSFP HA Data save file process 1 per journal per group


On Target node - iCluster jobs
JOB FUNCTION  DESCRIPTION No. OF JOBS
DMCHATL DMCHATL TCP/IP listening job 1
DMM00000MO DMCHAMO Monitor Information out 1
DMSXXXXXCI DMCHACI Control Information in many
DMSXXXXXCO DMCHACO Control Information out many
DMSXXXXXDI DMCHADI Data in many
DMSXXXXXDO DMCHADO Data out many
HAMON OMSM Processes monitor requests on source 1
DM_DEFMGR DM_DEFMGR Definition Manager 1 per active node
DM_MONITOR DM_MONITOR Definition Manager Help job 1 on each node
DM_GRPSTS DM_GRPSTS Group Status 1 on each node
TMONITOR OMTM Monitor   1 on each node
DM_GUIGENT DM_GUIGENT GUI Agent 1 on each node
DMDLTJRCV DMDLTJRCV Journal Management 1 on Master node
DMCHGAPMNT DMCHGAPMNT Access path maintenance 1 on Master node


On Target node - Group jobs
groupname OMRCV Receives object only entries & sends to stage store 1 per group
groupname OMTARGET Applies object changes 1 per group
groupname HADDASR HA Data audit scrape receiver - receives audit journal for data information from HADDAS job 1 per journal per group
groupname HADDJSR HA Data journal scrape receiver - receives data journal information from HADDJS job and sends it to stage area 1 per journal per group
groupname HADSFPR HA Data save file process receiver - receives save file from HADSFP and sends it to stage area  
groupname HADTUP HA Transaction Update process - match and merge process  




------------------------------
Don Broadbridge
Senior CSE
Rocket Internal - All Brands
Denver CO United States
------------------------------

Dear Mr. Broadbridge

In my iCluster 9.1.1 in the source system, I see a job named HAREACT running PGM-HAREACT that is not mentioned i n your list. May I ask what this job is for?

Thanks. 



------------------------------
Satid Singkorapoom
IBM i SME
Rocket Forum Shared Account
------------------------------

Dear Mr. Broadbridge

In my iCluster 9.1.1 in the source system, I see a job named HAREACT running PGM-HAREACT that is not mentioned i n your list. May I ask what this job is for?

Thanks. 



------------------------------
Satid Singkorapoom
IBM i SME
Rocket Forum Shared Account
------------------------------

Hi Satid. Any time you see OOS errors and use the option to activate them, these jobs will appear in the SBS. Activation of OOS can take very little time or much longer if the objects are large.

Regards,

D Broadbridge

  



------------------------------
don broadbridge
Principal Consultant
Rocket Internal - All Brands
Denver CO US
------------------------------

Hi Satid. Any time you see OOS errors and use the option to activate them, these jobs will appear in the SBS. Activation of OOS can take very little time or much longer if the objects are large.

Regards,

D Broadbridge

  



------------------------------
don broadbridge
Principal Consultant
Rocket Internal - All Brands
Denver CO US
------------------------------

Dear Mr. Broadbridge

Thank for your response but I have not run OOS activation for a long time now and this makes me curious. When I looked into HAREACT's job log, I found messages that it was doing automatic activation for several suspended objects which enlightened me on this. 



------------------------------
Satid Singkorapoom
IBM i SME
Rocket Forum Shared Account
------------------------------

Dear Mr. Broadbridge

Thank for your response but I have not run OOS activation for a long time now and this makes me curious. When I looked into HAREACT's job log, I found messages that it was doing automatic activation for several suspended objects which enlightened me on this. 



------------------------------
Satid Singkorapoom
IBM i SME
Rocket Forum Shared Account
------------------------------

Hi Satid,

Thanks for the reply.  I neglected to mention that iCluster will try to activate suspended objects (you will see Auto Rcvy column with *YES when working with object status, option '8') with retry attempts before it marks the object as S/O, after number of maximum attempts are reached. 



------------------------------
don broadbridge
Principal Consultant
Rocket Internal - All Brands
Denver CO US
------------------------------
Terms & ConditionsAccessibility statement