We tested performance of our cluster by making one of the node down but merge jobs got failed..
The thing is same jobs got successful initially after an hour of node down.
But same jobs got failed after 5 hours since node went down.
We cannot determine the exact reason. Can someone help us.
Here are vertica logs for the same-
2021-11-20 00:16:28.128 DistCall Dispatch:7f25d8cc3700-190000007c282bd [TM] updateCPE: Updating projection tracking_package_history_temp20160330231749_v2_b1 CPE to 0x2299401 from 0xffffffffffffffff for wos 0x210000032b4c648
2021-11-20 00:16:28.128 DistCall Dispatch:7f25d8cc3700-190000007c282bd [TM] updateCPE: Updating projection tracking_package_history_temp20160330231749_v2_b0 CPE to 0x2299401 from 0xffffffffffffffff for wos 0x210000032b4c600
2021-11-20 00:16:28.136 Init Session:7f212ffff700-a00000064ba36d [EE] plan_mergeout: after processing plan
2021-11-20 00:16:28.136 DistCall Dispatch:7f25d8cc3700 [Txn] Commit Complete: Txn: 190000007c282bd at epoch 0x2299402 and new global catalog version 59476569
2021-11-20 00:16:28.136 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) TM mergeout: Total row count = 97, Total delete row count = 3
2021-11-20 00:16:28.136 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) Executing the mergeout old replay delete plan
2021-11-20 00:16:28.150 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) End of executing the mergeout old replay delete plan
2021-11-20 00:16:28.150 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) Dropping source ROSs:
2021-11-20 00:16:28.150 Init Session:7f212ffff700-a00000064ba36d [EE] 45036012777134781
2021-11-20 00:16:28.150 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) Mergeout: Updating checkpoint epoch
2021-11-20 00:16:28.151 Init Session:7f212ffff700-a00000064ba36d [TM] makeProjectionCPECurrent: Updating CPE for proj 'order_refund_temp20180813170200_b0' to current
2021-11-20 00:16:28.152 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) Mergeout: CPE of proj oms.order_refund_temp20180813170200_b0 set to 36279298
2021-11-20 00:16:28.154 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) Mergeout projection oms.order_refund_temp20180813170200_b0 done
2021-11-20 00:16:28.179 Init Session:7f212ffff700-a00000064ba36d [TM] ThreadId = 0, Projection chosen = oms.order_refund_temp20180813170200_b0, REsType = 3, RECnt = 1, stratumNo = 0, partnKey =
2021-11-20 00:16:28.179 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) Mergeout projection oms.order_refund_temp20180813170200_b0 startEpoch = 36277926 endEpoch = 36277926
2021-11-20 00:16:28.179 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) Mergeout ROS list (from TMService):
2021-11-20 00:16:28.179 Init Session:7f212ffff700-a00000064ba36d [EE] 45036012777134731
2021-11-20 00:16:28.179 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) Mergeout ROS list
2021-11-20 00:16:28.179 Init Session:7f212ffff700-a00000064ba36d [EE] 45036012777134731
2021-11-20 00:16:28.179 Init Session:7f212ffff700-a00000064ba36d [EE] 1 ROSes should be merged
2021-11-20 00:16:28.180 Init Session:7f212ffff700-a00000064ba36d [EE] (a00000064ba36d) Executing the mergeout plan