Skip to main content
Question

Cluster rebalance & VBR Replicate question

  • January 18, 2019
  • 2 replies
  • 18 views

mkheir
Forum|alt.badge.img+1
  • Participating Frequently

Hello Experts,

I have two clusters of three nodes each that I need to keep in sync using vbr replicate.

I did the following:
- On the master cluster : Added two nodes + Rebalance
- On the replication cluster : Added two nodes

Then I run vbr replicate from the master cluster to the 2nd one and re-run the replication multiple times (without adding data):
Replication 1 Approximate bytes to copy: 8710674037 of 11005070916 total.
Replication 2 Approximate bytes to copy: 131287023 of 11003237414 total.
Replication 3 Approximate bytes to copy: 0 of 11003237414 total.

My question : Why the replication 2 had still data to replicate while no new data was added in mean time?

I've run multiple tests and found the same behaviour, second replication move data even if no data was added to the any cluster, only the third one have nothing to do.

I also did the same by adding only one node then rebalance and got the below.
Replication 1 Approximate bytes to copy: 9116700248 of 10398237944 total. ==> OK
Replication 2 Approximate bytes to copy: 81689996 of 10396713131 total. ==> Not sure I understand! I expected to see 0 bytes to copy.
Replication 3 Approximate bytes to copy: 0 of 10396713131 total.

Curious to know what the 2nd replicate is doing.

Note: I made sure that rebalance is complete before running the vbr replicate tool. there was also no running operation in TUPLE_MOVER_OPERATIONS table.

FYI: I m running these tests on my small environment to better schedule and plan a rebalance operation followed by a replication on a 50Tb cluster.

Many thanks for your help.
Mkheir

2 replies

mkheir
Forum|alt.badge.img+1
  • Author
  • Participating Frequently
  • January 18, 2019

Thank you for your answer,

I understand that a rebalance will modify ROSs and then need to be copied to the corresponding nodes during the replicate.

Imagine the following steps : rebalance ==> replication ==> replication
the 1st replication copy files, and that is understood, as you described,
I see also the 2nd replication moving data, here I m not sure what happends behind the scenes so data need to be moved.

Is there anything to monitor after rebalance complete before running the replication?

Many thanks.


praveshbhardwaj
Forum|alt.badge.img+1
  • Participating Frequently
  • January 21, 2019

Were there any completed mergeouts between the two replication attempts?