Skip to main content
Question

If one have 4 Kafka topic source and 4 vertica tables could he gather all into one scheduler?

  • November 10, 2019
  • 1 reply
  • 6 views

mosheg
Forum|alt.badge.img+2
  • Participating Frequently
Please share an example

1 reply

mosheg
Forum|alt.badge.img+2
  • Author
  • Participating Frequently
  • November 27, 2019

Thank you Serge Bonte for your answer:
If customer have one topic generating 70% of the traffic and the other 3 30%, it might be a good idea to have two schedulers.
When you define a scheduler you set a time frame (e.g scheduler creates microbatches to ingest for 2 minutes).
That time frame gets divided equally between topics (assuming one target/microbatch per topic) so topic 1 would get same duration as the other 3.
Given that topic1 gets 70% of data, it could fall behind unless you allocate it its own scheduler.
In general you want to avoid too wide of a production range between topics in a given scheduler.
Attaching an example.
Each script creates 1 scheduler, one with 1 topic and the other with 3 topics.
Those are derived from an example in /opt/vertica/packages/kafka/examples