Skip to main content

[vkconfig] Microbatch to dedicate and distribute to 'Kafka partition - Vertica node'

  • January 15, 2019
  • 0 replies
  • 2 views

keehoonlee
Forum|alt.badge.img
Hi,

I would like to know can Vertica vkconfig be set microbatch on Vertica node to dedicate specific Kafka partitions. During several test, I keep seeing that vkconfig reads source file when started and launch with all the partitions on every nodes.

What I would like to do here is that scheduler on a node to be distributed and dedicated to specific Kafka partition.

For example, if there are 3 node Vertica cluster and Kafka with 6 partition on a topic,
Kafka partition 0,1 ----> load to node 1 (dedicated)
Kafka partition 2,3 ----> load to node 2 (dedicated)
Kafka partition 4,5 ----> load to node 3 (dedicated)

It is same logic with doing so as below with COPY statement:
-- partition 0
COPY "public"."vendor_dimension" SOURCE KafkaSource(stream='test1|0|-2', brokers=...