for example, from log:
COPY "public"."events" SOURCE
KafkaSource(stream='events|0|1590085,events|1|1586923,events|2|-2',
brokers='1.2.3.4:9092', duration=interval '9872 milliseconds',
stop_on_eof=true, executionparallelism=1 ) PARSER KafkaJSONParser( )
REJECTED DATA AS TABLE public.kafka_rej6 DIRECT NO COMMIT
But:
/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 127.0.0.1:9092 --topic events --time -1
events:0:1590085
events:1:1586923
events:2:1597176
As you can see, vertica do not consume partition #2 at all. I trying to set --start-offset without results. BTW many other strange things happened with kafka package, like suddenly node downs, but here I whant to know, why my partitions did not consuming.
Yes, kafka_rej6 rejection table is empty, kafka's server.log not say anyting as same as vertica's dbLog.
Now look:
COPY "public"."events" SOURCE KafkaSource(stream='events|2|-2',
brokers='1.2.3.4:9092', duration=interval '9872 milliseconds',
stop_on_eof=true, executionparallelism=1 ) PARSER KafkaJSONParser( )
REJECTED DATA AS TABLE public.kafka_rej6 DIRECT NO COMMIT;
Rows Loaded
-------------
204505
If I keep only one partition in stream Vertica is succesfully consume messages. WHY?
I'm very disappointed with integration spended almost two weeks on it without any result. No any logs and silent running make it impossible to debug.
