Skip to main content
Question

Is there a way to return the number of messages sent to Kafka using KafkaExport?

  • Apr 29, 2021 10:39 AM UTC
  • 4 replies
  • 13 views

HyeontaeJu
Forum|alt.badge.img+2

Is there a way to return the number of messages sent to Kafka using KafkaExport?

4 replies

Jim_Knicely
Forum|alt.badge.img+2
  • Participating Frequently
  • Apr 29, 2021 12:42 PM UTC

You should be able to use KAFKACAT to list the messages sent to Kafka from Vertica.

Something like:

./kafkacat -C -t <Topic> -b <Broker>

See:
https://www.vertica.com/docs/latest/HTML/Content/Authoring/KafkaIntegrationGuide/TroubleShooting/UsingKafkacatToTroubleShootIssues.htm


SergeB
Forum|alt.badge.img
  • Participating Frequently
  • Apr 29, 2021 12:46 PM UTC

The number of messages produced to Kafka should match the number of rows returned by the query used in KafkaExport MINUS any error reported by KafkaExport.


HyeontaeJu
Forum|alt.badge.img+2
  • Author
  • Participating Frequently
  • Apr 29, 2021 12:58 PM UTC

@Jim_Knicely, @SergeB
Then,, Is it impossible return the number of message sent to Kafka topic immediately after KafkaExport query executing?


SergeB
Forum|alt.badge.img
  • Participating Frequently
  • Apr 29, 2021 2:59 PM UTC

You could determine the number of rows from your select statement (select count(*)....) immediately before or after calling KafkaExport