Skip to main content
Question

Is there a way to return the number of messages sent to Kafka using KafkaExport?

  • April 29, 2021
  • 4 replies
  • 6 views

HyeontaeJu
Forum|alt.badge.img+2

Is there a way to return the number of messages sent to Kafka using KafkaExport?

4 replies

Jim_Knicely
Forum|alt.badge.img+2
  • Participating Frequently
  • April 29, 2021

You should be able to use KAFKACAT to list the messages sent to Kafka from Vertica.

Something like:

./kafkacat -C -t <Topic> -b <Broker>

See:
https://www.vertica.com/docs/latest/HTML/Content/Authoring/KafkaIntegrationGuide/TroubleShooting/UsingKafkacatToTroubleShootIssues.htm


SergeB
Forum|alt.badge.img
  • Participating Frequently
  • April 29, 2021

The number of messages produced to Kafka should match the number of rows returned by the query used in KafkaExport MINUS any error reported by KafkaExport.


HyeontaeJu
Forum|alt.badge.img+2
  • Author
  • Participating Frequently
  • April 29, 2021

@Jim_Knicely, @SergeB
Then,, Is it impossible return the number of message sent to Kafka topic immediately after KafkaExport query executing?


SergeB
Forum|alt.badge.img
  • Participating Frequently
  • April 29, 2021

You could determine the number of rows from your select statement (select count(*)....) immediately before or after calling KafkaExport