Skip to main content
Question

Does the vertica offer a scheduler that transfer data from vertica to kafka???

  • March 17, 2021
  • 7 replies
  • 11 views

HyeontaeJu
Forum|alt.badge.img+2

Does the vertica offer a scheduler that transfer data from vertica to kafka???

7 replies

SergeB
Forum|alt.badge.img
  • Participating Frequently
  • March 17, 2021

Not around KafkaExport; you will need to schedule your exports outside of Vertica (much like you'd schedule exports to parquet, S3....).
Note that for Data Collector information, a notifier can be defined to automatically export it to Kafka at it gets generated: see: https://www.vertica.com/docs/10.1.x/HTML/Content/Authoring/KafkaIntegrationGuide/Notifiers/AutomaticallySendingNotificationsofDCTableChanges.htm


HyeontaeJu
Forum|alt.badge.img+2
  • Author
  • Participating Frequently
  • March 17, 2021

@SergeB
then if i use notifer i could send conditional data from a specific table to Kafka via KafkaExport every 5 minutes.?


SergeB
Forum|alt.badge.img
  • Participating Frequently
  • March 17, 2021

With Notifier, Vertica could send Data Collector information as soon as it gets generated (almost like a trigger). Note that this is possible only with Data Collector tables This blog might be helpful: https://www.vertica.com/blog/publish-data-collector-tables-apache-kafka/


HyeontaeJu
Forum|alt.badge.img+2
  • Author
  • Participating Frequently
  • March 18, 2021

Umm, I read the link recommended by you thank you
then, Is the Vertica cannot monitor and push general table's data to Kafka via Notifer(ex. custom table) ??
If this is not possible, how can i solve this problem?


mosheg
Forum|alt.badge.img+2
  • Participating Frequently
  • March 18, 2021

One of the options is to use a cron job as a time-based job scheduler which send conditional data from a specific table to Kafka via KafkaExport.
https://www.vertica.com/docs/10.1.x/HTML/Content/Authoring/KafkaIntegrationGuide/KafkaFunctions/KafkaExport.htm


HyeontaeJu
Forum|alt.badge.img+2
  • Author
  • Participating Frequently
  • March 18, 2021

@mosheg
then, do I need to registy KafkaExport query to the cron job??


mosheg
Forum|alt.badge.img+2
  • Participating Frequently
  • March 18, 2021

Using the DBadmin user Vertica admintools already uses the Linux cron package to schedule jobs that regularly rotate the DB logs.
See crontab -l output:

# Vertica administrator cron
# Minute    Hour     Day    Month    Day of Week            Command
   5         3        *       *         *                    /opt/vertica/oss/python3/bin/python3 -m vertica.do_logrotate &> /dev/null

So use a user with the right grants to run your script.