Skip to main content
Question

UDFilter with KafkaSource

  • November 5, 2019
  • 1 reply
  • 6 views
dochernyshov

Hello everyone!
Are there any best practices of filtering json messages from kafka?
I implemented UD Filter in C ++ (like SearchAndReplace - https://github.com/vertica/UDx-Examples/blob/master/Java-and-C++/FilterFunctions/SearchAndReplaceFilter.cpp) - works very slowly.
SearchAndReplace (AS IS) with option reject_on_empty_keys - not suitable, because a lot of rejects.
Maybe someone has already solved a similar problem?

    1 reply

    Bryan_H
    Forum|alt.badge.img+2
    • Bryan_H
    • Participating Frequently
    • November 7, 2019

    Hi, are you able to share more details? For example, what does the COPY command look like, are there other filters or parsers inline? What type of filtering is needed - check for JSON key/values, edit JSON key/values, etc.? Can the algorithm be parallelized, or maybe written in Java instead since Java has JSON libraries like Gson that may already be optimized for your project?

    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement