Skip to main content

If Vertica kafka Utilities support parsing Avro Enum type data?

  • March 31, 2016
  • 1 reply
  • 12 views
Jonna

In my avro schema, there are two enum type fields defined, like below.

 

"fields" : [ {"name" : "eventType", "type": {"name" : "EventType", "type": "enum", "symbols" : ["A", "C"]}}, {"name": "packetType", "type": {"name": "PacketType", "type": "enum", "symbols" : ["IIPMS", "SMS"]}}]

 

I use a kafka schedular to parse messages from kafka with KafkaAVROParser, but the values can't be persisted into db table.

 

Untitled picture.png

 

I want to know if vertica kafka utility support the Avro enum type data parsing and persistence. If no, I will change the enum type to string. If yes, why the data value can't be persisted correctly, is it because I miss some important configurations?

    1 reply

    Cornelio
    • Cornelio
    • Participating Frequently
    • April 8, 2016

    Hi

     

    Yes, KafkaAvroParser supports Avro Enum type.

    How are you writing the messages? see: https://community.dev.hpe.com/t5/Vertica-Forum/Loading-AVRO-data-using-KAFKA-Parser/m-p/235240

    Does your schema contains siteID and alarmString so those values were also loaded using KafkaAvroParser?

     

    E.g. Enum type is loaded using KafkaAvroParser with the following Avro schema (it is an Avro Record containing String fields and also the Enum fileds)

     

    {"type":"record","name":"Users","namespace":"test","fields":[{"name":"username","type":"string"},{"name":"age","type":"int"},{"name":"address","type":"string"},{"name" : "eventType", "type": {"name" : "EventType", "type": "enum", "symbols" : ["A", "C"]}}, {"name": "packetType", "type": {"name": "PacketType", "type": "enum", "symbols" : ["IIPMS", "SMS"]}}],"doc":"Record with enums"}

     

    So target FlexTable contains

    SELECT maptostring(__raw__) FROM myTargetTable;
    maptostring
    -----------------------------------------------------------------------------------------------------------------------------------------------------
    {
    "EventType" : "A",
    "PacketType" : "SMS",
    "__name__" : "Users",
    "address" : "Street 1 #54",
    "age" : "30",
    "username" : "John"
    }

    (1 row)

    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement