kafka integration with vertica in sandbox

Hi Mahesh,

    What exactly seems to be the issue?

-Chris

I'm seeing two things that might be creating issue for you.

First, in you scheduler, you are not defining the brokers.  You should add --brokers (server):9092

Secondly in the topic you should add the --config-schema flag, in your case, it should be --config-schema myScheduler

-Chris

Hmmm.. What is the exact error message?

Thanks,

-Chris

the eror is " no such agrument to config"...

Ok, I think I see the problem.  First, the brokers file doesn't have the -- before it.  But since you are going local host, you can ignore it and it will assume local host as the default.  Command should be:

/opt/vertica/packages/kafka/bin/vkconfig scheduler --add --config-schema myScheduler --operator dbadmin

now i am getting error for this command

/opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.kafka_tgt --rejection-table public.kafka_rej --topic messages --num-partitions 4

The error is " The schema kafka_config doesn't exits.......

Topic add requires the --config-schema as well.  Command should be:

/opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.kafka_tgt --rejection-table public.kafka_rej --topic messages --num-partitions 4  --config-schema myScheduler

-Chris

now i am getting error for commad
 
/opt/vertica/packages/kafka/bin/vkconfig launch --conf configFile.propertis
 
the error is " expection in the thread "main""   java error...

Are you using the conf file?  if not, maybe try just a /vkconfig launch &

In older versions, the launch command also needed the --config-schema  flag as well, however documentation does not indicate that.  You might want to also try /vkconfig launch --config-schema myScheduler &

-Chris

Here is the out put..

[dbadmin@localhost bin]$ ./vkconfig launch &
[1] 16212
[dbadmin@localhost bin]$ Exception in thread "Main" java.sql.SQLException: [Vertica][VJDBC](4650) ERROR: Schema "kafka_config" does not exist
at com.vertica.util.ServerErrorData.buildException(Unknown Source)
at com.vertica.dataengine.VQueryExecutor.executeSimpleProtocol(Unknown Source)
at com.vertica.dataengine.VQueryExecutor.execute(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeNoParams(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeQuery(Unknown Source)
at com.vertica.solutions.kafka.cli.CLIUtil.validateCopySyntaxes(CLIUtil.java:34)
at com.vertica.solutions.kafka.Launcher.run(Launcher.java:69)
at com.vertica.solutions.kafka.Launcher.main(Launcher.java:136)
Caused by: com.vertica.support.exceptions.ErrorException: [Vertica][VJDBC](4650) ERROR: Schema "kafka_config" does not exist
... 8 more

[1]+ Exit 1 ./vkconfig launch
[dbadmin@localhost bin]$ ./vkconfig launch -- config-schema myScheduler &
[1] 16296
[dbadmin@localhost bin]$ No such argument

So the first error indicates that it hinks the schema is kafka_config

And the where you issue the command with the --config-schema you placed a space between -- and config.  Try this:

./vkconfig launch --config-schema myScheduler &

-chris

i got this message now..

[dbadmin@localhost bin]$ ./vkconfig launch --config-schema myScheduler &
[1] 18002

 

after this do i need to remove broker information in this script
COPY
schema.target_
table
SOURCE
KafkaSource
(stream='topic1|1|1,topic2|2|2',
brokers='host1:9092,host2:9092',duration=
INTERVAL'timeslice',stop_
on_
eof=TRUE,
eof_
timeout=
INTERVAL'timeslice')
PARSER
KafkaJSONParser
(flatten_
arrays=False,
flatten_
maps=False)
REJECTED
DATA
AS
TABLE
schema.rejection_
table
TRICKLE

Hi,

    At this point the data with in the messages topic on kafka should be loading in microbatches into the public.kafka_tgt table in Vertica.  

you can check this by going into vsql and issuing this command:

Select count(*) from public.kafka_tgt;

-Chris

dbadmin=> Select count(*) from public.kafka_tgt;
 count
-------
     0
(1 row)

dbadmin=> COPY schema.target_table SOURCE KafkaSource (stream='topic1|1|1,topic2|2|2',brokers='host1:9092,host2:9092',duration=INTERVAL'timeslice',stop_on_eof=TRUE,eof_timeout=INTERVAL'timeslice') PARSER KafkaJSONPars(flatten_arrays=False,flatten_maps=False) REJECTED DATA AS TABLE schema.rejection_table TRICKLE;
ERROR 4856:  Syntax error at or near "schema" at character 6
LINE 1: COPY schema.target_table SOURCE KafkaSource (stream='topic1|...
             ^
dbadmin=>