Rreading data inti Vertica using StreamSets

Question

Hi,
I am trying to INSERT Json files into Vertica using Streamsets. I created a table in Vertica having column names like the field names in the json file. They can be NULL. The pipeline runs and shows the correct number of records processed, however, the records are not written to Vertica and the pipeline never stops, but no errors occur. I basically the same setting as in https://www.vertica.com/kb/StreamSetsCG/Content/Partner/StreamSetsCG.htm
 How can I see/test what the problem is? Any idea?

twall · Answer

There's many different ways to get JSON data into Vertica. Sometimes Vertica will doing the processing of JSON directly, other times it will happen upstream from Vertica in a third-party tool like StreamSets.

The first step to understanding what is going on is to understand which of the several approaches is being utilized here. You probably want to start by reading the vertica.log for the node that received the JDBC connections from StreamSets. You can also check the v_monitor.query_requests table to see the recent commands that Vertica has processed and v_monitor.error_messages for recent errors.

You will hopefully see some evidence of StreamSets running commands somewhere across those three places. Once you know specifically how StreamSets is interfacing with Vertica you can then troubleshoot further by focusing on the particular approach it is using. If you can share some of the queries or errors you see I can try to provide additional advice depending on what you find.

Sign up

Please log in or register:

Welcome to the Rocket Forum!

Please log in or register: