vertica spark connector

Hi Ira, 

From what I have heard, the spark connector isn't available right now since it's undergoing some bug fixes/performance improvement. 

Hi Kaurora,

thank you a lot for the reply! I see, but isn't there the old version available somewhere?

For now I didn't have any luck googling for it. 

Thanks a lot!

Hi Ira and others, are there any updates on the expected re-release of the connector, or previous versions?  I see some recent activity on https://community.dev.hpe.com/t5/Vertica-Forum/Loading-spark-dataframe-into-vertica/td-p/233482.

Best,

Sam

Any update on this? Even if there is unstable version?

Hi Sam and Schneider315,

For now I have used the SparkSql module and jdbc to read and write to Vertica.

As you've probably already seen here https://community.dev.hpe.com/t5/Vertica-Forum/Loading-spark-dataframe-into-vertica/td-p/233482 .

No news on the connector on my side - I'd be glad to get an update on that too.

Thanks,

Ira

Hello Ira,

I tried your solution it worked but only for dataframe that doesnt contain NULL in any of the rows. It errors out when there is a null present. Did you face the same issue? Do you hve any solutions in mind?

Regards,

JSON

Hi ira,

How are you connecting sparksql and Vertica with jdbc? What kind of jdbc driver are you using?

Thanks!

Hi,

For 8.0 there is a connector available for download on my.vertica.com downloads.

https://my.vertica.com/download/vertica/8-0-x/

Under the "Connector section" towards the bottom of the page.

And the documentation is here: 

https://my.vertica.com/docs/8.0.x/HTML/index.htm#Authoring/SparkConnector/ApacheSparkGuide.htm%3FTocPath%3DIntegrating%2520with%2520Apache%2520Spark%7C_____0

Thanks,
Mark

HPE Product Management

Hi,

you can get the jdbc driver from the vertica download site (https://my.vertica.com/download/vertica/client-drivers/) and then you can acces your data like this:

String theQueryToGetTheData = "Select column1, column2 from MyTable";
String myconnectionurl = "jdbc:vertica://127.0.0.1:5433/DBNAME?user=myusr&password=mypass";

Dataset<Row> df = session.read().format("jdbc").option("driver", "com.vertica.jdbc.Driver").option("urll", myconnectionurl)
				.option("dbtable", theQueryToGetTheData).load();

You have to make sure that the jdbc jar is available to spark, either package it in your application jar or provide it via the --jars option.

Concerning the null issue during the writing of dataframes I started playing with a custom sql dialect but I think the connector mentioned by Mark should help.

Haven't tried it yet, but it seems that it supports only Spark 1.6.

Cheers,

Ira