Skip to main content

Vertica connector for Spark

  • January 23, 2017
  • 3 replies
  • 19 views
Forum|alt.badge.img+2

Hey,

I want to integrate a Spark job with a Vertica DB and used the hpe-spark-connector.

It worked when I used spark 1.6, but now we want to upgrade to 2.1 and I've noticed that the connector doesn't support that version of Spark yet.

 

I wanted to know if there is planned release to upgrade the connector to work with spark 2.0+?

And I've also noticed it's pretty hard to even find the connector jar, is the project discontinued for now?

 

Thanks in advance for anyone helping me :)

    3 replies

    jlu
    • jlu
    • New Participant
    • February 15, 2017

    Hello,
    have you tried to write your final spark dataframe into parquet or orc files and read them with vertica ?

    https://my.vertica.com/docs/8.0.x/HTML/index.htm#Authoring/HadoopIntegrationGuide/NativeFormats/ReadingNativeFormats.htm

    jlu
    • jlu
    • New Participant
    • March 21, 2017

    It seems that the best case scenario is that the connector will be shipped with vertica 8.1 in the coming couple of months :(

    Connyrt-emp1
    Forum|alt.badge.img
    • Connyrt-emp1
    • Participating Frequently
    • March 23, 2017

    Thanks for posting your question and for your feedback. We hope to support Spark 2.0 with Scala 2.11 on the next Vertica release, a new jar would be available on my.vertica.com. Please make sure to download the correct jar for your environment, the name of the connector specifies the versions of vertica, spark and scala that should be used.

    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement