Hi,
I've just tried using the Vertica Spark connector as it seemed promising on papper.
However it is having issues saving data to avro files when the column is of type java.sql.Date or java.sql.Timestamp.
This is how Spark internally keeps the date or timestamp values inside a dataframe.
Here is a quick way to test this:
import java.sql.Timestamp
import org.apache.spark.sql.SaveMode
import sqlContext.implicits._
case class Person(name:String, age:Int, height:Float, lastSeen: java.sql.Timestamp)
val lastSeen = new Timestamp(1454413911000L)
val persons = List(Person("Adidas", 23, 1.72f, lastSeen))
val df = sqlContext.createDataFrame(persons)
val opts = Map(
"db" -> db,
"user" -> user,
"password" -> pwd,
"host" -> host,
"tmpdir" -> "/tmp",
"numPartitions" -> "4",
"dbschema" -> dbSchema)
df.write
.format("com.vertica.spark.datasource.DefaultSource")
.options(opts + ("table" -> "test_spark_connector"))
.mode(SaveMode.Overwrite)
.save()
The following error is being thrown:
Partition[3]: ERROR: Failed while COPYing data to Vertica. partition=3. Error message:java.lang.Exception: ERROR: S2VSaver.writeToAvroFile(): Failed writing to local avro file at path:/tmp/spark-vertica-connector-tmpfile-3492120993874597034-part-3.avro. ERROR:org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroRuntimeException: Unknown datum type java.sql.Timestamp: 2016-02-02 11:51:51.0
at com.vertica.spark.s2v.S2V$$anonfun$2.apply(S2V.scala:210)
at com.vertica.spark.s2v.S2V$$anonfun$2.apply(S2V.scala:124)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$22.apply(RDD.scala:745)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$22.apply(RDD.scala:745)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Am I missing something?