When concurrent Spark jobs are invoked, the Vertica Spark connector would fail to work with below error message - Caused by: com.vertica.spark.util.error.ConnectorException: Failed to create a valid column list for the write operation due to mismatch with the existing table. JDBC failure when trying to retrieve schema Connection to the JDBC source is down or invalid. Please ensure that the JDBC source is running properly. at com.vertica.spark.util.error.ErrorHandling$.logAndThrowError(ErrorHandling.scala:78) at com.vertica.spark.datasource.v2.VerticaBatchWrite.commit(VerticaDatasourceV2Write.scala:92) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:422) ... 84 more
Environment
- Vertica Analytics database: versions 11.1.1, 12.0.4
- Scala - 2.12.12
- Spark Connector - 3.3.5
Situation
When concurrent Spark jobs are invoked, the Vertica Spark connector would fail to work with an error message
