Our internal Vertica installation has been up and down a lot over the past few days. Three days ago our deadlock detection went off:
task-report-vertica_refresh id=321 state=BLOCKED *DEADLOCKED* - waiting to lock <0x517358cf> (a java.util.ArrayList) owned by vertica connection closer id=69709 - holding lock <0x440e2406> (a java.util.concurrent.ThreadPoolExecutor$Worker) - holding lock <0x7ee61c7e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at com.vertica.jdbc.common.SConnection.removeStatement(Unknown Source) at com.vertica.jdbc.common.SStatement.close(Unknown Source) - locked <0x10855740> (a java.lang.Object) - locked <0x478364b0> (a com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl) at com.vertica.jdbc.common.SPreparedStatement.close(Unknown Source) - locked <0x478364b0> (a com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl) at com.zaxxer.hikari.pool.ProxyStatement.close(ProxyStatement.java:76) at com.squareup.dbmigrate.util.JdbcHelper.closeQuietly(JdbcHelper.java:15) at com.squareup.reporter.jobs.SummaryFillSqlJob.lambda$fillSql$2(SummaryFillSqlJob.java:107) at
We've got two PreparedStatement objects running in the same transaction.
The lines of code that we control at the stacktrace above is:
JdbcHelper.closeQuietly(countStatement);
JdbcHelper.closeQuietly(queryStatement);
.
and the other thread:
vertica connection closer id=69709 state=BLOCKED *DEADLOCKED*
- waiting to lock <0x478364b0> (a com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl)
owned by task-smcc_payments_vertica_refresh id=321
- holding lock <0x2e7fafa4> (a java.util.concurrent.ThreadPoolExecutor$Worker)
at com.vertica.jdbc.common.SStatement.markResultSetClosed(Unknown Source)
at com.vertica.jdbc.common.SForwardResultSet.close(Unknown Source)
at com.vertica.jdbc.common.SStatement.closeResults(Unknown Source)
at com.vertica.jdbc.common.SConnection.handleTransactionBehavior(Unknown Source)
- locked <0x517358cf> (a java.util.ArrayList)
at com.vertica.jdbc.common.SConnection.rollback(Unknown Source)
- locked <0x20cdb0b7> (a java.lang.Object)
at com.vertica.jdbc.common.SConnection.closeChildObjects(Unknown Source)
- locked <0x20cdb0b7> (a java.lang.Object)
at com.vertica.jdbc.common.SConnection.close(Unknown Source)
- locked <0x14542516> (a com.vertica.jdbc.VerticaJdbc4ConnectionImpl)
- locked <0x06806642> (a java.lang.Object)
at com.zaxxer.hikari.pool.PoolBase.quietlyCloseConnection(PoolBase.java:136)
at com.zaxxer.hikari.pool.HikariPool.lambda$closeConnection$5(HikariPool.java:419)
at com.zaxxer.hikari.pool.HikariPool$$Lambda$153/1556661660.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Its locking the ArrayList object and the PreparedStatement in the opposite order in the statements above leading to the deadlock.