RabbitMQ and Vertica

Hi, I built an ActiveMQ connector for Vertica for a project. IIRC, RabbitMQ is another JMS protocol, so we could probably take a similar approach. It will require an external connector tool in (most likely) Java that uses JDBC to push messages to Vertica using PreparedStatement. Are you able to share details of your project?

High performance mechanism could require a UDSource plugin into the database.

Thanks Bryan, I have a meeting with the prospect next week. Good to know about this. Also good to know about the ActiveMQ part - had another prospect asking for it earlier this year.

@Ben_Vandiver I'm looking at building a UDSource for ActiveMQ and similar, but is that really the way to go for an unbounded stream source like MQ? Wouldn't it require a perpetually open session running the query that triggers the UDSource?

Usually, it's an outside service that issues a stream of SQL COPY commands to move data. Each COPY is only open for a certain length of time. This provides batching for performance, a transactional boundary for fault tolerance, and a time bound for an SLA. If you look at our Kafka implementation, you'll see all of these features. A key component is tracking how to restart the stream should either the stream service RabbitMQ fail or Vertica go down. Vertica will recover to a consistent snapshot, but you need to restart the stream at the correct point. In Kafka, we do this with <topic,partition,offset> tuples stored in the database that tell us where to restart. I don't know if RabbitMQ has a similar mechanism - I thought it tried to do the consumer tracking itself, rather than allowing a consumer to identify from whence it wished to consume.