We run a 6 node cluster V7.2.3-19. Node 6 unexpectedly crashed, the server itself rebooted and Vertica on that node stayed down for 2hrs until it was force restarted. The Vertica cluster stayed up throughout on the remaining 5 nodes, we run k-safe 1.
We have a F5 load balancer through which the application connects via. Although we can see that no connections got routed to node 6 while it was down, we still received the following errors from sessions that connected to other nodes (node 1 in this example) and attempted to run queries:
@v_marketdata_node0001: 08006/4539: Received no response from v_marketdata_node0006 in session reset
From looking through the logs of the 5 active nodes during this time, this is the pattern:
@v_marketdata_node0001: 00000/3316: Executing statement: '--^M
Comes:
[PQuery] TX:a00000214e173c(_marketdata_node0001-15601:0x461ce0) select reset_session();
Init Session:0x7f4a440ae880-a00000214e173c [Session] [Query] TX:a00000214e173c(_marketdata_node0001-15601:0x461ce0) select reset_session();
Followed by:
Rollback Txn: a00000214e173c
Init Session:0x7f4a440ae880 @v_marketdata_node0001: 08006/4539: Received no response from v_marketdata_node0006 in session reset
Any ideas as to why queries were still trying to fan out across node 6? Client is ado.net version 07.02.0312.
Thanks
