Error: 'Could not send data to client: No such file or directory'

HI,

The explicitly says "Query runtime exceeds limit, canceling".

Check the runtimecap for all resource pools involved.

I can mimic your vertica.log error as follows:

Example:

dbadmin=> ALTER RESOURCE POOL all_users RUNTIMECAP '2 seconds';
ALTER RESOURCE POOL

dbadmin=> ALTER USER jim resource pool all_users;
ALTER USER

dbadmin=> SELECT name, runtimecap FROM resource_pools WHERE name IN ('general', 'all_users');
   name    | runtimecap
-----------+------------
 general   |
 all_users | 00:00:02

dbadmin=> \c - jim
You are now connected as user "jim".

dbadmin=> SELECT COUNT(*) FROM big_fact2 CROSS JOIN big_fact2 b;
ERROR 3326:  Execution time exceeded run time cap of 00:00:02

My log entry matched yours:

    2019-01-31 09:55:31.534 Init Session:7ff793fff700-a000000014c9f8 [Session] <INFO> [Query] TX:a000000014c9f8(v_test_db_node0001-264917:0x18bd3b) SELECT COUNT(*) FROM big_fact2 CROSS JOIN big_fact2 b;
    2019-01-31 09:55:33.546 Init Session:7ff793fff700-a000000014c9f8 [EE] <INFO> Query runtime exceeds limit, canceling

Plus if you go a little further in the log, I see this message:

2019-01-31 09:59:53.250 Init Session:7ff798aff700-a000000014ca19 <ERROR> @v_test_db_node0001: 57014/3326: Execution time exceeded run time cap of 00:00:02

Hi Jim,

Thanks for the quick reply:)

I should have been more clear; the messages indicating 'Query runtime exceeds limit, canceling' are expected since we have three cascading resource pools with increasing runtimecaps (10s, 5m, 2h) and they're just part of the cascading process.

The issue is in the next section stating 'Could not send data to client: No such file or directory', and then 'Client has disconnected'. I'm pretty sure the client application does not actually close the connection, since it's not giving any issues for similar queries, but somehow Vertica thinks it does. Applications that use the vertica_python module keep hanging indefinitly, while applications using pyODBC exit with an error status, but no message. And for some reason when this happens, it's always after 30 minutes, yet we have no timeout setting anywhere for that amount of time (we do for some users have a longer timeout of 1 hour and 2 hours on the final resource pool). Also if some timeout was hit, I would expect an appropriate message in the logs, not 'Could not send data to client: No such file or directory'.

Currently this only happens for two queries, but it happens consistently. Both queries are pretty straight forward select statements, but they do fetch a lot of data (millions of rows). The fact that the issue never occured before and other queries are unaffected makes me doubt that it's an issue in our applications.

Hope this clarifies it somewhat. I am still at a loss as to what might cause this, so I'm also not sure I'm explaining it correctly:)

You might be hitting a known issue where while your query is in the initial pool, Vertica returns records to the client, but before the query finishes, the run time cap is hit... So Vertica will try to cancel the query and queue it on the new pool.

However, the query has already sent output rows to the client! In this case, Vertica does not know how to restart it and skip sending the rows back to the client that had already been sent.

There is an open JIRA on this issue and I will keep you updated on its progress.

Can you please send me your company's name so that I can add it to the JIRA as a client needing a resolution ASAP?

My email is james.knicely@microfocus.com

You might be able to mitigate the issue by adding an explicit ORDER BY on a unique key in the query.

@Jim_Knicely : Can you please update JIRA ID/Link if this is already taken care.