Hi -
I am trying to issue a copy statement that uses the hdfs connector with a wildcard to import all part files from an hdfs directory such as:
copy my_events source hdfs(url='http://my.server.com:50070/webhdfs/v1/path/to/my_events/part-000*', username='hdfs') delimiter E'\t';
If i specify just one file, or a handful of files, it works perfectly, however, once i try to do too many files it seems, i.e. 'part-000*' - I get this error (i.e. multiple times):
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): Broken pipe
ERROR 3399: Failure in UDx RPC call InvokeDestroyUDL(): UDx side process has exited abnormally
ERROR 3399: Failure in UDx RPC call InvokeSetupUDL(): UDx side process has exited abnormally
I'm wondering if this is a problem on the vertica side, or on the webhdfs side? And what to do about it.
My wild intuition tells me that maybe one of these is happening?:
- vertica is somehow overloading webhdfs (i see no errors in the hdfs audit logs though) with too many requests?
- that there is some sort of filesystem limit (number of open files? or otherwise?) that vertica or webhdfs is hitting?
- other configuration settings on either side that are preventing this?
If anyone has any idea how to debug this I'd appreciate it!