Some s3 export details

Domo recently asked some questions regarding s3 export, WRT performance. I didn't know, so I asked engineering. I'm including their response here, and Domo's follow up. I thought others would find it interesting.

1. Does the s3 export use all the nodes or just the initiator node for pushing data to s3? Can we benefit from node parallelization?
   The short answer is yes, all the nodes can be used. The longer answer is that S3 export is a user-defined transform function, so it should get parallelism based on a few factors: the segmentation of the projection being used, the EXECUTIONPARALLELISM of the enclosing resource pool, the OVER() clause of the export query, etc. You can also try using PARTITION BEST and PARTITION NODES. https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Functions/Analytic/window_partition_clause.htm

2. Do you have baseline speed tests for this function?
   Apparently not. They didn’t have anything to say on this one.

3. Does the s3 export function support gzip?
   Regarding compression, S3 export doesn't support it yet. If you'd like to export in a compressed format, the best thing we've got is Parquet export to S3.

4. Any ideas to help improve performance?
   Since the customer is using "OVER ()", they are running the export on a single partition of the table, which is to say, the entire table. This is probably happening in a single thread. Changing this in a way that helps it exploit their projection segmentation and/or partitioning should have a pretty good ratio of effort spent to performance gains.

5. What would you say is the fastest way to get data out of vertica? We are trying to snapshot tables into s3.
   Write the query to exploit their projections 😊
   This probably means changing the OVER clause.

6. Does the s3 export partition function use parallelization?
   It does if the query lets it!

And then Domo's response:

Good news.
We saw significant performance improvement using PARTITION BEST and PARTITION NODES and even decreasing the chunk size. We saw uploads to s3 take from 70 minutes to 12 minutes for a 22 gig raw size table. So around ~2 gigs a minute which is pretty good. And it scales with the number of nodes. Thanks for helping us out! We would have never figured that out 😝

Their original syntax:
SELECT s3export(* USING PARAMETERS
url='s3://my-domo-bucket/table_v1',
delimiter=',',
chunksize='5368709120',
record_terminator='\n',
from_charset='UTF-8',
to_charset='UTF-8',
prepend_hash='true')
OVER () FROM cust_domo.table1;