We have a customer who digest about 4 billion records, ~150GB every day.
They have defined a weekly partition for the hot data, each partition with ~28 billion rows.
In addition they accumulate all historical data to a huge “Archive” Parquet defined as one external table,
and worried how it will perform several years from now.
Increasing the number of partitions may end up with too many ROS containers per node per projection.
Although there is no limit on the number of partitions per table. When partitions are segregated into ROS containers,
Vertica limits the number of ROS containers per projection per node to 1024, so the limit is essentially 1024 partitions per table.
See: https://www.vertica.com/kb/Vertica-Partitions-The-FAQs/Content/FAQs/Vertica-Partitions-The-FAQs.htm
This is all relevant for data inside ROS files, but what about external tables?
Is it advised to increase the number of partitions in external tables?
Is there a limit?
Querying Parquet as external table using partition pruning, would you recommend to define also weekly partition on that external table?
What do you think?
