Hello Team,
We have got a harmful issue on the Production 7 nodes cluster (version - Vertica Analytic Database v6.1.3-0 ):
We have 3.7 T of the disk space on each node.
The following query:
select node_name,disk_space_used_mb
from disk_storage
where storage_usage ilike '%DATA%' and storage_status ilike 'Active'
shows result:
node_name disk_space_used_mb Storage_usage
v_sacvrt102_node0001 2,919,337 DATA,TEMP
v_sacvrt102_node0002 2,905,892 DATA,TEMP
v_sacvrt102_node0003 2,923,837 DATA,TEMP
v_sacvrt102_node0004 2,989,398 DATA,TEMP
v_sacvrt102_node0005 2,937,109 DATA,TEMP
v_sacvrt102_node0006 2,981,596 DATA,TEMP
v_sacvrt102_node0007 2,941,666 DATA,TEMP
But if we run the next query, we will see that really vertica uses following space for its data:
select node_name, sum(ros_used_bytes)/(1024*1024) disk_space_used_mb
from projection_storage
group by node_name order by 1
node_name disk_space_used_mb
v_sacvrt102_node0001 2,040,824.97
v_sacvrt102_node0002 2,041,714.96
v_sacvrt102_node0003 2,042,162.97
v_sacvrt102_node0004 2,041,177.15
v_sacvrt102_node0005 2,041,159.98
v_sacvrt102_node0006 2,040,719.20
v_sacvrt102_node0007 2,040,232.45
The free space decreases by 1% at hour on the our Cluster. Vertica release space after database stop and database start from admintools, which take a long time (about 1,1.5 h).
Please help us to solve this issue , because it is really critical for us.
Thanks so much for any help
disk space
+2Hi Anna,
If you have a critical issue with your production cluster, please contact Vertica Technical Support, by using your Enterprise account to log into http://my.vertica.com/ .
That said -- The first table is just looking at the actual utilization on the drives in question. It's equivalent to running "df -h" from the Linux command line. The second query is telling you how much space is used by Vertica data storage ONLY -- it is excluding catalog storage, temporary storage ("temporary" -- within individual queries; for example if a large query has to do an external re-sort), and disk usage by any other applications sharing the disk space with Vertica.
So, I would suggest a little more investigation: Is it Vertica's catalog, temporary-space usage, or some other application that's using up the space on the drive?
"Catalog" is easy to rule out -- is your catalog stored on a separate partition from your DATA/TEMP storage locations? It only uses space on its own partition.
For other applications, that seems unlikely since the space is cleaned up when Vertica is restarted. Maybe you have an ETL application that cleans up after itself when Vertica knows that it's down?
So, most likely TEMP space. Take a look through the SESSIONS and QUERY_REQUESTS tables -- do you have any ongoing long-running queries? If so, what's the query?
Also the TUPLE_MOVER_OPERATIONS table -- a big MERGEOUT operation can use as much disk space as the size of the table while it is running, though that space will be freed (and usually you will end up using less total disk space due to better compression) when the operation completes.
Note that, per the "Disk Space Requirements for HP Vertica" section of the Administrator's Guide*, Vertica does use a significant amount of space for temporary operations. (Also, running a disk near-full causes files to become fragmented, reducing performance.) As a result, we recommend not more than 60% utilization for your hard drives. If you have 3.7TB/node, 60% of that is 2.2TB/node. Sounds like you're already at 2.0TB/node. So you are, in fact, running out of space. It's often possible to tune particular workloads to use less disk space; Vertica Support can help with that; but you're also about due for some hardware upgrades (more nodes or more disks per node) if you plan to continue to grow this cluster.
Adam
* https://my.vertica.com/docs/7.0.x/PDF/HP_Vertica_7.0.x_AdministratorsGuide.pdf -- requirements are the same for past Vertica versions as well
If you have a critical issue with your production cluster, please contact Vertica Technical Support, by using your Enterprise account to log into http://my.vertica.com/ .
That said -- The first table is just looking at the actual utilization on the drives in question. It's equivalent to running "df -h" from the Linux command line. The second query is telling you how much space is used by Vertica data storage ONLY -- it is excluding catalog storage, temporary storage ("temporary" -- within individual queries; for example if a large query has to do an external re-sort), and disk usage by any other applications sharing the disk space with Vertica.
So, I would suggest a little more investigation: Is it Vertica's catalog, temporary-space usage, or some other application that's using up the space on the drive?
"Catalog" is easy to rule out -- is your catalog stored on a separate partition from your DATA/TEMP storage locations? It only uses space on its own partition.
For other applications, that seems unlikely since the space is cleaned up when Vertica is restarted. Maybe you have an ETL application that cleans up after itself when Vertica knows that it's down?
So, most likely TEMP space. Take a look through the SESSIONS and QUERY_REQUESTS tables -- do you have any ongoing long-running queries? If so, what's the query?
Also the TUPLE_MOVER_OPERATIONS table -- a big MERGEOUT operation can use as much disk space as the size of the table while it is running, though that space will be freed (and usually you will end up using less total disk space due to better compression) when the operation completes.
Note that, per the "Disk Space Requirements for HP Vertica" section of the Administrator's Guide*, Vertica does use a significant amount of space for temporary operations. (Also, running a disk near-full causes files to become fragmented, reducing performance.) As a result, we recommend not more than 60% utilization for your hard drives. If you have 3.7TB/node, 60% of that is 2.2TB/node. Sounds like you're already at 2.0TB/node. So you are, in fact, running out of space. It's often possible to tune particular workloads to use less disk space; Vertica Support can help with that; but you're also about due for some hardware upgrades (more nodes or more disks per node) if you plan to continue to grow this cluster.
Adam
* https://my.vertica.com/docs/7.0.x/PDF/HP_Vertica_7.0.x_AdministratorsGuide.pdf -- requirements are the same for past Vertica versions as well
Adam, thank you so much for help
Sign up
Already have an account? Login
Welcome to the Rocket Forum!
Please log in or register:
Employee Login | Registration Member Login | RegistrationEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.
77 4th Avenue
Waltham, MA 02451 USA
