Skip to main content
Question

s3 backups

  • October 25, 2018
  • 1 reply
  • 6 views

Vertica_Curtis
Forum|alt.badge.img+1

In this section:
https://www.vertica.com/docs/9.1.x/HTML/index.htm#Authoring/AdministratorsGuide/BackupRestore/CreatingBackupsonAmazonS3.htm?TocPath=Administrator's%20Guide|Backing%20Up%20and%20Restoring%20the%20Database|_____9

It says this:
Because S3 storage does not support file locking, Vertica uses either your local file system or an Amazon EC2 file system to handle file locks during a backup. You identify this location using the s3_backup_file_system_path parameter in your vbr configuration file. During a backup, Vertica creates a locked identity file on your local or EC2 instance, and a duplicate file in your S3 backup location. As long at the files match, Vertica proceeds with the backup, releasing the lock once the backup is complete. As long as the files remain identical, you can use the S3 location for backup and restore tasks.

But doesn’t really clarify how much space the “identify file” takes up.

The client has these questions:
So we have 16 nodes x 5TB data volume that means I need to have a single 90TB volume/locking directory for processing on the node running vbr? Trying to understand how backing up to S3 could save us storage costs if we have to have the same size volume locally before copying it to S3.

We currently backup to sc1 volumes to save cost, but I'm assuming this locking directory need to be fast for processing, like an ssd type, before sending files to S3?

1 reply

Vertica_Curtis
Forum|alt.badge.img+1
  • Author
  • Participating Frequently
  • October 25, 2018

“The identify file only takes a few KB, you it does not consume "any" storage at all. Basically we use it to avoid two vbr tasks running on the same backup location.”
– The location where we relocate the backup location locking mechanism. S3 file system does not support file locking, so we have to redirect the lock to a place where supports file locking. This can be a small NFS mount(like EFS in AWS), or a small micro instance. It does not require to be large, less than 1KB