Restoring to an old snapshot takes more storage

Yes, 
It can happen, for two reasons that interact with each other.

1. Tuple Mover Activity may result in new files getting created 
2. when we copy the snapshot back we first copy the files from the snapshot, and only then delete the files that were created by the Tuple Mover.

This can result in a short duration of time where both files coexists. the  ones from the older snapshot and the ones created by the Tuple Mover after the snapshot was taken.


see example below

1. Say I have a database that takes 428K space on disk

[skeswani@skeswani] [@:/]$ du -sh /scratch_b/verticadb/test/v_test_node0001_data/
428K    /scratch_b/verticadb/test/v_test_node0001_data/

2. I back it up

[skeswani@skeswani] [@:/]$ /opt/vertica/bin/vbr.py --task backup --config backup_snapshot.ini 
Starting backup of database test.
Participating nodes: v_test_node0001.
Snapshotting database.
Snapshot complete.
Approximate bytes to copy: 244032189 of 244032189 total.
[==================================================] 100%
Copying backup metadata.
Finalizing backup.
Backup complete!


3. Now add some data after the backup

skeswani=> insert /*+direct*/ into t select * from t; commit;
 OUTPUT 
 --------
 100000
(1 row)

4. Now the database is 620K on disk, it has 2 files one is the initial data in the table and the other the result of the above insert

[skeswani@skeswani] [@:/]$ du -sh /scratch_b/verticadb/test/v_test_node0001_data/
620K    /scratch_b/verticadb/test/v_test_node0001_data/

[skeswani@skeswani] [@:/]$ find /scratch_b/verticadb/test/v_test_node0001_data/ -name *.gt
/scratch_b/verticadb/test/v_test_node0001_data/731/028d04fcbcc5c6f1090c842bf744893800a0000000002243_0.gt
/scratch_b/verticadb/test/v_test_node0001_data/713/02edfdb8e99026fdaef7c4ac657be25a00a0000000002231_0.gt

5. now merge the data that was inserted with the old data (we do this manually now, but this is a background Tuple Mover task in vertica that occurs with time and data)

skeswani=> select do_tm_task('MergeOut'); 
                           do_tm_task                           
 ----------------------------------------------------------------
 Task: mergeout
(Table: public.t) (Projection: public.t_super)
(1 row)

So now there are three files, (a) initial data in the table, (b) insert we did, (c) TM created a merged file 

[skeswani@skeswani] [@:/]$ du -sh /scratch_b/verticadb/test/v_test_node0001_data/
1.1M    /scratch_b/verticadb/test/v_test_node0001_data/

[skeswani@skeswani] [@:/]$ find /scratch_b/verticadb/test/v_test_node0001_data/ -name *.gt
/scratch_b/verticadb/test/v_test_node0001_data/731/028d04fcbcc5c6f1090c842bf744893800a0000000002243_0.gt
/scratch_b/verticadb/test/v_test_node0001_data/739/028d04fcbcc5c6f1090c842bf744893800a000000000224b_0.gt
/scratch_b/verticadb/test/v_test_node0001_data/713/02edfdb8e99026fdaef7c4ac657be25a00a0000000002231_0.gt


6. Wait for the old files to get reaped (in about 2 min). Now we have only 1 file on disk and the database size is 656K

[skeswani@skeswani] [@:/]$ find /scratch_b/verticadb/test/v_test_node0001_data/ -name *.gt
/scratch_b/verticadb/test/v_test_node0001_data/739/028d04fcbcc5c6f1090c842bf744893800a000000000224b_0.gt
[skeswani@skeswani] [@:/]$ du -sh /scratch_b/verticadb/test/v_test_node0001_data/
656K    /scratch_b/verticadb/test/v_test_node0001_data/


7. Now restore an older snapshot, 
      I stop the database,  
      delete the catalog
      and restore the older snapshot

skeswani=> select shutdown();
          shutdown          
 ----------------------------
 Shutdown: moveout complete
 (1 row)


[skeswani@skeswani] [@:/]$ rm -rf /scratch_b/verticadb/test/v_test_node0001_catalog/*

[skeswani@skeswani] [@:/]$ /opt/vertica/bin/vbr.py --task restore --archive 20170515_162940 --config backup_snapshot.ini
Starting full restore of database test.
Participating nodes: v_test_node0001.
Restoring from restore point: backup_snapshot_20170515_162940
Determining what data to restore from backup.
[==================================================] 100%
Approximate bytes to copy: 244031578 of 244032189 total.
Syncing data from backup to cluster nodes.
[==================================================] 100%
Restoring catalog.
Restore complete!

8. Note that the data on disk is as before. So it works as expect - right. 
 (not quite if you have used most of your disk space)

[skeswani@skeswani] [@:/]$ du -sh /scratch_b/verticadb/test/v_test_node0001_data/
428K    /scratch_b/verticadb/test/v_test_node0001_data/


9. However if you really closely, 
 (by running a watch program on the data directory)

We first create the files that belong to the snapshot and then we delete the files that dont. during this time you can have both old and new files in the data volume. 
Hence if you are running close to the 100% on disk usage, your restore may fail.

skeswani@skeswani ~] [@:/]$ inotifywait -m -r --format '%e %f' -e delete,create /scratch_b/verticadb/test/v_test_node0001_data/
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
CREATE .02edfdb8e99026fdaef7c4ac657be25a00a0000000002231_0.gt6GLkNs   <== create first
DELETE 028d04fcbcc5c6f1090c842bf744893800a000000000224b_0.gt          <== delete later


Hope this answers your question.

- Sumeet

Off-course, the act of dropping the database would delete the files first.
We leave that for the user to do manually, if there are special circumstances like disks with little or no remaining space.

Thank you skeswani for the detailed explaination.
)