Skip to main content

Backup failing: vbr reports "Error: Missing critical file: [IP]:/path//Snapshots/.../[file].manifest

  • June 1, 2016
  • 4 replies
  • 23 views
Thorz-Hammer

Vertica Backup Failing

 

Hi

 

I have inherited a database from a DBA that is no longer with the company, and one of the known issues is the lack of backup.

 

Node 1 has a NAS location (RAID 0+1) for the backup destination, which it serves as an NFS mount to the rest of the nodes.  Performance has been checked using vioperf and it is excellent from all nodes in the cluster.

 

However, after successfully initializing the snapshot using "-t init", when I run the object level backup, it returns the following error:

 

$ vbr -t backup -c obj_bkp_husvm00-0.ini

Warning: config mapping is missing entries for nodes: (list of STANDBY nodes being ignored)

Starting backup of database PROD.

Objects: ['SCHEMA.tablename']

Participating nodes: (list of PERMANENT nodes to be backed up)

Enter vertica password:

Snapshotting database.

Snapshot complete.

Error: Missing critical file: [IP-address]:/path//Snapshots/obj_bkp_001_(timestamp)/v_PROD_node0018/obj_bkp_001.manifest

Backup FAILED.

 

If I go to the NAS mount point path on node 1, I don't see any directory named "Snapshots" - ignoring the weird "//" in the location.

 

Going to the node referenced by the IP-address (node 18), I also do not see any directory named Snapshots.

 

I assume this indicates that perhaps there must have been a failed backup in the past, which damaged the manifest, which for whatever reason is hidden from me?

 

How do I resolve/fix this?

 

Thanks in advance,

 

TH

    4 replies

    Rahul_Choudhary
    Forum|alt.badge.img
    • Rahul_Choudhary
    • Participating Frequently
    • June 6, 2016

    Hi 

     

    What version are running in? If your cluster version is greater than 7.2.x version I beleive you can run below full-check command to find if your backup doesn't have any unreferenced/missing objects left from any previous partial/failed backup.

     

    $ /opt/vertica/bin/vbr.py -t full-check -c <config_file_name.ini> --report-file=/tmp/backupquickcheck.json

    Request you to kindly run below two new switches/checks (quick-repair and collect-garbage) that have been newly introduced from 7.2.2-2 version and perform below task that can help you resolve this issue.

    The quick-repair task rebuilds the backup manifest based on the manifests contained in the backup location.

    $ vbr -t quick-repair -c <config_file_name.ini>


    Another command after that you need to run is mentioned as below which is with collect-garbage switch. The task of the same is to rebuild your backup manifest and deletes any backup objects that do not appear in the manifest.

    Use the following form to perform a garbage collection task:

    $vbr -t collect-garbage -c <config_file_name.ini> --report-file=/tmp/collect-garbage.json


    Please use below documentation link for detailed information on the same.

    https://my.vertica.com/docs/7.2.x/HTML/index.htm#Authoring/AdministratorsGuide/BackupRestore/RepairingBackups.htm

     

    Than

    Rahul

    Thorz-Hammer
    • Thorz-Hammer
    • Author
    • New Participant
    • June 6, 2016

    Unfortunately, we are running version 7.2.1-4, which apparently does not have any of those options available to us.

     

    Is it normal to have two slashes in a path name?  How does the OS interpret a file reference like that?

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • June 13, 2016

    Was the back up successful in past? Can you share me your ini file

     

     

    Sruthi

    Thorz-Hammer
    • Thorz-Hammer
    • Author
    • New Participant
    • June 14, 2016

    Hi,

     

    Apparently there was some kind of corruption in the target storage.

     

    After I deleted everything in the destination directories, and re-initialized (vbr.py -t init --config-file file.ini), the backup ran successfully, and we have not had any issues since then.

     

    Thank you to all who chimed in!

     

    Thor

    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement