I have a wrapped VBR backup processing failing (ArcSight platform 'db_backup.sh backup').
Error was originally:
At 20%
Error: on host (node3): Error accessing remote storage: When coping file 943/blah_0.gt (bucket s3://recon/events) to reconbackup/Objects/943/blah_0.gt (bucket arcsight-backup): 6149898240 read, but total bytes expected is 6599881709..: returncode=None
Backup FAILED
Failed to execute the backup task by vbr script. See.... for details.
After a bit of attempting collect-garbage and quick-repair, error now seems to be:
At 55%:
Error: On host (node3): Error accessing remote storage: When copying file 943/blah_0.gt (bucket s3://recon/events) to reconbackup/Objects/943/blah_0.gt (bucket arcsight-backup): An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist..: returncode=None
This error causes the backup process to stop (by design), so I guess that's ok.
Eventually the S3 object will be purged, but until then I need to work out how to get backups running again.
I suspect that searching in Recon will also encounter errors as that S3 object wont be able to be restored (either before the garbage collection when corrupt, or after the collection now theres NoSuchKey).
Deleting the source file from s3://recon/events is the, possibly, simplest way forward. Flow on effects mightn't be great.
I believe if there is a local Vertica cache/event issue, I can stop Vertica, delete local corrupt file and then allow EON mode to fix it from S3. Dont suppose it works another way? Thinking probably not, and I imagine this corruption occurred when I had one of the nodes experience a critical failure a few weeks back. Meaning that the local copy is probably broken too.
Any other nuggets of wisdom re: recommendations on how to deal with this?
Thanks!
Ian
