How to restore a full backup on Kubernetes operator based setup ?

After reading the operator's source code I found a way to override the readiness probe, still interested in knowing the best way of doing a full restoration on a Kubernetes based setup.

Actually even though the readiness/startup probes are overridden nothing changes, when I play the SELECT SHUTDOWN(); everything goes down, and pods restarts everywhere, which is normal, but then again, how can we do a restoration in such case ?

There is a restore process in the documentation at https://docs.vertica.com/12.0.x/en/containerized/backup-restore/#restore-from-a-backup that describes the process including extending the livenessProbe.

Thank you for the insight, indeed it works, did a full restoration but the issue is the following command:

$ /opt/vertica/bin/admintools -t re_ip --file=/opt/vertica/config/ipMap.txt --noprompt --force 

that fails all the time, as the operator tries to run it on all nodes but nothing gets applied because of the nodes state, it throws a rollback error at the end:

Writing new settings to the catalogs of database XXXX ...
The catalog change was not applied to the following nodes:
    v_vertdb_node0001: PrepareFailed
    v_vertdb_node0004: PrepareFailed
    v_vertdb_node0006: PrepareFailed
    v_vertdb_node0002: PrepareFailed
    v_vertdb_node0010: PrepareFailed
    v_vertdb_node0011: PrepareFailed
    v_vertdb_node0012: PrepareFailed
    v_vertdb_node0003: PrepareFailed
    v_vertdb_node0008: PrepareFailed
    v_vertdb_node0007: PrepareFailed
    v_vertdb_node0005: PrepareFailed
    v_vertdb_node0009: PrepareFailed
Failed. Catalog change not committed on a quorum of nodes or there are nodes in unsafe states.

All databases changes failed. Rolling back all the changes ...
Rolledback.

Is there something else we need to do ?

Actually I have a better understanding of what's going on, I think the cluster needs still access to old IPs, the fact that we can't run a re_map is because it's calling old IPs. Even though I ran this command to set the redirection

$ iptables -t nat -A OUTPUT -d OLD_IP -j DNAT --to-destination NEW_IP

Which seem to at least help start the process, it still it fails for the same reason:

Writing new settings to the catalogs of database XXXX ...
The catalog change was not applied to the following nodes:
    v_vertdb_node0001: PrepareFailed
    v_vertdb_node0004: PrepareFailed
    v_vertdb_node0006: PrepareFailed
    v_vertdb_node0002: PrepareFailed
    v_vertdb_node0010: PrepareFailed
    v_vertdb_node0011: PrepareFailed
    v_vertdb_node0012: PrepareFailed
    v_vertdb_node0003: PrepareFailed
    v_vertdb_node0008: PrepareFailed
    v_vertdb_node0007: PrepareFailed
    v_vertdb_node0005: PrepareFailed
    v_vertdb_node0009: PrepareFailed
Failed. Catalog change not committed on a quorum of nodes or there are nodes in unsafe states.

All databases changes failed. Rolling back all the changes ...
Rolledback.


  
    
      No changes applied.
    
  

Re-ip should not be necessary with Eon mode database restore, it should happen automatically if the primary node count is correct, as we expect node IP's might change in an Eon mode cluster. Does the database start anyways if you configure the resource to start from the restore location? If not, please open a ticket if possible, and I'll see if support can look at this during business hours.

If you mean changing restore location to the one its meant to be used, I didn't change anything, left as the script did it, which is exactly the same as the original backup /data for data and /depot for depot. Just to make sure I checked the configuration file metadata/DATABASE_NAME/cluster_config.json

// jq ".Node[0]" cluster_config.json
      {
         "address" : "XX.XX.XX.XX",
         "addressFamily" : "ipv4",
         "catalogPath" : "/data/vertdb/v_vertdb_node0007_catalog/Catalog",
         "clientPort" : 5433,
         "controlAddress" : "XX.XX.XX.XX",
         "controlAddressFamily" : "ipv4",
         "controlBroadcast" : "XX.XX.XX.XX",
         "controlNode" : 45035996512324596,
         "controlPort" : 4803,
         "ei_address" : 0,
         "hasCatalog" : false,
         "isEphemeral" : false,
         "isPrimary" : true,
         "isRecoveryClerk" : false,
         "name" : "v_vertdb_node0007",
         "nodeParamMap" : [],
         "nodeType" : 0,
         "oid" : 45035996512324596,
         "parentFaultGroupId" : 45035996273704980,
         "replacedNode" : 0,
         "schema" : 0,
         "siteUniqueID" : 16,
         "tag" : 0
      }

Also tried to create support ticket here but it seems I don't have enough rights, can you help ?

Thanks a lot,

I am not able to change settings on the support portal, please contact your AE or field team for help there.

Thanks, already did it, still waiting to get access to support portal so I could create the ticket

Are you running the re_ip manually? Or is this what you see when the operator tries to run it. The output from the re_ip doesn't tell us much. We may need to look at the contents of /opt/vertica/log/adminTools.log to see why it failed.

No it's what the operator is trying to do in an infinite loop, circling through all nodes over and over. I attached adminTools.log of one of the nodes, let me know if you need something else.

What is the image that you have current deployed in your cluster? Is it the same version of vertica that the backup was taken from?

The adminTools.log had this error at each of the nodes "*** Core dump before operational or shutting down". We will need to collect scrutinize and investigate other logs. So, best to handle this through a ticket with our support org.

Sorry for the late response, exactly the same, nothing has changed. And yes already created a support ticket, waiting for response

Thank you,

@domidunas : Could you please share the error you are experiencing?