how to start database when .gt file deleted

Hi,

Are you on a multi-node K-Safe cluster? If so Vertica can recover for you.

Example:

I'll delete a .gt file:

[dbadmin@vertica01 681]$ rm -f 0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt

[dbadmin@vertica01 681]$ vsql
Password:
Welcome to vsql, the Vertica Analytic Database interactive terminal.

Type:  \h or \? for help with vsql commands
       \g or terminate with semicolon to execute query
       \q to quit

dbadmin=> select * from test;
ERROR 6407:  StorageBundleReader: Error opening file [/home/dbadmin/rename_me/v_rename_me_node0001_data/681/0297d0a17c34fe9261c352d8f2c551a400a0000000002dc9_0.gt] for read: No such file or directory

dbadmin=>\q

That's not good!

Now I'll stop the DB to get to where you are ...

[dbadmin@vertica01 681]$ admintools -t stop_db -d rename_me -p ****** -F
        Connecting to database
        Issuing shutdown command to database
Database rename_me stopped successfully

Next I will try to restart:

[dbadmin@vertica01 681]$ admintools -t start_db -d rename_me -p vertica8
        Starting nodes:
                v_rename_me_node0001 (192.168.2.200)
                v_rename_me_node0002 (192.168.2.201)
                v_rename_me_node0003 (192.168.2.202)
        Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (DOWN) v_rename_me_node0003: (DOWN)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (INITIALIZING) v_rename_me_node0003: (INITIALIZING)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0002: (UP) v_rename_me_node0003: (UP)
Nodes UP: v_rename_me_node0003, v_rename_me_node0002
Nodes DOWN: v_rename_me_node0001 (may be still initializing).
        Found these errors in startup.logs on hosts:
host ['192.168.2.200'] report: @v_rename_me_node0001: VX001/2973: Data consistency problems found; startup aborted
        HINT:  Check that all file systems are properly mounted.  Also, the --force option can be used to delete corrupted data and recover from the cluster
        LOCATION:  mainEntryPoint, /scratch_a/release/svrtar28000/vbuild/vertica/Basics/vertica.cpp:1696
Do you want to continue waiting? (yes/no) [yes] no
        Server startup was successful on some nodes, but not complete

Press RETURN to continue
Database rename_me did not start successfully: No better than partial startup

Not a full start :'( Note the hint about using the --force option to recover.

Which node is down?

[dbadmin@vertica01 681]$ admintools -t list_allnodes | grep rename_me
 v_rename_me_node0001 | 192.168.2.200 | DOWN  | vertica-9.0.1.7 | rename_me
 v_rename_me_node0002 | 192.168.2.201 | UP    | vertica-9.0.1.7 | rename_me
 v_rename_me_node0003 | 192.168.2.202 | UP    | vertica-9.0.1.7 | rename_me

Restart the node using the --force option:

[dbadmin@vertica01 681]$ admintools -t restart_node -s 192.168.2.200 -d rename_me --force
Info: no password specified, using none
*** Restarting nodes for database rename_me ***
        restart host 192.168.2.200 with catalog v_rename_me_node0001_catalog
        issuing multi-node restart
        Starting nodes:
                v_rename_me_node0001 (192.168.2.200)
        Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
        Node Status: v_rename_me_node0001: (DOWN) v_rename_me_node0003: (UP)
        Node Status: v_rename_me_node0001: (UP) v_rename_me_node0003: (UP)
Restart Nodes result:  1

[dbadmin@vertica01 681]$ admintools -t list_allnodes | grep rename_me
 v_rename_me_node0001 | 192.168.2.200 | UP    | vertica-9.0.1.7 | rename_me
 v_rename_me_node0002 | 192.168.2.201 | UP    | vertica-9.0.1.7 | rename_me
 v_rename_me_node0003 | 192.168.2.202 | UP    | vertica-9.0.1.7 | rename_me

Is my data ok? Yup!

[dbadmin@vertica01 681]$ vsql
Password:
Welcome to vsql, the Vertica Analytic Database interactive terminal.

Type:  \h or \? for help with vsql commands
       \g or terminate with semicolon to execute query
       \q to quit

dbadmin=> select * from test;
 c1
----
  5
  1
  2
  3
  4
(5 rows)

Thank u @Jim_Knicely