Vertica service not starting after server reboot

Hi Hari

Can you kindly provide us what version of vertica you have been running on?

$ rpm -qa | grep vertica

Also provide us the output of

$admintools -t list_allnodes

Along with the above output please pass on the vertica.log file of the same node to have a look into it.

Thanks

Rahul

Hi Rahul, below are the details you asked: -

$ rpm -qa | grep vertica

vertica-7.2.1-0.x86_64

$admintools -t list_allnodes

attached the output as screenshot

vertica.log

attached herewith

Hi Rahul,

The server had been going down for quite a few times in May. But the databases were coming up when started after server reboot. But Yesterday again the server went down and it was started manually, since then we are unable to up the databases. When we try to start the database from adminTools, after selecting database and entering the password for the database we get the

*** Starting database : database name ***

display and the process comes back the adminTools in a flash.

I have attached the adminTools.log and

latest vertica.log file I have shared in the below link as the size is more: -

https://www.sendspace.com/file/b34uge

Hi Hari

Its a tricky situation where you catalog is not getting initialized and since you don't have k-safety feature set (with no extra nodes) up which can help in performing in such scenerios.

I am afraid you can bring this db up without that.

I will suggest you kindly raise a vertica support case if you have a valid account for the same.Else try restoring from backup or create a new db.

snippet of admintools-dbadmin.log

=============================

2016-06-08 13:39:09.972 admintools/2431:0x7fd50697f700 [adminExec.display] <INFO> Unable to read database catalogs - cannot start database.
2016-06-08 13:39:13.132 admintools/2431:0x7fd50697f700 [adminExec.removeRunningDbs] <INFO> items '[('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0)]' runningDBs '[]'
2016-06-08 13:39:13.133 admintools/2431:0x7fd50697f700 [adminController.dbStart] <INFO> candidates to start (no active) '[('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0)]'
2016-06-08 13:39:18.336 admintools/2431:0x7fd50697f700 [Passwords.getpass] <INFO> get password '['DatabaseName']'
2016-06-08 13:39:23.262 admintools/2431:0x7fd4fea2a700 [BashAdapter._exec_module_impl] <INFO> Running command ['/opt/vertica/oss/python/bin/python', '-m', 'vertica.engine.api.at_runner', '--no-prompt', '--module', 'vertica.engine.api.compute_vdatabase']#{"catalogpath": "/home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog"}
2016-06-08 13:39:23.814 at_exec/2510:0x7fc83568b700 [root.setup_custom_logging] <INFO> New log for 'at_exec'
2016-06-08 13:39:23.815 at_exec/2510:0x7fc83568b700 [root.setup_custom_logging] <INFO> sys.argv: '/opt/vertica/share/eggs/vertica/engine/api/at_runner.py' --no-prompt --module 'vertica.engine.api.compute_vdatabase'
2016-06-08 13:39:23.880 admintools/2431:0x7fd50697f700 [adminExec.display] <INFO> Unable to read database catalogs - cannot start database.
2016-06-08 13:39:47.310 admintools/2431:0x7fd50697f700 [adminExec.checkLicenseNodeCount] <WARNING> License file /opt/vertica/config/share/license.key was not valid, returncode -11, message:
None

2016-06-08 14:35:54.346 admintools/4763:0x7fa8f0fb4700 [adminExec.getLastGoodEpoch] <ERROR> failed to find majority of nodes (1) reporting the same ASR epoch (largest was epoch 0, count 0).
2016-06-08 14:35:54.347 admintools/4763:0x7fa8f0fb4700 [adminController.restartDBFromEpoch] <WARNING> Too few nodes are available to restart cluster
2016-06-08 14:35:55.351 admintools/4763:0x7fa8f0fb4700 [adminController.restartDBFromEpoch] <WARNING> no good epoch found -- warning user
2016-06-08 14:35:56.138 admintools/4763:0x7fa8f0fb4700 [adminController.restartDBFromEpoch] <INFO> found last epoch -1 corresponding to timestamp ''
2016-06-08 14:36:45.232 admintools/5021:0x7f66496df700 [root.setup_custom_logging] <INFO> New log for 'admintools'
2016-06-08 14:36:45.233 admintools/5021:0x7f66496df700 [root.setup_custom_logging] <INFO> sys.argv: /opt/vertica/bin/adminTools -t command_host -cstatus
2016-06-08 14:36:45.751 admintools/5021:0x7f66496df700 [SshAuth.setup_from_admintools_conf_if_needed] <INFO> Setup complete, default ssh user is dbadmin, default ssh options are -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes.
2016-06-08 14:36:45.752 admintools/5021:0x7f66496df700 [adminExec.__init__] <INFO> Unique Ports: False
2016-06-08 14:36:45.752 admintools/5021:0x7f66496df700 [adminExec._is_current_user_conf_owner] <INFO> Owner of admintools.conf (dbauser) is dbadmin.
2016-06-08 14:36:45.754 admintools/5021:0x7f66496df700 [adminExec.isOkToRun] <INFO>
2016-06-08 14:36:45.754 admintools/5021:0x7f66496df700 [__main__.<module>] <INFO> Use unique ports = False
2016-06-08 14:36:45.754 admintools/5021:0x7f66496df700 [adminExec._is_current_user_conf_owner] <INFO> Owner of admintools.conf (dbauser) is dbadmin.
2016-06-08 14:36:45.755 admintools/5021:0x7f66496df700 [adminExec.isOkToRun] <INFO>
2016-06-08 14:36:46.208 admintools/5041:0x7fd92a18e700 [root.setup_custom_logging] <INFO> New log for 'admintools'
2016-06-08 14:36:46.209 admintools/5041:0x7fd92a18e700 [root.setup_custom_logging] <INFO> sys.argv: '/opt/vertica/bin/TXTlcheck.py' /opt/vertica/config /opt/vertica/bin
2016-06-08 14:36:46.213 admintools/5021:0x7f66496df700 [commandLineCtrl.commandHost] <INFO> processing status
2016-06-08 14:36:46.311 admintools/5021:0x7f66496df700 [commandLineCtrl.commandHost] <INFO> this host's DBs: {'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}}

Thanks

Rahul

Oh ok. So there is no other way by which we can bring up the database. Is there a certain reason why this has happened? it was working fine until the server went down and came up.

Even if we have backup files, we will have to create new databases to dump the backup files, since now the old databases are like corrupted, right?

You can straight away restore the database using the restore process (task restore used with vbr.py) if you have a valid backup.

Please refer documentation for the same.

Thanks

Rahul

Hi ,

 did you change any of the OS values ? 

Like memory increse or other stuff like that in between reboots ? 

Hi Rahul,

I have a valid backup but the we are unable to start the vertica service at all. When we check the 

service verticad status

we get 

Status not OK

and even if we try to start the service, we get

start not OK                [FAILED]

Also, we are unable to create any database. When I try to create a new database in adminTools, I get 

*** Creating database: testing ***

Press RETURN to continue

then I get an error:- 

Installation error:

Host=127.0.0.1 version=NO VERSION FOUND.

Check the file system ownership on the server !! make sure dbadmin and verticadba group have full access.For catalog path and data paths 

Hi Adrian,

I'm not aware of such changes in the OS yet. This server is situated at a remote location and the server management guys say they had just restarted the server. Let me check with the server guys and come back.

Ok. Let me check those too. 

I got a reply saying they just rebooted without any changes. 

Now can we re-install vertica on top the existing vertica setup?

We have installed vertica after complete uninstallation. Installation happened without any error but when we give the license in adminTools which we have to during the first time, we get and error

"Error: License key /opt/vertica/config/licensing/vertica_comminuty_edition.license.key not installed. Invalid license status " .

What could be the problem here?