$ admintools -t start_db -d my_database -U
Info: no password specified, using none
Could not login with SSH to host 10.3.2.110
Starting nodes:
v_my_database_node0001 (10.3.2.110) ***UNAVAILABLE***
v_my_database_node0002 (10.3.2.111)
v_my_database_node0003 (10.3.2.112)
Vertica names unique storage locations using a random algorithm. The operating system improves the randomness of its pseudo-random-number generator by gathering 'entropy' from the environment. Either some nodes currently have insufficient entropy to provide sufficient randomness (in which case, the problem is solved by waiting for sufficient entropy to accumulate, a process which should not take much more than a minute) or some error has prevented reading the entropy value from the node(s) (if this process doesn't make progress in a few minutes --- perhaps longer on larger clusters --- contact Vertica technical support).
Starting Vertica on all nodes. Please wait, databases with large catalog may take a while to initialize.
10.3.2.111 failed. Result:
status=1 host=10.3.2.111 content={u'command_list': [u'/opt/vertica/bin/vertica', u'-D', u'/data/my_database/v_my_database_node0002_catalog', u'-C', u'my_database', u'-n', u'v_my_database_node0002', u'-h', u'10.3.2.111', u'-p', u'5433', u'-P', u'4803', u'-Y', u'ipv4', u'-U'], u'db_log_path': u'/data/my_database/dbLog', u'start_stdout': None, u'special_environment': None, u'return_code': 1, u'start_stderr': None}
Check /data/my_database/dbLog for more information.
Press RETURN to continue
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Node Status: v_my_database_node0001: (DOWN) v_my_database_node0002: (DOWN) v_my_database_node0003: (DOWN)
Error starting database, no nodes are up
Press RETURN to continue
Database my_database did not start successfully
We saw the below in startup.log
v_my_database_node0003_catalog]$ tail -f startup.log
{
"node" : "v_my_database_node0003",
"stage" : "Database Halted",
"text" : "Not enough open file handles allowed (1024 available/32768 required); see 'ulimit -n'.\n",
"timestamp" : "2020-10-28 10:43:03.353"
}
We did the below in /etc/security/limits.conf (followed by reboot)
- soft nofile 191918
- hard nofile 191918
We still saw the same error
When we get a fresh login using su -l -s /bin/bash, then we were able to see ulimit -n showing 191918 for root user.
Can somebody point us where is the problem here ? A reboot didn't help but a new session using su -l -s /bin/bash helps always. We are wondering what should be done to permanently get this fixed. Each time we shutdown vertica, we are not able to bring it back up without getting a new session using su -l -s /bin/bash
