Skip to main content

unable to read node metadata

  • April 1, 2015
  • 1 reply
  • 12 views
serhiip

Hello!

 

I've tried installing Vertica on my local Debian 7 machine. I've dowloaded an install script from https://my.vertica.com/download-community-edition/, installed the server according ot instructions in http://my.vertica.com/docs/7.1.x/HTML/index.htm#Authoring/InstallationGuide/Other/InstallationGuide.htm%3FTocPath%3DInstallation%2520Guide|_____0, launched /opt/vertica/bin/admintools and tried to create a new database - went through all steps and at the end got error:

Spoiler
(Highlight to read)
*** Creating database: biddy ***
172.16.1.65 OK [vertica][(7, 1, 1)][000][all]
Creating database biddy
Starting bootstrap node v_biddy_node0001 (172.16.1.65)
ERROR: Unable to read bootstrap node metadata
Press RETURN to continue
*** Creating database: biddy ***172.16.1.65 OK [vertica][(7, 1, 1)][000][all]Creating database biddyStarting bootstrap node v_biddy_node0001 (172.16.1.65)ERROR: Unable to read bootstrap node metadataPress RETURN to continue

I am aware of this two threads

https://community.dev.hp.com/t5/Vertica-Forum/Unable-to-read-bootstrap-node-metadata/m-p/220871/highlight/true#M8011

https://community.dev.hp.com/t5/Vertica-Forum/Unable-to-read-bootstrap-node-metadata/td-p/218551/highlight/true/page/2

 

People there have the same issue but there is no working solution.

 

My ulimit -u is 63267

 

Here is relevant section of my log:

Spoiler
(Highlight to read)
Apr 1 18:10:32 [19435] Creating database
Apr 1 18:10:32 [19435] [SSH.createDBMultiNodes] bootstrapping with idx 0 node v_biddy_node0001 on host 172.16.1.65
Apr 1 18:10:32 [19435] [SSH.createDBMultiNodes] bootstrap cmd '/opt/vertica/bin/bootstrap-catalog -C 'biddy' -H '172.16.1.65' -s 'v_biddy_node0001' -D '/home/dbadmin/biddy/v_biddy_node0001_catalog' -S '/home/dbadmin/biddy/v_biddy_node0001_data' -L '/opt/vertica/config/share/license.key' -p '5433' -c '172.16.1.65' -B '172.16.3.255' '
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65> (Bootstrapping database biddy): <command hidden>
Apr 1 18:10:32 [19435] [SSH.createDBMultiNodes] cmd output '['0', ['License installed for Vertica Community Edition ', '', 'Catalog successfully bootstrapped']]'
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65> (Reading bootstrap catalog): ['/opt/vertica/oss/python/bin/python -m vertica.config.VDatabase /home/dbadmin/biddy/v_biddy_node0001_catalog silent']
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=255) ['Stopping editor', 'Error stopping editor: [Errno 3] No such process', 'Traceback (most recent call last):', ' File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main', ' "__main__", fname, loader, pkg_name)', ' File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code', ' exec code in run_globals', ' File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 382, in <module>', ' readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent")', ' File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 373, in readAndOutputJSON', ' db.loadFromCatalog(ce)', ' File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 221, in loadFromCatalog', ' self.name = ce.sendCmd("get singleton Database name").strip()', ' File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 72, in sendCmd', ' self.ceproc.stdin.write(cmd+"\\n")', 'IOError: [Errno 32] Broken pipe']
Apr 1 18:10:32 [19435] Unable to read bootstrap node metadata. Expected JSON DB desc, got: Stopping editor
Error stopping editor: [Errno 3] No such process
Traceback (most recent call last):
File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 382, in <module>
readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent")
File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 373, in readAndOutputJSON
db.loadFromCatalog(ce)
File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 221, in loadFromCatalog
self.name = ce.sendCmd("get singleton Database name").strip()
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 72, in sendCmd
self.ceproc.stdin.write(cmd+"\n")
IOError: [Errno 32] Broken pipe
Apr 1 18:10:32 [19435] Multi-node start returns: 0
Apr 1 18:10:32 [19435] [killVerticaProcess] [ -e /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.pid ]
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['[ -e /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.pid ]']
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=1) []
Apr 1 18:10:32 [19435] [removeCatDataDirs] cat /home/dbadmin/biddy/dbLog >> /opt/vertica/log/adminTools-dbadmin.log
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['cat /home/dbadmin/biddy/dbLog >> /opt/vertica/log/adminTools-dbadmin.log']
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=1) ['cat: /home/dbadmin/biddy/dbLog: No such file or directory']
Apr 1 18:10:32 [19435] [removeCatDataDirs] cat /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['cat /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log']
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=1) ['cat: /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.log: No such file or directory']
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['rm -rf /home/dbadmin/biddy']
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=0) []
Apr 1 18:10:32 [19435] Multi-node DB creation unsuccessful
Apr 1 18:10:32 [19435] Create DB complete, signaling FAIL
Apr 1 18:10:32 [19435] Saved Configurator(clusterSize=1, dbs=()) -> /opt/vertica/config/admintools.conf
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['test -e /opt/vertica/config/d5415f948449e9d4c421b568f2411140.dat && echo good || echo missing']
Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=0) ['good']
Apr 1 18:10:32 [19435] Creating databaseApr 1 18:10:32 [19435] [SSH.createDBMultiNodes] bootstrapping with idx 0 node v_biddy_node0001 on host 172.16.1.65Apr 1 18:10:32 [19435] [SSH.createDBMultiNodes] bootstrap cmd '/opt/vertica/bin/bootstrap-catalog -C 'biddy' -H '172.16.1.65' -s 'v_biddy_node0001' -D '/home/dbadmin/biddy/v_biddy_node0001_catalog' -S '/home/dbadmin/biddy/v_biddy_node0001_data' -L '/opt/vertica/config/share/license.key' -p '5433' -c '172.16.1.65' -B '172.16.3.255' 'Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65> (Bootstrapping database biddy): <command hidden>Apr 1 18:10:32 [19435] [SSH.createDBMultiNodes] cmd output '['0', ['License installed for Vertica Community Edition ', '', 'Catalog successfully bootstrapped']]'Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65> (Reading bootstrap catalog): ['/opt/vertica/oss/python/bin/python -m vertica.config.VDatabase /home/dbadmin/biddy/v_biddy_node0001_catalog silent']Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=255) ['Stopping editor', 'Error stopping editor: [Errno 3] No such process', 'Traceback (most recent call last):', ' File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main', ' "__main__", fname, loader, pkg_name)', ' File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code', ' exec code in run_globals', ' File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 382, in <module>', ' readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent")', ' File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 373, in readAndOutputJSON', ' db.loadFromCatalog(ce)', ' File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 221, in loadFromCatalog', ' self.name = ce.sendCmd("get singleton Database name").strip()', ' File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 72, in sendCmd', ' self.ceproc.stdin.write(cmd+"\\n")', 'IOError: [Errno 32] Broken pipe']Apr 1 18:10:32 [19435] Unable to read bootstrap node metadata. Expected JSON DB desc, got: Stopping editorError stopping editor: [Errno 3] No such processTraceback (most recent call last):File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main"__main__", fname, loader, pkg_name)File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_codeexec code in run_globalsFile "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 382, in <module>readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent")File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 373, in readAndOutputJSONdb.loadFromCatalog(ce)File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 221, in loadFromCatalogself.name = ce.sendCmd("get singleton Database name").strip()File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 72, in sendCmdself.ceproc.stdin.write(cmd+"\n")IOError: [Errno 32] Broken pipeApr 1 18:10:32 [19435] Multi-node start returns: 0Apr 1 18:10:32 [19435] [killVerticaProcess] [ -e /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.pid ]Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['[ -e /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.pid ]']Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=1) []Apr 1 18:10:32 [19435] [removeCatDataDirs] cat /home/dbadmin/biddy/dbLog >> /opt/vertica/log/adminTools-dbadmin.logApr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['cat /home/dbadmin/biddy/dbLog >> /opt/vertica/log/adminTools-dbadmin.log']Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=1) ['cat: /home/dbadmin/biddy/dbLog: No such file or directory']Apr 1 18:10:32 [19435] [removeCatDataDirs] cat /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.logApr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['cat /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log']Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=1) ['cat: /home/dbadmin/biddy/v_biddy_node0001_catalog/vertica.log: No such file or directory']Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['rm -rf /home/dbadmin/biddy']Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=0) []Apr 1 18:10:32 [19435] Multi-node DB creation unsuccessfulApr 1 18:10:32 [19435] Create DB complete, signaling FAILApr 1 18:10:32 [19435] Saved Configurator(clusterSize=1, dbs=()) -> /opt/vertica/config/admintools.confApr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: ['test -e /opt/vertica/config/d5415f948449e9d4c421b568f2411140.dat && echo good || echo missing']Apr 1 18:10:32 [19435] <BashAdapter: dbadmin@172.16.1.65>: (rc=0) ['good']

 My admintools.conf:

Spoiler
(Highlight to read)

[Configuration]
last_port = 5433
tmp_dir = /tmp
default_base = /home/dbadmin
format = 3
install_opts = --hosts '172.16.1.65' '--failure-threshold=FAIL' -L CE --accept-eula
spreadlog = False
controlsubnet = default
controlmode = broadcast

[Cluster]
hosts = 172.16.1.65

[Nodes]
node0001 = 172.16.1.65,/home/dbadmin,/home/dbadmin

[Configuration]last_port = 5433tmp_dir = /tmpdefault_base = /home/dbadminformat = 3install_opts = --hosts '172.16.1.65' '--failure-threshold=FAIL' -L CE --accept-eulaspreadlog = Falsecontrolsubnet = defaultcontrolmode = broadcast[Cluster]hosts = 172.16.1.65[Nodes]node0001 = 172.16.1.65,/home/dbadmin,/home/dbadmin

I have tried installing Vertica like 5+ times with different Vertica distribution versions and configuration options - nothing helped the error is always the same. Please help or give some suggestions how I can fix this. Thanks!

 

P. S. Please, disregard the date of the post. This is serious!!!

    1 reply

    Forum|alt.badge.img+2
    • Anonymous
    • Participating Frequently
    • April 24, 2015

    If your error message starts with something like below:

    Apr 21 19:49:59  [45008] Unable to read bootstrap node metadata.  Expected JSON DB desc, got: Stopping editor Error stopping editor: [Errno 3] No such process

     

    The error text regarding "Error stopping editor: [Errno 3] No such process" implies that the script couldn't launch a vertica process with special args to try to load and read the catalog created by an earlier boostrap routine. One potential root cause is open file limit. If the open file limit (ulimit -n, ulimit -Hn, ulimit -Sn) are too low vertica won't start. In the situation I ran into they were set to 16k and customer bumped it to 32k and then the create worked.

     

    You can test this by cd to the catalog dir and executing vertica from the command line, noting the documented recommendation is 65536 or mb of memory, whichever is highest.
    [dbadmin@scrossman6 v_af_node0001_catalog]$ /opt/vertica/bin/vertica -D .
    ERROR: Host does not meet minimum requirements:
    Not enough open file handles allowed (16192 available/32768 required); see 'ulimit -n'.

    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement