Unable to read database catalogs

Do you have ip tables turned on? Can you share me /opt/vertica/log/admintools.log

Thanks for the reply.

No IP tables are off. Sorry, I do not have the log file since we had to do kickstart to recover the setup.

Hi, I am facing a similar problem. The weird fact is that my colleague, who is working from US (and I am located in Europe), is able to start the database without issues (we are using the same commands, with the same user and permissions). The server is hosted on AWS EC2 instance (with his company account) in the US region. The post caught my attention because it is the only one that mentions iptables. I thought that the problem could be related to different IPs from which we are accessing the server.
Did you solve this issue? If yes, can you please provide the answer?
Thanks!

The output of admintools.log file is too long for a single post, so I give the rest of it in the next reply.
2020-09-22 11:06:50.818 admintools/19137:0x7f6171018700 [root.setup_custom_logging] <INFO> New log for 'admintools' 2020-09-22 11:06:50.819 admintools/19137:0x7f6171018700 [root.setup_custom_logging] <INFO> sys.argv: /opt/vertica/bin/admintools -t start_db -d <DB_NAME_PLACEHOLDER> 2020-09-22 11:06:51.877 admintools/19137:0x7f6171018700 [ATMain.main] <INFO> ATMain.py:main called with args: ['/opt/vertica/bin/admintools', '-t', 'start_db', '-d', '<DB_NAME_PLACEHOLDER>'] 2020-09-22 11:06:51.882 admintools/19137:0x7f6171018700 [SshAuth.setup_from_admintools_conf_if_needed] <INFO> Setup complete, default ssh user is dbadmin, default ssh options are -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes. 2020-09-22 11:06:51.882 admintools/19137:0x7f6171018700 [adminExec.__init__] <INFO> Unique Ports: False 2020-09-22 11:06:51.882 admintools/19137:0x7f6171018700 [adminExec._is_current_user_conf_owner] <INFO> Owner of admintools.conf (<USERNAME_PLACEHOLDER>) is dbadmin. 2020-09-22 11:06:51.882 admintools/19137:0x7f6171018700 [adminExec.isOkToRun] <INFO> 2020-09-22 11:06:51.882 admintools/19137:0x7f6171018700 [ATMain.main] <INFO> Use unique ports = False 2020-09-22 11:06:51.883 admintools/19137:0x7f6171018700 [adminExec._is_current_user_conf_owner] <INFO> Owner of admintools.conf (<USERNAME_PLACEHOLDER>) is dbadmin. 2020-09-22 11:06:51.883 admintools/19137:0x7f6171018700 [adminExec.isOkToRun] <INFO> 2020-09-22 11:06:52.002 admintools/19143:0x7f863ae00700 [root.setup_custom_logging] <INFO> New log for 'admintools' 2020-09-22 11:06:52.002 admintools/19143:0x7f863ae00700 [root.setup_custom_logging] <INFO> sys.argv: '/opt/vertica/bin/TXTlcheck.py' /opt/vertica/config /opt/vertica/bin 2020-09-22 11:06:52.008 admintools/19137:0x7f6171018700 [adminExec.setInteractive] <INFO> setting to True 2020-09-22 11:06:52.008 admintools/19137:0x7f6171018700 [adminExec._gcs_filterCandidatesCollectDbNodeLists] <INFO> unobserved entry: 127.0.0.1-5433; dbname: <DB_NAME_PLACEHOLDER>; orignode: v_<DB_NAME_PLACEHOLDER>_node0001 2020-09-22 11:06:52.011 admintools/19137:0x7f6171018700 [NewSSH.collectProcessResult] <INFO> Running ['/opt/vertica/bin/vsql', '--no-vsqlrc', '-n', '-p', '5433', '-h', '127.0.0.1'] with env {'SSH_ASKPASS': '/usr/libexec/openssh/gnome-ssh-askpass', 'LC_CTYPE': 'UTF-8', 'LESSOPEN': '||/usr/bin/lesspipe.sh %s', 'VSQL_OPTIONS': 'clusterstatus', 'CVS_RSH': 'ssh', 'LOGNAME': 'dbadmin', 'USER': 'dbadmin', 'QTDIR': '/usr/lib64/qt-3.3', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/opt/vertica/bin', 'LD_LIBRARY_PATH': '/opt/vertica/lib', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm-256color', 'SHELL': '/bin/bash', 'TZ': 'America/Chicago', 'SHLVL': '2', 'G_BROKEN_FILENAMES': '1', 'HISTSIZE': '1000', 'SUDO_USER': 'ec2-user', 'HOME': '/home/dbadmin', 'USERNAME': 'root', 'SUDO_UID': '500', '_': '/opt/vertica/bin/admintools', 'SUDO_COMMAND': '/bin/su', 'SUDO_GID': '500', 'HOSTNAME': 'ip-1<IP_PLACEHOLDER>.ec2.internal', 'PWD': '/home/ec2-user', 'MAIL': '/var/spool/mail/ec2-user', 'LS_COLORS': 'rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lz=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.bz=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.rar=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:'} 2020-09-22 11:06:52.153 admintools/19137:0x7f616952f700 [SshAuth.get_ssh_info_default_user] <INFO> Connection info for 127.0.0.1 is SshConnectionInfo: ssh_user=dbadmin api_user=None ident=None env={'PYTHON_EGG_CACHE': '/tmp'} options=-oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes 2020-09-22 11:06:52.159 admintools/19137:0x7f616952f700 [BashAdapter.connect_using_stored_auth] <INFO> Connected to host 127.0.0.1 2020-09-22 11:06:52.160 admintools/19137:0x7f616952f700 [BashAdapter._exec_module_impl] <INFO> Running command ['/opt/vertica/oss/python/bin/python', '-m', 'vertica.engine.api.at_runner', '--no-prompt', '--module', 'vertica.engine.api.vertica_process_is_up']#{"catalogpath": "/vertica/data/<DB_NAME_PLACEHOLDER>/v_<DB_NAME_PLACEHOLDER>_node0001_catalog"} 2020-09-22 11:06:52.389 at_exec/19152:0x7ff9e7c85700 [root.setup_custom_logging] <INFO> New log for 'at_exec'

Log file continues:
2020-09-22 11:06:52.390 at_exec/19152:0x7ff9e7c85700 [root.setup_custom_logging] <INFO> sys.argv: '/opt/vertica/share/eggs/vertica/engine/api/at_runner.py' --no-prompt --module 'vertica.engine.api.vertica_process_is_up' 2020-09-22 11:06:52.391 at_exec/19152:0x7ff9e7c85700 [ATRunner._parse_command] <INFO> Reading a line from stdin... 2020-09-22 11:06:52.391 at_exec/19152:0x7ff9e7c85700 [ATRunner._parse_command] <INFO> Reading complete 2020-09-22 11:06:52.391 at_exec/19152:0x7ff9e7c85700 [ATRunner.exec_module] <INFO> ATRunner exec_module: command: <ATCommand> module=None version=1.0 args={u'catalogpath': u'/vertica/data/<DB_NAME_PLACEHOLDER>/v_<DB_NAME_PLACEHOLDER>_node0001_catalog'} 2020-09-22 11:06:52.391 at_exec/19152:0x7ff9e7c85700 [vertica_process_is_up.run_in_subprocess] <INFO> run_in_subprocess: ['/opt/vertica/bin/vertica', '--status', '-D', u'/vertica/data/<DB_NAME_PLACEHOLDER>/v_<DB_NAME_PLACEHOLDER>_node0001_catalog']None 2020-09-22 11:06:52.591 at_exec/19152:0x7ff9e7c85700 [vertica_process_is_up.run_in_subprocess] <INFO> run_in_subprocess: ['ps', '-C', 'vertica', '-o', 'args']None 2020-09-22 11:06:52.639 admintools/19137:0x7f6169f30700 [BashAdapter._exec_module_impl] <INFO> Running command ['/opt/vertica/oss/python/bin/python', '-m', 'vertica.engine.api.at_runner', '--no-prompt', '--module', 'vertica.engine.api.compute_vdatabase']#{"catalogpath": "/vertica/data/<DB_NAME_PLACEHOLDER>/v_<DB_NAME_PLACEHOLDER>_node0001_catalog"} 2020-09-22 11:06:52.856 at_exec/19159:0x7ff3593e2700 [root.setup_custom_logging] <INFO> New log for 'at_exec' 2020-09-22 11:06:52.857 at_exec/19159:0x7ff3593e2700 [root.setup_custom_logging] <INFO> sys.argv: '/opt/vertica/share/eggs/vertica/engine/api/at_runner.py' --no-prompt --module 'vertica.engine.api.compute_vdatabase' 2020-09-22 11:06:52.858 at_exec/19159:0x7ff3593e2700 [ATRunner._parse_command] <INFO> Reading a line from stdin... 2020-09-22 11:06:52.858 at_exec/19159:0x7ff3593e2700 [ATRunner._parse_command] <INFO> Reading complete 2020-09-22 11:06:52.859 at_exec/19159:0x7ff3593e2700 [ATRunner.exec_module] <INFO> ATRunner exec_module: command: <ATCommand> module=None version=1.0 args={u'catalogpath': u'/vertica/data/<DB_NAME_PLACEHOLDER>/v_<DB_NAME_PLACEHOLDER>_node0001_catalog'} 2020-09-22 11:06:52.899 admintools/19137:0x7f6171018700 [adminExec.display] <INFO> Unable to read database catalogs - cannot start database.