create DB failed due to node connectivity issues

Hi,

All nodes need to communicate with each other...

Make sure:

node1 can SSH to node2
node1 can SSH to node3

node2 can SSH to node1
node2 can SSH to node3

node3 can SSH to node1
node3 can SSH to node2

One of 'em might be missing the required SSH fingerprint?

Hi Jim,
Thanks for your answer but yes, all node can communicate with each other..
I also tried to stop the firewall and nothing changes.

Manually, the ssh works perfectly between dbadmin users in all hosts.

Hmm. What about the firewall? Did you open the appropriate ports?

https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/InstallationGuide/BeforeYouInstall/EnsurePortsAreAvailable.htm

The ports seems available, (none is used by another app), and the firewall don't bllock them.

I just saw that in the logs :

2017-08-22 16:42:03.260 admintools/521026:0x66f9bb1b0700 [SSHConnection._login_if_needed] : login error: Error! pty.fork() failed: out of pty devices
2017-08-22 16:42:03.260 admintools/521026:0x66f9c4797740 [AdapterConnectionPool_3.connect2_using_stored_auth] Unable to SSH as dbadmin to 188.XX.XX.95: Error! pty.fork() failed: out of pty devices
2017-08-22 16:42:03.261 admintools/521026:0x66f9ba92a700 [SSHConnection._do_login] : pexpect.spawn failed too many times: Error! pty.fork() failed: out of pty devices
Traceback (most recent call last):
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/pexpect_adapter.py", line 251, in _do_login
connectStr
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/utils/pexpect.py", line 429, in init
self._spawn (command, args)
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/utils/pexpect.py", line 537, in _spawn
raise ExceptionPexpect('Error! pty.fork() failed: ' + str(e))
ExceptionPexpect: Error! pty.fork() failed: out of pty devices

I don't understand why I get this error because my number of PTy is fixed at 4096 (/proc/sys/kernel/pty/max) and the current number of used pty (/proc/sys/kernel/pty/nr) is less than 3 on all hosts.

Maibe is this a hint ?

Thanks

Make sure that the "User Process Limit" is set correctly on each node. 4096 is probably not high enough. See:

https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/InstallationGuide/BeforeYouInstall/UserProcessLimit.htm

Hi,
I'm trying again to install vertica but face the exact same problem :
I downloaded vertica9 and also tried with verticaMC. I also verified that the passwordless ssh works between all hosts.
I'm still getting the same error (could not reach node) and in the logs I can see :

: login start: ssh -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes -l dbadmin 37.XX.XX.53
: pexpect.spawn failed: Error! pty.fork() failed: out of pty devices, trying again.
: pexpect.spawn failed too many times: Error! pty.fork() failed: out of pty devices

But when I copy the exact same command (ssh -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes -l dbadmin 37.XX.XX.53) in my terminal, the ssh works perfectly...

I finally checked the code :
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/pexpect_adapter.py", line 344, in _do_login

In the comment of this file, I saw that my problem seems to be known (line 332) :
" # Workaround for VER-2830:
# Sometimes os.forkpty() raises OSError: "out of pty devices". Did not
# find good explanation why it's happening. There are plenty
# devices(/proc/sys/kernel/pty/max=4096) available on the system level
# and not much of them are used(/proc/sys/kernel/pty/nr is not grater
# than 4) the error happens and goes away for unclear reasons. Trying
# to login 5 times in a row.
"
It seems that this is my exact problem but trying 5 fives or more does not work for me.

Do you have an idea on how I could fix it ?

Thanks
Félicien

Wow, that "Workaround" was implemented way back on 09/18/2007!

Do you have /dev/ptys correctly mounted?

Here's what I have:

[dbadmin@s18384357 ~]$ cat /etc/fstab | grep devpts
none    /dev/pts        devpts  rw,gid=5,mode=620       0       0

[dbadmin@s18384357 ~]$ mount | grep devpts
none on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)

Hi Jim ! Thanks for your answer, sorry I haven't check before today

/dev/pty doesn't seem wrong, is it ? :

dbadmin@ns321212:~$ cat /etc/fstab | grep devpts
devpts          /dev/pts        devpts  defaults        0       0
dbadmin@ns321212:~$ mount | grep devpts
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)