Spread not communicating between nodes

Hi,

<!-- @page { margin: 0.79in } P { margin-bottom: 0.08in } -->

Verify this 2 things in spread :

1- Check the /opt/vertica/config/vspread.conf files on all nodes – they need to be identical.

2- Do spread test with spuser to see if nodes at least are communicating between them.

You can do spuser -r test like this
/opt/vertica/spread/bin/spuser -r
At the prompt User >
enter in “j test”
j is for join and you are creating a grouping called test.

Repeat this on all nodes to confirm that they are connecting to each other. You should see the new connections come in with each node connecting.

If the spuser -r is working is fine, it is not the spread error. Can you check if there is a dbLog file or if there is any additional info in /opt/vertica/log/adminTools-dbadmin.log file?


Also Vertica recommends removing firewalls between nodes in the cluster. If you are running your cluster on private interfaces, a firewall is not needed. A firewall on the public side should only need port 5433 open. A database cluster should depend preferably on an external firewall provided by a gateway rather than Linux. Vertica is designed for the server processes in the cluster machines to work together as though they were a single powerful machine. Firewalls can slow network throughput even with the necessary ports open. If you must use firewalls, open the following ports on each host. Vertica 5433 TCP (All client connections) 32768-61000 TCP (server server) Dynamic ports , range depends on OS configuration, check /proc/sys/net/ipv4/ip_local_port_range ) Spread 4803 TCP (Client connections) 4803 UDP (Daemon Daemon) 4804 UDP (Daemon Daemon) 4805 UDP (Monitor to Daemon) (optional and only if "DangerousMonitor = yes" in config file) Administration tools (adminTools, install_vertica) 22 TCP (SSH connections within cluster) Again, Vertica does not recommend having firewall rules for the private network interfaces.



Regards,

Bhawana

 

Hi Bhawan,
Thanks for your reply. As I mentioned I checked vspread.conf file on all servers and they are identical. And I have tested with spuser and the nodes can't communicate with each other; i.e. I don't see any new connections coming. So it seem the problem is with spread not being able to communicate between nodes. 

I don't have firewall enabled. And not seeing any warnings or errors in the /opt/vertica/log/adminTools-dbadmin.log file. I've also checked /opt/vertica/log/spread_xxxxx.log file and there is no helpful information. just messages as:
[Thu 07 Nov 2013 17:22:50] Send_join: State is 4
[Thu 07 Nov 2013 17:22:51] Send_join: State is 4
[Thu 07 Nov 2013 17:22:52] Send_join: State is 4

Any ideas on how I can further troubleshoot?

Farzam are you using Amazon?

I was having the same issue until I read some documentation on hew spread works. That and this:

http://aws.amazon.com/vpc/faqs/#R4

Led me to using the -T option of the install_vertica script:

  [-T] direct transmit only: disallow UDP broadcast traffic

Now everything looks ok.