Skip to main content

Vertica cluster with unbalanced work load cross nodes

  • May 30, 2017
  • 8 replies
  • 29 views

John_Y

We have a vertica cluster with 5 nodes in our company, but one of the nodes (node 5), always has significant higher work load than other nodes, high IO, memory, #number of running queries, and number of threads are all much higher.
Please refer to the following figure, can someone help to explain why?

Thank you

8 replies

RaghavA
Forum|alt.badge.img
  • Participating Frequently
  • May 31, 2017

Hi John,

Emoreno's suggestion is a good way to start investigating. Just to modify it, please add "query_type" too in above SQL to see which type of query is not evenly load balances by your application/load_balancer.

In my experience, it is seen that certain type of load creates like "LOAD" events are evenly distributed but 1 application is only firing SELECT DML's to 1 node only which causes this imbalance.

Try using this SQL
select node_name, date_trunc('day',query_start::timestamp) day_query,query_type,count(*) from query_profiles group by 1,2,3 order by 2 desc,1 limit 100;

it has to be some Application issue which is connecting to database.

Regards,
Raghav Agrawal.


John_Y
  • Author
  • New Participant
  • May 31, 2017

thank you very much for your help, those queries do help!


RaghavA
Forum|alt.badge.img
  • Participating Frequently
  • June 1, 2017
Hi John,

Glad to hear that.

ckotsidimos
Forum|alt.badge.img+1
  • Participating Frequently
  • June 1, 2017

Is your issue solved?? Have a similar issue, but concluded that a projection change is needed.


John_Y
  • Author
  • New Participant
  • June 1, 2017

we were able to find some queries which hits directly to one of our nodes, and redirect them to the load balance, but we are still having the problem of high IO usage., the IO is almost 100% all the time, although CPU and memory are reasonable. Overall queries speed is super slow. Please take a look at the graph below

Can anybody suggest us some ways to determine the reason of problem?
Thank you.


TomM
Forum|alt.badge.img
  • Participating Frequently
  • June 1, 2017

Have you heard of the script vioperf? I wonder if that would help you diagnose what's going on. https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/InstallationGuide/scripts/vioperf.htm?Highlight=vioperf


John_Y
  • Author
  • New Participant
  • June 1, 2017

Thank you, I just tested and here's the result, it looks like the throughput is really low? any way to dive deep into it?


ckotsidimos
Forum|alt.badge.img+1
  • Participating Frequently
  • June 1, 2017

@John_Y I would suggest, profile the queries the db has. Run a designer to get to see what projections you are missing and then Check the tuple mover operations