Skip to main content

How can we scale out and in Vertica cluster in Eon mode automatically?

  • June 19, 2018
  • 2 replies
  • 14 views

kanako_o
Forum|alt.badge.img

Our existing customer plans to migrate from Enterprise mode to Eon mode. Currently they are asking us which would be the best way to scale out/in Vertica cluster in Eon mode automatically.

What they want to achieve is as follows:

Monday

6 nodes are UP.

From Tuesday to Sunday

3 nodes are UP.

So, they want to do the below steps automatically:

In case of scaling out

(1) Start 3 nodes (EC2)
(2) Add 3 nodes in Vertica

In case of scaling in

(3) Remove 3 nodes in Vertica
(4) Stop 3 nodes (EC2)

Our partner is thinking to utilize AWS Lambda regarding step (1) and (4). What kind of way to scale out/in Vertica cluster do we recommend regarding the above four steps usually?

Any use case, idea or comment would be highly appreciated!

Thanks,
Kanako

2 replies

imarchenko
  • New Participant
  • June 19, 2018

There are some initial ideas floating around about this on Confluence here confluence.verticacorp.com:8080/display/PM/Automatic+Scaling+of+Eon+Cluster.

The short answer is that it is possible with Auto Scaling Groups in AWS. This can be programmatically done by adding/removing nodes using admintools. https://my.vertica.com/docs/9.1.x/HTML/index.htm#Authoring/AdministratorsGuide/AdminTools/WritingAdministrationToolsScripts.htm However, since it takes so long to scale up/down Vertica nodes on EON (20-30 minutes potentially), a reactive approach (scale up rapidly in respond to a workload increase) is probably not the best approach. A proactive approach would work better, meaning you scale up or down ahead of a planned workload change (like scaling up for business hours and down at night).

The other current limitation of EON is that scaling up or down will allow you to increase query concurrency since you're introducing more execution groups. This does not necessarily mean you will increase the performance of the cluster for individual queries. This is because the shard count needs to remain constant in current versions of EON.


kanako_o
Forum|alt.badge.img
  • Author
  • Participating Frequently
  • June 20, 2018

Thanks a lot for your valuable comment on this!
I assume that there is no actual use case regarding Auto Scaling Groups in AWS with Eon yet however it would be a good option.

Regarding this end user, they want to have more nodes only every Monday. So, this would be planned as proactive. They want to add more nodes in terms of concurrency not individual performance.