Eon Cluster Idle Memory Usage

Check your resource pool usage in the resource_pools and resource_pool_status tables.
Vertica will pre-allocate memory based on query budget and MEMORYSIZE defined for each resource pool.
From the status table, you can see current usage in the memory_inuse_kb field, which may explain the usage.
An idle cluster may be performing work such as tuple mover tasks (see tuple_mover_operations table), depot fetch and write (see depot_fetches and depot_uploads tables), etc. which will require working memory as larger as the ROS files being worked on.
You can also check data collector tables, notably dc_memory_info_by_minute, since it could simply be soft allocation of buffers or cache.

could you please share perf top output? which process is consuming memory?

Thanks Bryan and Sruthi for your response, looking at the addition metrics you recommended and I cannot get the mem numbers to add up. I'm going to open up a case.

Here is some additional information from Vertica support for others who may see similar situation regarding idle memory usage on eon cluster.

Vertica doesn't immediately gets rid of the memory usage when its idle, but the memory is still eligible for reallocation and gets reallocated for newer queries / vertica processes in an event they need this memory space. This is reason it would go down when the database is restarted as vertica will try to get rid of all of the data blocks in memory. This is a very harmless behavior of vertica.
I would be concerned in case the memory usage keeps growing over time to the point where vertica is not able to reclaim the memory blocks which were allocated by an earlier process which is no longer active and hence the memory block is not in use. This would be a sign of memory leak and can cause vertica to go oom (out of memory). In such events vertica will dump memory leak information in MemoryReport.txt file in the catalog directory. This file holds information of the queries which caused memory leak and hence as a workaround these queries can be re-written to avoid ooming.