Skip to main content
Question

Data Collector not collecting for 1 of 4 nodes

  • March 7, 2022
  • 9 replies
  • 8 views
bmurrell
Forum|alt.badge.img+2

Running v10.1.

The following SQL shows that node 3 is not collecting : -
SELECT node_name, first_time, last_time, memory_buffer_size_kb, disk_size_kb
FROM v_monitor.data_collector
WHERE table_name = 'dc_cpu_aggregate_by_minute'
ORDER BY table_name;

Node First Time Last Time Mem Disk
v_****node0004 2022-02-28 16:34:00 2022-03-07 15:04:00 64 25600
v****node0003 2022-02-22 21:08:00 2022-02-28 14:49:00 64 25600
v****node0002 2022-03-02 03:21:00 2022-03-07 15:04:00 64 25600
v****_node0001 2022-03-02 02:08:00 2022-03-07 15:04:00 64 25600

I've tried to reset using : -
SELECT set_data_collector_policy('CpuAggregateByMinute',64,25600)

and changing to different values, and then back, but nothing helps.

    9 replies

    SergeB
    Forum|alt.badge.img
    • SergeB
    • Participating Frequently
    • March 7, 2022

    What is the current status of your 4 nodes?

    bmurrell
    Forum|alt.badge.img+2
    • bmurrell
    • Author
    • Participating Frequently
    • March 8, 2022

    @SergeB said:
    What is the current status of your 4 nodes?

    All up and running. Just noticed that our collection of DataCollector tables into a central source wasn't working for 1 node on this cluster.

    SergeB
    Forum|alt.badge.img
    • SergeB
    • Participating Frequently
    • March 9, 2022

    Is you issue with your collection process? "Just noticed that our collection of DataCollector tables into a central source". Can you share details?

    bmurrell
    Forum|alt.badge.img+2
    • bmurrell
    • Author
    • Participating Frequently
    • March 9, 2022

    @SergeB said:
    Is you issue with your collection process? "Just noticed that our collection of DataCollector tables into a central source". Can you share details?

    Can't be the collection process. I'm checking the table v_internal.dc_cpu_aggregate_by_minute locally, and it doesn't contain details for node3.

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • March 9, 2022

    Is node3 participating in queries? The data collector logs are generated in /v__node0001_catalog/DataCollector folder. Could you please check node3 to see if it is generating logs for dc tables. Is the issue specific to cpu info tables or all the tables?

    bmurrell
    Forum|alt.badge.img+2
    • bmurrell
    • Author
    • Participating Frequently
    • March 9, 2022

    @SruthiA said:
    Is node3 participating in queries? The data collector logs are generated in /v__node0001_catalog/DataCollector folder. Could you please check node3 to see if it is generating logs for dc tables. Is the issue specific to cpu info tables or all the tables?

    Yes, node 3 is participating in queries. There are no logs for that dc table in that folder, although there is on the other 3 nodes.

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • March 9, 2022

    for the cpu, io, network related tables, we get the information from OS level and populate these dc tables. By any chance on node3 are OS metrics not populating? Please try restarting node3 and see if fixes the issue.

    bmurrell
    Forum|alt.badge.img+2
    • bmurrell
    • Author
    • Participating Frequently
    • March 10, 2022

    Restarting the database (on all nodes) fixed the issue. However, it didn't cleanly shutdown on node 3 and had to be killed. I guess that was related to this problem.

    SruthiA
    Forum|alt.badge.img+1
    • SruthiA
    • Participating Frequently
    • March 10, 2022

    @bmurrell : Thanks for the update. It looks like node3 was hung. When you experience such behavior in future, please open a support case so that we can find the exact root cause.

    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement