Hello , i have a vertica setup cluster with 7 nodes and all of them went down. i followed this link that i found online https://www.vertica.com/blog/what-should-i-do-when-the-database-node-is-down/ and did all the steps as mentioned and no luck. Could someone reach out if they have gone through the same situation. This is one of of development setup and currently proceeding with a fresh install so that i can initialize the cluster again. I am worried if the same happens on a production environment.
Question
Vertica node is down and not able to bring it up
+1+1What is the error message in logs?
+1Which logs would you like to see ?
+1please share me vertica.log and startup.log
+1I was checking the spread and it showed me this.
+1I was checking the spread and it showed me this
+1It might be possible that spread is down. please share me vertica.log and startup.log.
+1{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.460"
}
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 273,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 2872,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 4341580,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Applying transaction log (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 4341580,
"node" : "v_insights_data_node0001",
"progress" : 4341580,
"stage" : "Reading Catalog",
"text" : "Applying transaction log (bytes)",
"timestamp" : "2019-09-10 01:12:15.642"
}
{
"goal" : 3003,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Indexing Objects",
"timestamp" : "2019-09-10 01:12:15.642"
}
{
"goal" : 3003,
"node" : "v_insights_data_node0001",
"progress" : 2743,
"stage" : "Reading Catalog",
"text" : "Indexing Objects",
"timestamp" : "2019-09-10 01:12:15.647"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Connecting to Spread",
"text" : "Connecting to spread /opt/vertica/spread/tmp/4803",
"timestamp" : "2019-09-10 01:12:15.660"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.661"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 5565650,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.677"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 13333375,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.699"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 19271376,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.718"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 29272721,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.740"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 35040684,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.757"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 48870052,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.797"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 54867564,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.815"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Check Storage",
"text" : "Removing unnecessary storage files",
"timestamp" : "2019-09-10 01:12:15.857"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.349"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 22,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.350"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 44,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.350"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 66,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 88,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 110,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 132,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 154,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 172,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 6,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 6,
"node" : "v_insights_data_node0001",
"progress" : 6,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Prepare to be invited",
"timestamp" : "2019-09-10 01:12:22.000"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Prepare to be invited",
"timestamp" : "2019-09-10 01:12:24.000"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Ready to be invited",
"timestamp" : "2019-09-10 01:12:24.661"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Invited",
"timestamp" : "2019-09-10 01:12:31.993"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Startup Complete",
"text" : "Node is UP",
"timestamp" : "2019-09-10 01:12:32.415"
}
+1Startup.log shows node is UP right? What is the output of
admintools -t list_allnodes
+1Yes , but could you check the timestamp on the log file ? The data
population stopped on 11 th October and today is 16 th October .
population stopped on 11 th October and today is 16 th October .
+1Please share vertica.log from Oct 11 to review root cause
+1{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.460"
}
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 273,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 2599,
"node" : "v_insights_data_node0001",
"progress" : 2872,
"stage" : "Reading Catalog",
"text" : "Reading Checkpoint (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 4341580,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Applying transaction log (bytes)",
"timestamp" : "2019-09-10 01:12:15.461"
}
{
"goal" : 4341580,
"node" : "v_insights_data_node0001",
"progress" : 4341580,
"stage" : "Reading Catalog",
"text" : "Applying transaction log (bytes)",
"timestamp" : "2019-09-10 01:12:15.642"
}
{
"goal" : 3003,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Reading Catalog",
"text" : "Indexing Objects",
"timestamp" : "2019-09-10 01:12:15.642"
}
{
"goal" : 3003,
"node" : "v_insights_data_node0001",
"progress" : 2743,
"stage" : "Reading Catalog",
"text" : "Indexing Objects",
"timestamp" : "2019-09-10 01:12:15.647"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Connecting to Spread",
"text" : "Connecting to spread /opt/vertica/spread/tmp/4803",
"timestamp" : "2019-09-10 01:12:15.660"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.661"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 5565650,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.677"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 13333375,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.699"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 19271376,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.718"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 29272721,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.740"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 35040684,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.757"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 48870052,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.797"
}
{
"goal" : 55301669,
"node" : "v_insights_data_node0001",
"progress" : 54867564,
"stage" : "Read DataCollector",
"text" : "Inventory files (bytes)",
"timestamp" : "2019-09-10 01:12:15.815"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Check Storage",
"text" : "Removing unnecessary storage files",
"timestamp" : "2019-09-10 01:12:15.857"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.349"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 22,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.350"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 44,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.350"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 66,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 88,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 110,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.351"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 132,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 154,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 172,
"node" : "v_insights_data_node0001",
"progress" : 172,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 6,
"node" : "v_insights_data_node0001",
"progress" : 0,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"goal" : 6,
"node" : "v_insights_data_node0001",
"progress" : 6,
"stage" : "Check Storage",
"text" : "Confirming storage matches catalog (files)",
"timestamp" : "2019-09-10 01:12:16.352"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Prepare to be invited",
"timestamp" : "2019-09-10 01:12:22.000"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Prepare to be invited",
"timestamp" : "2019-09-10 01:12:24.000"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Ready to be invited",
"timestamp" : "2019-09-10 01:12:24.661"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Waiting for Cluster Invite",
"text" : "Invited",
"timestamp" : "2019-09-10 01:12:31.993"
}
{
"node" : "v_insights_data_node0001",
"stage" : "Startup Complete",
"text" : "Node is UP",
"timestamp" : "2019-09-10 01:12:32.415"
}
@SruthiA said:
Startup.log shows node is UP right? What is the output ofadmintools -t list_allnodes
dbadmin@pl2:/$ admintools -t list_allnodes
Node | Host | State | Version | DB
--------------------------+---------+-------+------------------+---------------
v_insights_data_node0001 | 3.0.0.1 | DOWN | unavailable | insights_data
v_insights_data_node0002 | 3.0.0.2 | DOWN | unavailable | insights_data
v_insights_data_node0003 | 3.0.0.3 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0004 | 3.0.0.4 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0005 | 3.0.0.5 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0006 | 3.0.0.6 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0007 | 3.0.0.7 | DOWN | vertica-8.1.1.10 | insights_data
+1
+1
+1Based on the below output, I recommend you to start node3 till node7. Once those are UP, you can try bringing node1 and node2 UP
dbadmin@pl2:/$ admintools -t list_allnodes
Node | Host | State | Version | DB
--------------------------+---------+-------+------------------+---------------
v_insights_data_node0001 | 3.0.0.1 | DOWN | unavailable | insights_data
v_insights_data_node0002 | 3.0.0.2 | DOWN | unavailable | insights_data
v_insights_data_node0003 | 3.0.0.3 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0004 | 3.0.0.4 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0005 | 3.0.0.5 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0006 | 3.0.0.6 | DOWN | vertica-8.1.1.10 | insights_data
v_insights_data_node0007 | 3.0.0.7 | DOWN | vertica-8.1.1.10 | insights_data
Sign up
Already have an account? Login
Welcome to the Rocket Forum!
Please log in or register:
Employee Login | Registration Member Login | RegistrationEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.
77 4th Avenue
Waltham, MA 02451 USA
