Upgrade 7.1.1.0 to 7.2.3.18 and start db fail cause?

Hi Jeff,

Vertica 7.2.3.x was considered a major release. Did you try to upgrade the system in steps ?

Try upgrading the system from v7.1.1.0 to & v7.2.1.x then from v7.2.1.x to v7.2.3x

i hope this should resolve the dependencies error which you are facing. If you still face it, then please share the vertica.log file to analyze it.

Regards,
Raghav

Hi Jeff,
Please share vertica.log so we can help you resolve issue.

Today morning , i try to upgrade to 7.1.2.21(latest version of 7.1) success , all admintools.conf is same . but upgrade to 7.2.0.1 is failed(first version of 7.2)
so i think that is 7.1 to 7.2 issue ? is it could be gcc issue?

I still facing this error , also share my vertica.log in
https://1drv.ms/u/s!AsuQLFVdzYYfgYE8O8TXvT24QMHWxQ

Please help!

Hi Jeff,

It seems that the upgrade you are tested has corrupted your UDX libraries and license file.

Initially you had v 7.1.1.0 then you tried multiple attempts of upgrade:

  5: 2017-05-11 03:06:01.395 Main:0x7be0930 [Init] <INFO> Starting up Vertica Analytic Database v7.1.1-0
202189: 2017-05-11 09:35:13.779 unknown:0x7f937af97780 [Init] <INFO> Starting up Vertica Analytic Database v7.2.3-19
216380: 2017-05-11 09:36:39.517 unknown:0x7f041576c780 [Init] <INFO> Starting up Vertica Analytic Database v7.2.3-19
230571: 2017-05-11 09:45:56.808 unknown:0x7f05a9dd1700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.1-12
240627: 2017-05-11 09:57:36.960 unknown:0x7f04ed18c780 [Init] <INFO> Starting up Vertica Analytic Database v7.2.3-12
267798: 2017-05-11 10:27:58.203 unknown:0x7fe27f5e9720 [Init] <INFO> Starting up Vertica Analytic Database v7.2.0-1
284010: 2017-05-11 10:31:30.339 unknown:0x7fa503159700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.2-21
294257: 2017-05-11 10:34:08.060 unknown:0x7f7fd6642700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.2-21
309187: 2017-05-11 11:02:52.929 unknown:0x7f05009a9720 [Init] <INFO> Starting up Vertica Analytic Database v7.2.0-1
390510: 2017-05-11 11:06:36.209 unknown:0x7ff03d649700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.2-21

Your main problem arised when you attempted to upgrade the system from v7.1.1.0 to v7.2.3-19. There was an INTERNAL error raised which caused corruption in your license.

2017-05-11 09:37:03.431 Main:0x97af340 [Catalog] <INFO> Existing License rawString is empty
2017-05-11 09:37:03.432 Main:0x97af340 [Catalog] <INFO> Internal error that the rawLicenseString parsing results is Vertica license cannot be read: check path and permissions 
2017-05-11 09:37:03.432 Main:0x97af340 [Catalog] <INFO> Existing License rawString is empty
2017-05-11 09:37:03.432 Main:0x97af340 [Catalog] <INFO> Internal error that the rawLicenseString parsing results is Vertica license cannot be read: check path and permissions 
2017-05-11 09:37:03.432 Main:0x97af340 <PANIC> @v_aipt_node0001: V2001/6570: License is tampered  License  was tampered, signature verification failed License f was tampered, signature verification failed
    HINT:  Contact Vertica to resolve the issue
    LOCATION:  doCatalogStartupTasks, /scratch_a/release/svrtar26587/vbuild/vertica/Catalog/CatalogHooks.cpp:2967
2017-05-11 09:37:04.837 Main:0x97af340 [Main] <PANIC> Wrote backtrace to ErrorReport.txt

May i ask for ErrorReport.txt file ?

But this file must be overwritten by other upgrades you attempted. check the timestamp in that file. It contains Stack Traces about why upgrade failed.

Also, do you have a copy of your license file ?
I would suggest you to attempt to run update_vertica using your backup license file.

Sample Command:

/opt/vertica/sbin/update_vertica -r /tmp/Vertica/vertica-7.2.3-19.x86_64.RHEL6.rpm -L /tmp/license.key

Regards,
Raghav Agrawal

Hi Jeff,

it looks your database is experiencing corrupted UDX and License keys. You have attempted to upgrade the system to many version now.

  5: 2017-05-11 03:06:01.395 Main:0x7be0930 [Init] <INFO> Starting up Vertica Analytic Database v7.1.1-0
202189: 2017-05-11 09:35:13.779 unknown:0x7f937af97780 [Init] <INFO> Starting up Vertica Analytic Database v7.2.3-19
216380: 2017-05-11 09:36:39.517 unknown:0x7f041576c780 [Init] <INFO> Starting up Vertica Analytic Database v7.2.3-19
230571: 2017-05-11 09:45:56.808 unknown:0x7f05a9dd1700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.1-12
240627: 2017-05-11 09:57:36.960 unknown:0x7f04ed18c780 [Init] <INFO> Starting up Vertica Analytic Database v7.2.3-12
267798: 2017-05-11 10:27:58.203 unknown:0x7fe27f5e9720 [Init] <INFO> Starting up Vertica Analytic Database v7.2.0-1
284010: 2017-05-11 10:31:30.339 unknown:0x7fa503159700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.2-21
294257: 2017-05-11 10:34:08.060 unknown:0x7f7fd6642700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.2-21
309187: 2017-05-11 11:02:52.929 unknown:0x7f05009a9720 [Init] <INFO> Starting up Vertica Analytic Database v7.2.0-1
390510: 2017-05-11 11:06:36.209 unknown:0x7ff03d649700 [Init] <INFO> Starting up Vertica Analytic Database v7.1.2-21

in your first attempt to upgrade the system to v7.2.3-19, there is a license corruption.

2017-05-11 09:35:38.348 Main:0x80e8340 [Catalog] <INFO> License Audit Record Hash Check Passed.
2017-05-11 09:35:38.348 Main:0x80e8340 [Catalog] <INFO> Full License Hash Check Passed.
2017-05-11 09:35:38.348 Main:0x80e8340 [Catalog] <INFO> Existing License rawString is empty
2017-05-11 09:35:38.348 Main:0x80e8340 [Catalog] <INFO> Internal error that the rawLicenseString parsing results is Vertica license cannot be read: check path and permissions 
2017-05-11 09:35:38.348 Main:0x80e8340 [Catalog] <INFO> Existing License rawString is empty
2017-05-11 09:35:38.348 Main:0x80e8340 [Catalog] <INFO> Internal error that the rawLicenseString parsing results is Vertica license cannot be read: check path and permissions 
2017-05-11 09:35:38.349 Main:0x80e8340 <PANIC> @v_aipt_node0001: V2001/6570: License is tampered  License  was tampered, signature verification failed License f was tampered, signature verification failed
    HINT:  Contact Vertica to resolve the issue
    LOCATION:  doCatalogStartupTasks, /scratch_a/release/svrtar26587/vbuild/vertica/Catalog/CatalogHooks.cpp:2967
2017-05-11 09:35:39.430 Main:0x80e8340 [Main] <PANIC> Wrote backtrace to ErrorReport.txt

Is it possible for you to share the ErrorReport.txt file too ?

I have a recommendation for you, by any chance you have backup of your license file. If yes, they you may attempt to upgrade the vertica again using your backup license file.

Sample command :
/opt/vertica/sbin/update_vertica -r /tmp/vertica-7.2.3-19.x86_64.RHEL6.rpm -L /tmp/license.key

Regards,
Raghav

ErrorReport.txt

https://1drv.ms/t/s!AsuQLFVdzYYfgYE9I0uKgccnSVh1DQ

Hi Jeff,

Thanks for sharing the file. It appears you should consider fixing your license file to start the database.

Do you have backup of that file ?

If not, then you need to raise the case to verticaSupport to get a new file.

Regards

Today morning , i try to upgrade to 7.1.2.21(latest version of 7.1) success , all admintools.conf is same . but upgrade to 7.2.0.1 is failed(first version of 7.2)
so i think that is 7.1 to 7.2 issue ? is it could be gcc issue?

I still facing this error , also share my vertica.log in https://1drv.ms/u/s!AsuQLFVdzYYfgYE8O8TXvT24QMHWxQ
Please help!

Hi
I feed the 1TB enterprise edtion license key and reupgrade. still facing same error. below is dbLog error message.

05/12/17 08:27:36 SP_disconnect: mbox=9, pid=509281, send_group=#node_a#N010009123011
/===========================================================================\
| The Spread Toolkit. |
| Copyright (c) 1993-2014 Spread Concepts LLC |
| All rights reserved. |
| |
| The Spread package is licensed under the Spread Basic IPv6 Support |
| for HP Vertica Test and Evaluation License |
| You may only use this software in compliance with the License. |
| A copy of the license is included in the source distribution. |
| |
| This product uses software developed by Spread Concepts LLC for use |
| in the Spread toolkit. For more information about Spread, |
| see http://www.spread.org |
| |
| This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF |
| ANY KIND, either express or implied. |
| |
| Creators: |
| Yair Amir yairamir@cs.jhu.edu |
| Michal Miskin-Amir michal@spreadconcepts.com |
| Jonathan Stanton jstanton@gwu.edu |
| John Schultz jschultz@spreadconcepts.com |
| |
| Major Contributors: |
| Ryan Caudy rcaudy@gmail.com - contribution to process groups.|
| Claudiu Danilov claudiu@acm.org - scalable, wide-area support. |
| Cristina Nita-Rotaru crisn@cs.purdue.edu - GC security. |
| Theo Schlossnagle jesus@omniti.com - Perl, autoconf, old skiplist. |
| Dan Schoenblum dansch@cnds.jhu.edu - Java interface. |
| |
| Special thanks to the following for discussions and ideas: |
| Ken Birman, Danny Dolev, Jacob Green, Mike Goodrich, Ben Laurie, |
| David Shaw, Gene Tsudik, Robbert VanRenesse. |
| |
| Partial funding provided by the Defense Advanced Research Project Agency |
| (DARPA) and the National Security Agency (NSA) 2000-2004. The Spread |
| toolkit is not necessarily endorsed by DARPA or the NSA. |
| |
| For a full list of contributors, see Readme.txt in the distribution. |
| |
| WWW: www.spread.org www.spreadconcepts.com |
| Contact: info@spreadconcepts.com |
| |
| Version 4.02.00 Vertica IPv6-3 Built 19/Jan/2015 |
\===========================================================================/
Conf_load_conf_file: using file: /v01/db/AIPT/v_aipt_node0001_catalog/spread.conf
Setting active IP version to 0
Successfully configured Segment 0 [10.9.123.11]:4803 with 1 procs:
N010009123011: 10.9.123.11
Successfully configured Segment 1 [10.9.123.12]:4803 with 1 procs:
N010009123012: 10.9.123.12
Successfully configured Segment 2 [10.9.123.13]:4803 with 1 procs:
N010009123013: 10.9.123.13
Connecting to spread at 4803
Connected to spread on local domain socket /opt/vertica/spread/tmp/4803
auto restart closing socket
05/12/17 08:39:23 SP_connect: DEBUG: Auth list is: NULL
05/12/17 08:39:23 SP_connect: connected with private group(21 bytes): #node_a#N010009123011; mbox=14, pid=514770

Hi Jeff,

There is no indication in ErrorReport.txt file about the gcc lib errors.

It seems your library files are corrupted and there is no savior.

Your vertica.log file is full of incompatibility errors as shown below. if you have a valid backup of your system, then please consider dropping this instance and recreating it.

As my recommendation, your vbr backup must be taken on v7.1.10 so, you should consider installing this version only. Restore the data and then perform the upgrade in incremental steps.

v7.1.1.0 -> v7.2.1.0
v7.2.1.0 -> 7.2.3.19

Errors :
2017-05-11 11:07:06.601 Main:0x8b465d0 [Recover] Error during setting up function, message: Library built with unsupported version of Vertica SDK [Version: v7.1.1-0, Revision: 148158]
2017-05-11 11:07:06.603 Main:0x8b465d0 [Command] (fff0000000000cc1) Library Handle: 0xb229200
2017-05-11 11:07:06.603 Main:0x8b465d0 [Command] Library file has been unloaded successfully
2017-05-11 11:07:06.603 Main:0x8b465d0 [Recover] Error during setting up function, message: Library built with unsupported version of Vertica SDK [Version: v7.1.1-0, Revision: 148158]
2017-05-11 11:07:06.604 Main:0x8b465d0 [Command] (fff0000000000cc1) Library Handle: 0xb229200
2017-05-11 11:07:06.604 Main:0x8b465d0 [Command] Library file has been unloaded successfully
2017-05-11 11:07:06.604 Main:0x8b465d0 [Recover] Error during setting up function, message: Library built with unsupported version of Vertica SDK [Version: v7.1.1-0, Revision: 148158]
2017-05-11 11:07:06.606 Main:0x8b465d0 [Command] (fff0000000000cc1) Library Handle: 0xb229200
2017-05-11 11:07:06.606 Main:0x8b465d0 [Command] Library file has been unloaded successfully
2017-05-11 11:07:06.606 Main:0x8b465d0 [Recover] Error during setting up function, message: Library built with unsupported version of Vertica SDK [Version: v7.1.1-0, Revision: 148158]
2017-05-11 11:07:06.607 Main:0x8b465d0 [Command] (fff0000000000cc1) Library Handle: 0xb229200
2017-05-11 11:07:06.608 Main:0x8b465d0 [Command] Library file has been unloaded successfully
2017-05-11 11:07:06.608 Main:0x8b465d0 [Recover] Error during setting up function, message: Library built with unsupported version of Vertica SDK [Version: v7.1.1-0, Revision: 148158]
2017-05-11 11:07:06.609 Main:0x8b465d0 [Command] (fff0000000000cc1) Library Handle: 0xb229200
2017-05-11 11:07:06.609 Main:0x8b465d0 [Command] Library file has been unloaded successfully
2017-05-11 11:07:06.609 Main:0x8b465d0 [Recover] Error during setting up function, message: Library built with unsupported version of Vertica SDK [Version: v7.1.1-0, Revision: 148158]
2017-05-11 11:07:06.610 Main:0x8b465d0 [Command] (fff0000000000cc1) Library Handle: 0xb229200

Regards,
Raghav