Skip to main content

Sudden node shutdown

  • August 5, 2014
  • 3 replies
  • 17 views
Tibor_Mozes
Forum|alt.badge.img+1
Hello,

One of the nodes on our 3 node AWS cluster started to shut down with the following message:

2014-08-03 22:26:54.328 EEcmdq:0x7f32f0cec950 [Main] <PANIC> Received fatal signal SIGSEGV.

2014-08-03 22:26:54.328 EEcmdq:0x7f32f0cec950 [Main] <PANIC> Info: si_code: 2, si_pid: -2087288392, si_uid: 32564, si_addr: 0x7f34839681b8

This has been happening the last 2 days out of the sudden, happened multiple times and only to the same node. 


Can someone help me to troubleshoot this?

Thank you,

Tibor


    3 replies

    Prasanta_Pal
    Forum|alt.badge.img
    • Prasanta_Pal
    • Participating Frequently
    • August 5, 2014
    It looks like the vertica process is killed by some thing - may be kernel
    You may run the below and see if some unusual things
    $ dmesg

    If you are a customer of vertica, you may open a case and support engineer will help you



    Forum|alt.badge.img+2
    • Anonymous
    • Participating Frequently
    • August 5, 2014
    Hi Tibor,

    If you have a PANIC vertica should had printed a back trace of the issue in the ErrorReport.txt file that you find under the catalog directory. 
    Please check there what caused the PANIC, if it is an statement, please try to run it again to see if it is reproducible.
    Please also print the back trace here and tell us what Version of Vertica you are using.
    Thanks,
    Eugenia
    Tibor_Mozes
    Forum|alt.badge.img+1
    • Tibor_Mozes
    • Author
    • Participating Frequently
    • August 5, 2014
    Hi Eugenia,

    Thanks for helping with this. Attached are 3 traces that caused the PANIC subsequently.
    It looks like all the 3 crashed were generated by 2 very similar queries (cross join on a table).
    Tried to run both queries multiple times just now but couldnt reproduce the behavior.

    Both queries ran usually in 3 minutes. Please let me know how we can prevent this happening or if I can provide anything else.

    Thank  you for your help,
    Tibor


    BEGIN BACKTRACE
    Vertica Backtrace at Sun Aug  3 22:26:53 2014
    -------------------------
    Vertica Analytic Database v7.0.1-0 $BrandId$
    vertica(v7.0.1-0) built by release@build2.verticacorp.com from releases/VER_7_0_RELEASE_BUILD_1_0_20140212@130255 on 'Wed Feb 12 19:00:56 America/New_York 2014' $BuildId$
    00400000-0498a000 r-xp 00000000 ca:41 392074                             /opt/vertica/bin/vertica
    04b89000-04e15000 rw-p 04589000 ca:41 392074                             /opt/vertica/bin/vertica
    04e15000-04f25000 rw-p 00000000 00:00 0 
    050c3000-054f7000 rw-p 00000000 00:00 0                                  [heap]
    3e79a00000-3e79a20000 r-xp 00000000 ca:41 4585                           /lib64/ld-2.12.so
    3e79c1f000-3e79c20000 r--p 0001f000 ca:41 4585                           /lib64/ld-2.12.so
    3e79c20000-3e79c21000 rw-p 00020000 ca:41 4585                           /lib64/ld-2.12.so
    3e79c21000-3e79c22000 rw-p 00000000 00:00 0 
    3e79e00000-3e79f8b000 r-xp 00000000 ca:41 4702                           /lib64/libc-2.12.so
    3e79f8b000-3e7a18a000 ---p 0018b000 ca:41 4702                           /lib64/libc-2.12.so
    3e7a18a000-3e7a18e000 r--p 0018a000 ca:41 4702                           /lib64/libc-2.12.so
    3e7a18e000-3e7a18f000 rw-p 0018e000 ca:41 4702                           /lib64/libc-2.12.so
    3e7a18f000-3e7a194000 rw-p 00000000 00:00 0 
    3e7a200000-3e7a202000 r-xp 00000000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a202000-3e7a402000 ---p 00002000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a402000-3e7a403000 r--p 00002000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a403000-3e7a404000 rw-p 00003000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a600000-3e7a617000 r-xp 00000000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a617000-3e7a817000 ---p 00017000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a817000-3e7a818000 r--p 00017000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a818000-3e7a819000 rw-p 00018000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a819000-3e7a81d000 rw-p 00000000 00:00 0 
    3e7b200000-3e7b207000 r-xp 00000000 ca:41 33714                          /lib64/librt-2.12.so
    3e7b207000-3e7b406000 ---p 00007000 ca:41 33714                          /lib64/librt-2.12.so
    3e7b406000-3e7b407000 r--p 00006000 ca:41 33714                          /lib64/librt-2.12.so
    3e7b407000-3e7b408000 rw-p 00007000 ca:41 33714                          /lib64/librt-2.12.so
    3e7b600000-3e7b622000 r-xp 00000000 ca:41 12401                          /lib64/libncurses.so.5.7
    3e7b622000-3e7b821000 ---p 00022000 ca:41 12401                          /lib64/libncurses.so.5.7
    3e7b821000-3e7b822000 rw-p 00021000 ca:41 12401                          /lib64/libncurses.so.5.7
    3e7c200000-3e7c21d000 r-xp 00000000 ca:41 3740                           /lib64/libtinfo.so.5.7
    3e7c21d000-3e7c41d000 ---p 0001d000 ca:41 3740                           /lib64/libtinfo.so.5.7
    3e7c41d000-3e7c421000 rw-p 0001d000 ca:41 3740                           /lib64/libtinfo.so.5.7
    3e7d600000-3e7d616000 r-xp 00000000 ca:41 33711                          /lib64/libgcc_s-4.4.7-20120601.so.1
    3e7d616000-3e7d815000 ---p 00016000 ca:41 33711                          /lib64/libgcc_s-4.4.7-20120601.so.1
    3e7d815000-3e7d816000 rw-p 00015000 ca:41 33711                          /lib64/libgcc_s-4.4.7-20120601.so.1
    7f35f5735000-7f35fb5c6000 r--p 00000000 ca:41 8176                       /usr/lib/locale/locale-archive
    7f35fb5c6000-7f35fc50b000 r--s 00000000 ca:41 400203                     /opt/vertica/share/icu/icudt42l.dat
    7f35fc50b000-7f360c66b000 rw-p 00000000 00:00 0 
    7f360c66b000-7f360ca7f000 rw-p 00000000 00:00 0 
    7f360ca7f000-7f360ca8b000 r-xp 00000000 ca:41 3449                       /lib64/libnss_files-2.12.so
    7f360ca8b000-7f360cc8b000 ---p 0000c000 ca:41 3449                       /lib64/libnss_files-2.12.so
    7f360cc8b000-7f360cc8c000 r--p 0000c000 ca:41 3449                       /lib64/libnss_files-2.12.so
    7f360cc8c000-7f360cc8d000 rw-p 0000d000 ca:41 3449                       /lib64/libnss_files-2.12.so
    7f360cc8d000-7f360cc92000 rw-p 00000000 00:00 0 
    7f360cc95000-7f360cca2000 rw-p 00000000 00:00 0 
    7fffdcc87000-7fffdcc9f000 rw-p 00000000 00:00 0                          [stack]
    7fffdcdff000-7fffdce00000 r-xp 00000000 00:00 0                          [vdso]
    ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

    Backtrace Generated by Error
    Signal: [0x000000000000000b] PID: [0x0000000000000803] PC: [0x0000000001699184] FP: [0x00007f3469e5a460] SIGSEGV: SEGV_ACCERR SI_ADDR : [0x00007f34839681b8]
    /opt/vertica/bin/vertica(_ZN6Basics9Backtrace11DoBacktraceEiiPvS1_+0x8cc)[0x33a011e]
    /opt/vertica/bin/vertica(_ZN6Basics20GlobalSignalHandlers14logFatalSignalEiPvS1_+0xc7)[0x341f305]
    /opt/vertica/bin/vertica[0x341f8f3]
    /lib64/libc.so.6[0x3e79e329a0]
    /opt/vertica/bin/vertica(_ZN2EE11GroupByHash3runEv+0xf9e)[0x1699184]
    /opt/vertica/bin/vertica[0x15ef298]
    /lib64/libc.so.6[0x3e79e43bf0]
    END BACKTRACE
    THREAD CONTEXT
    Thread type: EE Internal Command Queue Thread
    Request: INSERT INTO dev.brand_household_summary SELECT a.brand_id AS anchor_brand_id, b.brand_id, count(DISTINCT b.user_id) AS user_count FROM ((dev.f_household_brand a JOIN dev.f_household_brand b ON ((a.user_id = b.user_id))) JOIN dev.top_top_brands ttb ON ((b.brand_id = ttb.brand_id))) GROUP BY a.brand_id, b.brand_id
              68: Send
               0: FifoBuffer
               4: Router
               3: ValExpr
               5: Copy
    FAULT =>   9: NewEENode
              10) ExprEval         depth=(0) parent=0, peer#0; outTup nCol=4,nkey=0,inlSz=160,fixSz=32
              11) ParallelMerge    (1) 10, #0; nInputs=3; out 3,0,152,24
              12) GroupByHash      (2) 11, #0; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              13) ParallelUnion    (3) 12, #0; out 3,0,152,24
              14) NetworkRecv      (4) 13, #0; CRPstate=2; out 3,0,152,24
              15) NetworkSend      (5) 14, #0; out 3,0,152,24
              16) ParallelUnion    (6) 15, #0; nInputs=3; out 3,0,152,24
              17) GroupByHash      (7) 16, #0; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              18) ParallelUnion    (8) 17, #0; nInputs=3; out 3,0,152,24
              19) GroupByPipe      (9) 18, #0; out 3,0,152,24
              20) GroupByHash      (10) 19, #0; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              21) ParallelUnion    (11) 20, #0; out 3,0,152,24
              22) NetworkRecv      (12) 21, #0; CRPstate=2; out 3,0,152,24
              23) NetworkSend      (13) 22, #0; out 3,0,152,24
              24) ParallelUnion    (14) 23, #0; nInputs=3; out 3,0,152,24
              25) GroupByHash      (15) 24, #0; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              26) StorageUnion     (16) 25, #0; out 3,0,152,24
              27) GroupByPipe      (17) 26, #0; out 3,0,152,24
              28) Join             (18) 27, #0; nInputs=2; CRPstate=2; out 3,0,152,24
              29) Scan             (19) 28, #0; out 1,0,8,8
              30) NetworkRecv      (19) 28, #0; CRPstate=4; out 2,0,80,16
              31) NetworkSend      (20) 30, #0; out 2,0,80,16
              32) GroupByHash      (21) 31, #0; CRPstate=4;hash sca=16; kvlen=(16,0) out 2,0,80,16
              33) Join             (22) 32, #0; nInputs=2; CRPstate=4; out 2,0,80,16
              34) NetworkRecv      (23) 33, #0; CRPstate=4; out 2,0,80,16
              35) NetworkSend      (24) 34, #0; out 2,0,80,16
              36) GroupByPipe      (25) 35, #0; out 2,0,80,16
              37) StorageUnion     (26) 36, #0; out 2,0,80,16
              38) GroupByPipe      (27) 37, #0; out 2,0,80,16
              39) Scan             (28) 38, #0; out 2,0,80,16
              40) GroupByPipe      (23) 33, #0; out 1,0,72,8
              41) StorageUnion     (24) 40, #0; out 1,0,72,8
              42) GroupByPipe      (25) 41, #0; out 1,0,72,8
              43) Scan             (26) 42, #0; out 1,0,72,8
              44) GroupByHash      (15) 24, #1; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              45) StorageUnion     (16) 44, #1; out 3,0,152,24
     (PPFAULT) => GroupByHash      (id=46) (15) 24, #2; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              47) StorageUnion     (16) 46, #2; out 3,0,152,24
              48) GroupByPipe      (9) 18, #1; out 3,0,152,24
              49) GroupByHash      (10) 48, #1; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              50) ParallelUnion    (11) 49, #1; out 3,0,152,24
              51) GroupByPipe      (9) 18, #2; out 3,0,152,24
              52) GroupByHash      (10) 51, #2; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              53) ParallelUnion    (11) 52, #2; out 3,0,152,24
              54) GroupByHash      (7) 16, #1; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              55) ParallelUnion    (8) 54, #1; out 3,0,152,24
              56) GroupByHash      (7) 16, #2; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              57) ParallelUnion    (8) 56, #2; out 3,0,152,24
              58) GroupByHash      (2) 11, #1; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              59) ParallelUnion    (3) 58, #1; out 3,0,152,24
              60) GroupByHash      (2) 11, #2; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              61) ParallelUnion    (3) 60, #2; out 3,0,152,24
    Transaction: [0x00a000000026b856]
    END THREAD CONTEXT

    BEGIN BACKTRACE
    Vertica Backtrace at Sun Aug  3 23:55:27 2014
    -------------------------
    Vertica Analytic Database v7.0.1-0 $BrandId$
    vertica(v7.0.1-0) built by release@build2.verticacorp.com from releases/VER_7_0_RELEASE_BUILD_1_0_20140212@130255 on 'Wed Feb 12 19:00:56 America/New_York 2014' $BuildId$
    00400000-0498a000 r-xp 00000000 ca:41 392074                             /opt/vertica/bin/vertica
    04b89000-04e15000 rw-p 04589000 ca:41 392074                             /opt/vertica/bin/vertica
    04e15000-04f25000 rw-p 00000000 00:00 0 
    05850000-05c84000 rw-p 00000000 00:00 0                                  [heap]
    3e79a00000-3e79a20000 r-xp 00000000 ca:41 4585                           /lib64/ld-2.12.so
    3e79c1f000-3e79c20000 r--p 0001f000 ca:41 4585                           /lib64/ld-2.12.so
    3e79c20000-3e79c21000 rw-p 00020000 ca:41 4585                           /lib64/ld-2.12.so
    3e79c21000-3e79c22000 rw-p 00000000 00:00 0 
    3e79e00000-3e79f8b000 r-xp 00000000 ca:41 4702                           /lib64/libc-2.12.so
    3e79f8b000-3e7a18a000 ---p 0018b000 ca:41 4702                           /lib64/libc-2.12.so
    3e7a18a000-3e7a18e000 r--p 0018a000 ca:41 4702                           /lib64/libc-2.12.so
    3e7a18e000-3e7a18f000 rw-p 0018e000 ca:41 4702                           /lib64/libc-2.12.so
    3e7a18f000-3e7a194000 rw-p 00000000 00:00 0 
    3e7a200000-3e7a202000 r-xp 00000000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a202000-3e7a402000 ---p 00002000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a402000-3e7a403000 r--p 00002000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a403000-3e7a404000 rw-p 00003000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a600000-3e7a617000 r-xp 00000000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a617000-3e7a817000 ---p 00017000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a817000-3e7a818000 r--p 00017000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a818000-3e7a819000 rw-p 00018000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a819000-3e7a81d000 rw-p 00000000 00:00 0 
    3e7b200000-3e7b207000 r-xp 00000000 ca:41 33714                          /lib64/librt-2.12.so
    3e7b207000-3e7b406000 ---p 00007000 ca:41 33714                          /lib64/librt-2.12.so
    3e7b406000-3e7b407000 r--p 00006000 ca:41 33714                          /lib64/librt-2.12.so
    3e7b407000-3e7b408000 rw-p 00007000 ca:41 33714                          /lib64/librt-2.12.so
    3e7b600000-3e7b622000 r-xp 00000000 ca:41 12401                          /lib64/libncurses.so.5.7
    3e7b622000-3e7b821000 ---p 00022000 ca:41 12401                          /lib64/libncurses.so.5.7
    3e7b821000-3e7b822000 rw-p 00021000 ca:41 12401                          /lib64/libncurses.so.5.7
    3e7c200000-3e7c21d000 r-xp 00000000 ca:41 3740                           /lib64/libtinfo.so.5.7
    3e7c21d000-3e7c41d000 ---p 0001d000 ca:41 3740                           /lib64/libtinfo.so.5.7
    3e7c41d000-3e7c421000 rw-p 0001d000 ca:41 3740                           /lib64/libtinfo.so.5.7
    3e7d600000-3e7d616000 r-xp 00000000 ca:41 33711                          /lib64/libgcc_s-4.4.7-20120601.so.1
    3e7d616000-3e7d815000 ---p 00016000 ca:41 33711                          /lib64/libgcc_s-4.4.7-20120601.so.1
    3e7d815000-3e7d816000 rw-p 00015000 ca:41 33711                          /lib64/libgcc_s-4.4.7-20120601.so.1
    7f1284114000-7f1289fa5000 r--p 00000000 ca:41 8176                       /usr/lib/locale/locale-archive
    7f1289fa5000-7f128aeea000 r--s 00000000 ca:41 400203                     /opt/vertica/share/icu/icudt42l.dat
    7f128aeea000-7f129b04a000 rw-p 00000000 00:00 0 
    7f129b04a000-7f129b45e000 rw-p 00000000 00:00 0 
    7f129b45e000-7f129b46a000 r-xp 00000000 ca:41 3449                       /lib64/libnss_files-2.12.so
    7f129b46a000-7f129b66a000 ---p 0000c000 ca:41 3449                       /lib64/libnss_files-2.12.so
    7f129b66a000-7f129b66b000 r--p 0000c000 ca:41 3449                       /lib64/libnss_files-2.12.so
    7f129b66b000-7f129b66c000 rw-p 0000d000 ca:41 3449                       /lib64/libnss_files-2.12.so
    7f129b66c000-7f129b671000 rw-p 00000000 00:00 0 
    7f129b674000-7f129b681000 rw-p 00000000 00:00 0 
    7fffc746f000-7fffc7488000 rw-p 00000000 00:00 0                          [stack]
    7fffc74e9000-7fffc74ea000 r-xp 00000000 00:00 0                          [vdso]
    ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

    Backtrace Generated by Error
    Signal: [0x000000000000000b] PID: [0x0000000000003986] PC: [0x0000000001699184] FP: [0x00007f10cd659460] SIGSEGV: SEGV_ACCERR SI_ADDR : [0x00007f109e3721b8]
    /opt/vertica/bin/vertica(_ZN6Basics9Backtrace11DoBacktraceEiiPvS1_+0x8cc)[0x33a011e]
    /opt/vertica/bin/vertica(_ZN6Basics20GlobalSignalHandlers14logFatalSignalEiPvS1_+0xc7)[0x341f305]
    /opt/vertica/bin/vertica[0x341f8f3]
    /lib64/libc.so.6[0x3e79e329a0]
    /opt/vertica/bin/vertica(_ZN2EE11GroupByHash3runEv+0xf9e)[0x1699184]
    /opt/vertica/bin/vertica[0x15ef298]
    /lib64/libc.so.6[0x3e79e43bf0]
    END BACKTRACE
    THREAD CONTEXT
    Thread type: EE Internal Command Queue Thread
    Request: INSERT INTO dev.brand_household_summary SELECT a.brand_id AS anchor_brand_id, b.brand_id, count(DISTINCT b.user_id) AS user_count FROM ((dev.f_household_brand a JOIN dev.f_household_brand b ON ((a.user_id = b.user_id))) JOIN dev.top_top_brands ttb ON ((b.brand_id = ttb.brand_id))) GROUP BY a.brand_id, b.brand_id
              67: Send
               0: FifoBuffer
               4: Router
               3: ValExpr
               5: Copy
    FAULT =>   9: NewEENode
              10) ExprEval         depth=(0) parent=0, peer#0; outTup nCol=4,nkey=0,inlSz=160,fixSz=32
              11) ParallelMerge    (1) 10, #0; nInputs=3; out 3,0,152,24
              12) GroupByHash      (2) 11, #0; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              13) ParallelUnion    (3) 12, #0; out 3,0,152,24
              14) NetworkRecv      (4) 13, #0; CRPstate=2; out 3,0,152,24
              15) NetworkSend      (5) 14, #0; out 3,0,152,24
              16) ParallelUnion    (6) 15, #0; nInputs=3; out 3,0,152,24
              17) GroupByHash      (7) 16, #0; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              18) ParallelUnion    (8) 17, #0; nInputs=3; out 3,0,152,24
              19) GroupByPipe      (9) 18, #0; out 3,0,152,24
              20) GroupByHash      (10) 19, #0; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              21) ParallelUnion    (11) 20, #0; out 3,0,152,24
              22) NetworkRecv      (12) 21, #0; CRPstate=2; out 3,0,152,24
              23) NetworkSend      (13) 22, #0; out 3,0,152,24
              24) ParallelUnion    (14) 23, #0; nInputs=3; out 3,0,152,24
              25) GroupByHash      (15) 24, #0; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              26) StorageUnion     (16) 25, #0; out 3,0,152,24
              27) GroupByPipe      (17) 26, #0; out 3,0,152,24
              28) Join             (18) 27, #0; nInputs=2; CRPstate=2; out 3,0,152,24
              29) Scan             (19) 28, #0; out 1,0,8,8
              30) NetworkRecv      (19) 28, #0; CRPstate=4; out 2,0,80,16
              31) NetworkSend      (20) 30, #0; out 2,0,80,16
              32) GroupByHash      (21) 31, #0; CRPstate=4;hash sca=16; kvlen=(16,0) out 2,0,80,16
              33) Join             (22) 32, #0; nInputs=2; CRPstate=4; out 2,0,80,16
              34) NetworkRecv      (23) 33, #0; CRPstate=4; out 2,0,80,16
              35) NetworkSend      (24) 34, #0; out 2,0,80,16
              36) GroupByPipe      (25) 35, #0; out 2,0,80,16
              37) StorageUnion     (26) 36, #0; out 2,0,80,16
              38) GroupByPipe      (27) 37, #0; out 2,0,80,16
              39) Scan             (28) 38, #0; out 2,0,80,16
              40) GroupByPipe      (23) 33, #0; out 1,0,72,8
              41) StorageUnion     (24) 40, #0; out 1,0,72,8
              42) GroupByPipe      (25) 41, #0; out 1,0,72,8
              43) Scan             (26) 42, #0; out 1,0,72,8
              44) GroupByHash      (15) 24, #1; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              45) StorageUnion     (16) 44, #1; out 3,0,152,24
     (PPFAULT) => GroupByHash      (id=46) (15) 24, #2; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              47) StorageUnion     (16) 46, #2; out 3,0,152,24
              48) GroupByPipe      (9) 18, #1; out 3,0,152,24
              49) GroupByHash      (10) 48, #1; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              50) ParallelUnion    (11) 49, #1; out 3,0,152,24
              51) GroupByPipe      (9) 18, #2; out 3,0,152,24
              52) GroupByHash      (10) 51, #2; CRPstate=2;hash sca=19; kvlen=(24,0) out 3,0,152,24
              53) ParallelUnion    (11) 52, #2; out 3,0,152,24
              54) GroupByHash      (7) 16, #1; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              55) ParallelUnion    (8) 54, #1; out 3,0,152,24
              56) GroupByHash      (7) 16, #2; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              57) ParallelUnion    (8) 56, #2; out 3,0,152,24
              58) GroupByHash      (2) 11, #1; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              59) ParallelUnion    (3) 58, #1; out 3,0,152,24
              60) GroupByHash      (2) 11, #2; CRPstate=2;hash sca=10; kvlen=(16,8) out 3,0,152,24
              61) ParallelUnion    (3) 60, #2; out 3,0,152,24
    Transaction: [0x00a000000026bcee]
    END THREAD CONTEXT

    BEGIN BACKTRACE
    Vertica Backtrace at Mon Aug  4 01:51:17 2014
    -------------------------
    Vertica Analytic Database v7.0.1-0 $BrandId$
    vertica(v7.0.1-0) built by release@build2.verticacorp.com from releases/VER_7_0_RELEASE_BUILD_1_0_20140212@130255 on 'Wed Feb 12 19:00:56 America/New_York 2014' $BuildId$
    00400000-0498a000 r-xp 00000000 ca:41 392074                             /opt/vertica/bin/vertica
    04b89000-04e15000 rw-p 04589000 ca:41 392074                             /opt/vertica/bin/vertica
    04e15000-04f25000 rw-p 00000000 00:00 0 
    057ab000-05bdf000 rw-p 00000000 00:00 0                                  [heap]
    3e79a00000-3e79a20000 r-xp 00000000 ca:41 4585                           /lib64/ld-2.12.so
    3e79c1f000-3e79c20000 r--p 0001f000 ca:41 4585                           /lib64/ld-2.12.so
    3e79c20000-3e79c21000 rw-p 00020000 ca:41 4585                           /lib64/ld-2.12.so
    3e79c21000-3e79c22000 rw-p 00000000 00:00 0 
    3e79e00000-3e79f8b000 r-xp 00000000 ca:41 4702                           /lib64/libc-2.12.so
    3e79f8b000-3e7a18a000 ---p 0018b000 ca:41 4702                           /lib64/libc-2.12.so
    3e7a18a000-3e7a18e000 r--p 0018a000 ca:41 4702                           /lib64/libc-2.12.so
    3e7a18e000-3e7a18f000 rw-p 0018e000 ca:41 4702                           /lib64/libc-2.12.so
    3e7a18f000-3e7a194000 rw-p 00000000 00:00 0 
    3e7a200000-3e7a202000 r-xp 00000000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a202000-3e7a402000 ---p 00002000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a402000-3e7a403000 r--p 00002000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a403000-3e7a404000 rw-p 00003000 ca:41 6333                           /lib64/libdl-2.12.so
    3e7a600000-3e7a617000 r-xp 00000000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a617000-3e7a817000 ---p 00017000 ca:41 33708                          /lib64/libpthread-2.12.so
    3e7a817000-3e7a818000 r--p 00017000 ca:41 33708
    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement