Skip to main content
Question

Get recently loaded partitions

  • February 11, 2019
  • 3 replies
  • 11 views

mkheir
Forum|alt.badge.img+1
  • Participating Frequently

Hello,

I m working on preparing a load table logic using partitions.

Tables are partitioned by a month period.
Depending on the content of input files, I may have to load only the latest period, but in certain cases, I may get data from old periods (old partitions) to be loaded.

Once data is ready on staging tables, I have to move it the final table.

How can I get partitions keys of only recently (last 3 hours) loaded or updated partitions? so I can use them in functions like COPY_PARTITIONS_TO_TABLE?

Many thanks.
mkheir.

3 replies

Jim_Knicely
Forum|alt.badge.img+2
  • Participating Frequently
  • February 11, 2019

Maybe by checking the table's EPOCH column?

Example:

dbadmin=> CREATE TABLE parts (c1 INT NOT NULL, c2 INT) PARTITION BY c1;
CREATE TABLE

dbadmin=> INSERT INTO parts SELECT 1, 1;
 OUTPUT
--------
      1
(1 row)

dbadmin=> INSERT INTO parts SELECT 1, 2;
 OUTPUT
--------
      1
(1 row)

dbadmin=> COMMIT;
COMMIT

dbadmin=> INSERT INTO parts SELECT 2, 3;
 OUTPUT
--------
      1
(1 row)

dbadmin=> INSERT INTO parts SELECT 3, 5;
 OUTPUT
--------
      1
(1 row)

dbadmin=> COMMIT;
COMMIT

dbadmin=> SELECT c1 AS partition_key
dbadmin->   FROM parts
dbadmin->  WHERE parts.epoch = (SELECT MAX(epoch)
dbadmin(>                         FROM parts);
 partition_key
---------------
             2
             3
(2 rows)

dbadmin=> UPDATE parts SET c2 = 10 WHERE c1 = 1 AND c2 = 1;
 OUTPUT
--------
      1
(1 row)

dbadmin=> COMMIT;
COMMIT

dbadmin=> SELECT c1 AS partition_key
dbadmin->   FROM parts
dbadmin->  WHERE parts.epoch = (SELECT MAX(epoch)
dbadmin(>                         FROM parts);
 partition_key
---------------
             1
(1 row)

But you asked for activity in the past 3 hours, so you will need to find the latest Epoch 3 hours ago. One option to get that info is to query the DC_REQUESTS_ISSUED table hoping that it has records that are at least 3 hours old :)

dbadmin=> SELECT DISTINCT c1 AS partition_key
dbadmin->   FROM parts
dbadmin->  WHERE epoch >= (SELECT MIN(query_start_epoch)
dbadmin(>                    FROM dc_requests_issued
dbadmin(>                   WHERE time >= (SYSDATE::TIMESTAMP - INTERVAL '3 HOURS'));
 partition_key
---------------
             1
             2
             3
(3 rows)

But for my example, I will use a smaller interval.

dbadmin=> INSERT INTO parts SELECT 5, 100;
 OUTPUT
--------
      1
(1 row)

dbadmin=> INSERT INTO parts SELECT 6, 1000;
 OUTPUT
--------
      1
(1 row)

dbadmin=> COMMIT;
COMMIT

dbadmin=> SELECT DISTINCT c1 AS partition_key
dbadmin->   FROM parts
dbadmin->  WHERE epoch >= (SELECT MIN(query_start_epoch)
dbadmin(>                    FROM dc_requests_issued
dbadmin(>                   WHERE time >= (SYSDATE::TIMESTAMP - INTERVAL '1 MINUTE'));
 partition_key
---------------
             5
             6
(2 rows)

Jim_Knicely
Forum|alt.badge.img+2
  • Participating Frequently
  • February 11, 2019

@Vertica_Curtis - Fyi, I sent an email to Vanilla support on the posting SQL issue!


Jim_Knicely
Forum|alt.badge.img+2
  • Participating Frequently
  • February 12, 2019

@Vertica_Curtis - The issue that caused your SQL post to fail has been resolved. Can you please post the query you mentioned for the benefit of the Community? :)