Skip to main content
Question

How to purge RejectionTableData ?

  • October 12, 2018
  • 2 replies
  • 16 views

chaima
Forum|alt.badge.img+2
  • Participating Frequently

Hi team,

Is there a way to automatically purge the RejectionTableData directory? Or to purge an external table?
Customer has a Kafka scheduler running as part of the extended monitoring feature and it is generating a lot of rejected data files stored in external tables (dcschema.rej_dc_*)
Is it safe to just empty the directory?

Thanks,
Chaima

2 replies

mosheg
Forum|alt.badge.img+2
  • Participating Frequently
  • October 13, 2018
Hello Chaima,

Although files are deleted , the external table definitions remain in Vertica catalog.
Therefor any session which will refer to those deleted files will produce errors and warnings as shown below.

$ cat check_ext_tbl.sql
\echo "________________________________________________________TESTING.."
DROP TABLE IF EXISTS e1 CASCADE;
\! printf "1,AA\n2,BB\n3,CC\n" > e1.csv
\set FILEPATH `pwd`/e1.csv
\set FILEPATH '''':FILEPATH''''
CREATE EXTERNAL TABLE e1 (f1 int, f2 varchar(10)) AS COPY FROM :FILEPATH DELIMITER ',' ABORT ON ERROR;
SELECT * FROM e1;
\! rm -f e1.csv
SELECT * FROM e1;
SELECT table_name, source_format, total_file_size_bytes FROM external_table_details;
SELECT ANALYZE_EXTERNAL_ROW_COUNT('');
SELECT table_definition FROM TABLES WHERE TABLE_NAME = 'e1';
SELECT event_timestamp, message FROM error_messages WHERE event_timestamp BETWEEN NOW()-INTERVAL '1 minute' AND NOW();

$ vsql -ef check_ext_tbl.sql
"________________________________________________________TESTING.."
DROP TABLE IF EXISTS e1 CASCADE;
DROP TABLE
CREATE EXTERNAL TABLE e1 (f1 int, f2 varchar(10)) AS COPY FROM '/home/dbadmin/EXT_TABLES/e1.csv' DELIMITER ',' ABORT ON ERROR;
CREATE TABLE
SELECT * FROM e1;
f1 | f2
----+----
1 | AA
2 | BB
3 | CC
(3 rows)

SELECT * FROM e1;
vsql:check_ext_tbl.sql:9: ERROR 2886: Could not open file [/home/dbadmin/EXT_TABLES/e1.csv] for reading; No such file or directory
HINT: Make sure the file is available on the specified node. If using ON ANY NODE, it must be available at this path on all nodes in the cluster since any node could load the file
SELECT table_name, source_format, total_file_size_bytes FROM external_table_details;
table_name | source_format | total_file_size_bytes
-------------+---------------+-----------------------
e1 | DELIMITED |
ext_custDim | DELIMITED | 44
(2 rows)

SELECT ANALYZE_EXTERNAL_ROW_COUNT('');
vsql:check_ext_tbl.sql:11: WARNING 3314: Exception while running analyze_statistics: Could not open file [/home/dbadmin/EXT_TABLES/e1.csv] for reading; No such file or directory
ANALYZE_EXTERNAL_ROW_COUNT
----------------------------
0
(1 row)

SELECT table_definition FROM TABLES WHERE TABLE_NAME = 'e1';
table_definition
--------------------------------------------------------------------------
COPY FROM '/home/dbadmin/EXT_TABLES/e1.csv' DELIMITER ',' ABORT ON ERROR
(1 row)

SELECT event_timestamp, message FROM error_messages WHERE event_timestamp BETWEEN NOW()-INTERVAL '1 minute' AND NOW();
event_timestamp | message
-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------
2018-10-13 07:55:28.090035-04 | Could not open file [/home/dbadmin/EXT_TABLES/e1.csv] for reading; No such file or directory
2018-10-13 07:55:28.092889-04 | Could not open file [/home/dbadmin/EXT_TABLES/e1.csv] for reading; No such file or directory
2018-10-13 07:55:28.112713-04 | Could not open file [/home/dbadmin/EXT_TABLES/e1.csv] for reading; No such file or directory
2018-10-13 07:55:28.114868-04 | Exception while running analyze_statistics: Could not open file [/home/dbadmin/EXT_TABLES/e1.csv] for reading; No such file or directory
(4 rows)

Kind Regards,
Moshe

chaima
Forum|alt.badge.img+2
  • Author
  • Participating Frequently
  • October 22, 2018

Many thanks Moshe for your help!