Skip to main content

Concurrency: what happens when HDFS file changes or is removed during a query?

  • February 13, 2018
  • 2 replies
  • 7 views
Bryan_H
Forum|alt.badge.img+2

I have this question from a prospect:

  • Concurrency: what happens when HDFS file changes or is removed during a query?
    I would assume we try to snapshot state at the beginning of the query, but if they are writing to a partition on HDFS, or if someone deletes a file during a query, what happens? What works and what doesn't?

    2 replies

    Bryan_H
    Forum|alt.badge.img+2
    • Bryan_H
    • Author
    • Participating Frequently
    • February 14, 2018

    From email chain: "HDFS doesn't document a mechanism to reference count files, therefore, if a file that was included during plan is deleted before vertica has finished with it an error will be raised."

    Bryan_H
    Forum|alt.badge.img+2
    • Bryan_H
    • Author
    • Participating Frequently
    • February 14, 2018

    Also from email: "[During inserts or updates,] It is possible to see partial results. That is, Isolation isn't supported. Until Hive added "ACID support", which is only available for ORC, not Parquet, Isolation was only supported in Hive by turning on locking. Unlike locking in Vertica, inserts into a Hive table block reads."

    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement