Skip to main content

How do broken projection refreshes handle disk space?

  • July 4, 2017
  • 3 replies
  • 12 views
DieterC
Forum|alt.badge.img+2

Hi forum,

a Vertica customer reports a disk full situation after a series of broken refresh statements.

From the customer email:
"Our actual base disk usage is 25% before we started the projection. We suspect that the refresh is not cleaning up the data from the previously failed refresh, so over time and due to numerous automatic restarts to the projection refresh jobs, the disk is consumed in its entirety. Is this a bug?
We also could not find any recommendation about ensuring that we do not run refresh jobs from scratch while loading data."

Can you please help us with these topics

  • If projection refreshes fail: will the disk space filled so far be given back to the OS?
    I am not clear how a stopped refresh will handle intermediate results and if disk capacity will be returned.

  • How to proceed in an unclear situation after several failed refreshes: Will new refreshes continue the work started or are the intermediate results lost? How can we get clear on the situation present on the system?

Thank you
Dieter

    3 replies

    MAW
    Forum|alt.badge.img+1
    • MAW
    • Participating Frequently
    • July 4, 2017

    Hello Dieter,

    Just to clarify, the customer in question killed the CREATE PROJECTION after it had been running for ~1 day and they noticed their free disk space was <4%.

    On reviewing their vertica.log, I could also see they were loading (COPY) data from S3 into their base table at least 3 times per minute, with 10's of gzipped / JSON files in each COPY. That was thousands of COPY statements over the period the CREATE PROJECTION was running! No wonder they ran out of disk space!

    Though your questions are still valid in understanding whether space is returned to the OS in the event of a "failure" and whether it will continue from where it left off (which I am guessing it does not)

    Regards
    Mark

    TomM
    Forum|alt.badge.img
    • TomM
    • Participating Frequently
    • July 5, 2017

    Hey Dieter,

    My understanding is an operation that cannot successfully complete will be rolled back. However, it sounds like you're in some uncharted waters here, and perhaps this is not what's happening. Perhaps you've already reviewed this, and would the PROJECTION_REFRESHES system table help you determine if a particular REFRESH was successful? I note in the doc. it mentions a function that may help, and it's called CLEAR_PROJECTION_REFRESHES

    https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/SQLReferenceManual/SystemTables/MONITOR/PROJECTION_REFRESHES.htm?TocPath=SQL%20Reference%20Manual|Vertica%20System%20Tables|V_MONITOR%20Schema|_____48

    DieterC
    Forum|alt.badge.img+2
    • DieterC
    • Author
    • Participating Frequently
    • July 6, 2017

    Hi Tom,
    your comment is exactly to the point:
    is REFRESH a transactional operation or not?

    The "rollback" would be very simple in this case: just delete the created files.
    Giving our transactional model an approach could also be: keep the intermediate results and continue from there once REFRESH is restarted.
    Unfortunately I have no access to the customer system, I am supporting a colleague per email.
    Thanks
    Dieter

    Still scrolling?
    Ask a question
    Terms & ConditionsAccessibility statement