Rocket U2 | UniVerse & UniData

Expand all | Collapse all

UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

  • 1.  UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

    Posted 02-18-2021 01:15
    Recently I had to get involved in a particularly challenging problem that started happening to a process that has been rock-solid for many years.

    Context: UV 11.2.5 on RHEL 7

    We have a process that performs numerous transaction-bracketed updates.
    It is capable of being run as a phantom.
    Under the right conditions the phantom process would just stop.
    We have a COMO recording active while the phantom process is running, but it captured nothing of any value.
    The last line in the COMO was always Message[001467]

    For those that are interested, this is an item in the SYS.MESSAGE file
    >CT SYS.MESSAGE 001467
         001467
    0001 Spooler Entry #%i
    0002
    >

    The program that is active at the time of the abort is performing BASIC WRITE to a file whilst an TRANSACTION is active.

    After using strace to monitor the phantom pid I came to understand that when the phantom would die it was after updating a cacheXXXXX file in the UVTEMP directory.
    These are dynamic files, and are used as disk overflow of the in-memory transaction cache.

    Digging around for information on these files lead me to the release notes for UV 11.1.4, which included this note:
    "UNV-5474 – Under a specific set of conditions, UniVerse might core dump. The conditions involved updating a file which contained a virtual field index, transactions were being used (i.e. this would be true if using triggers), the size of the transaction exceeded the in memory transaction cache buffer as defined by the uvconfig parameter TXMEM, and the virtual field index returns the null (i.e. empty string) as the result. The workaround at this release is to increase TXMEM so the transaction remains in memory. This issue has been corrected in the 11.1.5 release of UniVerse."

    The file being written to did have a new index added to it recently. It is a virtual field index (it uses the SUBR() function to call a subroutine, which in turn is locking and updating other records). The virtual field returns null, and the index is created with the NO.NULLS option.
    We had not changed the TXMEM setting of UniVerse, so it was still set to 32.

    So the issue UNV-5474 is still alive in UV 11.2.5.
    For us it was triggered in our code when we added a new index that calls a subroutine that added extra file io to the transaction cache.
    It seems that when UV 11.2.5 has to use the disk cache during the transaction it can trigger a corruption that aborts the process.

    The fix is to adjust the TXMEM setting. We have adjusted this up to 256 and the update process is no longer aborting.

    #txmem #UNV-5474 #UniVerse #11.2.5 #rhel

    ​​​​​​

    ------------------------------
    Gregor Scott
    gregors
    ------------------------------


  • 2.  RE: UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

    ROCKETEER
    Posted 02-18-2021 09:56
    Hi Gregor,
    While the characteristics seem quite similar to UNV-5474, it does not appear to be the exact same issue previously fixed. I ran the reproduction case for UNV-5474 at 11.2.5 on both AIX and Linux without failure. Also ran at 11.3.x without issue. One thing of note is that the situation with UNV-5474 did not involve a NO.NULLS index so the failure was related to updating an index file which accepted NULL entries. But for completeness, I altered the test case to create the index with the NO.NULLS keyword and the test also ran successfully to completion.
    What you describe does certainly sound like a problem that requires a transaction to overflow the TXMEM setting and to have virtual field indexes involved. Do you have a reproduction case which we could review at Rocket?
    Thank you.
    Neil

    ------------------------------
    Neil Morris
    UniVerse Advanced Technical Support
    Rocket Software
    ------------------------------



  • 3.  RE: UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

    Posted 02-18-2021 20:10
    Thanks for checking into this Neil.

    I can see the difference between the UNV-5474 scenario and ours.

    We could not boil the scenario down to running a single program on an idle system and have the issue trigger.
    We were able to get a VM snapshot of the server immediately prior to the production system attempting to do the background update process, and that particular dataset produced the problem.
    With the VM copied we tried removing all the other background processing from the server to reproduce the problem. This just resulted in the background process completing without issues. We had to have a server with all the other background processing actively running before our background update process would fail.
    This makes me think the effort and time impact of having to marshal group and record lock access played a part.
    We tried adjusting the logic in the subroutine called by the virtual field to remove it's record updating logic. This resulted in the background update process completing successfully. This indicated to me that it was the addition of the new index, one that adds additional record updates to the active transaction, was the catalyst for the issue.

    Does the TXMEM setting add to the memory of each UV process, or is it simply the limiter of how much of the PID's already allocated memory that will be used for the transaction cache?
    Is there any reason the TXMEM default setting is just 32k?

    ------------------------------
    Gregor Scott
    gregors
    ------------------------------



  • 4.  RE: UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

    ROCKETEER
    Posted 02-19-2021 16:30
    Hi Gregor,
    Thanks you for the feedback. I was concerned that the behavior you reported might not be reproducible and that appears to be the case. I believe the TXMEM setting is memory allocated to each uvsh process. Which may be why it has a default value of 32K. Let me do some investigation this weekend to see if I can provide a definitive answer. I think any application using transactions today will likely want to configure TXMEM larger but raising the default for applications not using transactions may not be appropriate. I will let you know.
    Thanks,
    Neil

    ------------------------------
    Neil Morris
    UniVerse Advanced Technical Support
    Rocket Software
    ------------------------------



  • 5.  RE: UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

    Posted 02-21-2021 21:54
    I look forward to hearing the results of your experiments, Neil.

    I think it would be super helpful to have the transaction handling sections of the BASIC manuals reference the TXMEM uvconfig setting as an influence in the performance of handling transactions. It might help those new to transactions in BASIC to understand there is a setting they could, and probably should, adjust.

    ------------------------------
    Gregor Scott
    Software Architect
    Pentana Solutions
    Mount Waverley VIC Australia
    ------------------------------



  • 6.  RE: UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

    ROCKETEER
    Posted 02-26-2021 08:06
    Hi Gregor,
    Just to let you know I haven't forgotten this one. I'm working with development to confirm the behavior of TXMEM. Once I have that I will let you know. I will also open up a documentation ticket to have TXMEM mentioned in those sections of the BASIC manual which discuss the handling of transactions.
    Thanks,
    Neil

    ------------------------------
    Neil Morris
    UniVerse Advanced Technical Support
    Rocket Software
    ------------------------------



  • 7.  RE: UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

    Posted 02-28-2021 21:01
    Thanks for pursuing it, Neil.

    I look forward to seeing the results.

    ------------------------------
    Gregor Scott
    Software Architect
    Pentana Solutions
    Mount Waverley VIC Australia
    ------------------------------



  • 8.  RE: UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

    ROCKETEER
    Posted 03-09-2021 09:56
    Hi Gregor,
    Sorry for the delay but I do now have additional details on how memory is used based on the TXMEM uvconfig parameter. The size specified by TXMEM defines the maximum amount of memory a uv process will use to hold transaction data before writing information to disk. The memory is allocated at run time while the transaction is processing. What that means is that size specified by TXMEM will only be used if required by the transaction. Smaller transactions will only use what is required by that transaction. And if a process does not perform any transactional updates at all, the TXMEM setting will have no impact. So when set to a larger value, the memory would only be allocated and used when needed.
    I will also get a documentation ticket open with a request to add a note about TXMEM in the BASIC Commands Reference guide and the BASIC Users guide in the sections were transaction semantics are discussed.
    Hope that answers your questions. If not, please advise.
    Thanks,
    Neil

    ------------------------------
    Neil Morris
    UniVerse Advanced Technical Support
    Rocket Software
    ------------------------------



  • 9.  RE: UNV-5474 is still a problem on UV 11.2.5 - Increasing TXMEM is the solution

    Posted 03-09-2021 18:16
    Thanks for the clarification, Neil.

    It is good to know that setting the TXMEM to a value higher than the default of 32 will not cause each UV process to automatically use more memory.

    This certainly does answer my questions around the TXMEM setting.

    ------------------------------
    Gregor Scott
    Software Architect
    Pentana Solutions
    Mount Waverley VIC Australia
    ------------------------------