Every morning, seemingly once per day our customer gets a memory access violation that causes Acuserver to shutdown

First example

removing client 'ITDRENMVP17143' user 'xxxxxx.xxxxxx' pid 13588 fid 374
Number of files decreased to 152
failed: 12
7311553 - Memory access violation - shutting down

Second example

Checking access for 'd:/data/southpac/prod/v8.6/retail/prod/LV405283.DAT': 0xf, 1
opening data segment (#0): d:/data/southpac/prod/v8.6/retail/prod/LV405283.DAT
>>Header accesses: 0 updates, 3 locked reads, 0 unlocked reads
>>0 unlocked reads waited for update, 0 unlocked reads retried
closing data segment (#0): d:/data/southpac/prod/v8.6/retail/prod/LV405283.DAT
removing data segment (#0): d:/data/southpac/prod/v8.6/retail/prod/LV405283.DAT
succeeded (1)
v_read: 140, 491, 0, 0x1 - invalid file ID
failed: 1, 9
fwrite: 140, 488, 1, 31 - invalid file ID
failed: 9
fflush: 140, 488 - invalid file ID
failed: 9
6584107 - Memory access violation - shutting down

Can anyone advise what the "failed: 12| and "failed:9" mean?

This is running Acuserver 6.1 on windows server 2012 (don't laugh)

I'm not sure this will help, but

"failed: 12" means the operation ran out of memory

"failed: 9" means there was a bad file handle given to a file operation

"failed: 1, 9" means System Error, bad file handle (returned from the vision system)

This is a very old system (I don't need to tell you that), and there is a possibility that any bug has been fixed already. Though figuring out which ECN fixed it would be impossible at this point.

But is it possible that the system is running out of memory? That might explain all the failures.

Thanks for that info.  Is there an online manual with that in it?

I like the idea of the memory , because it would sit well with the idea that something is running at the same time.  It occurs effectively once per day in the morning.

I shall re-iterate the need to check for background tasks

Thanks

Jim

We don't document trace information, sorry. That is essentially for internal use only.

I got the client to take a full backup of the vision files before the expected crash.  No-one was logged in and like clockwork Acuserver failed with memory violation at 7:31am.

The client took a backup after the Acuserver crash.  I compared the before and after data folders with Beyond Compare and not a single file had been update.  So it would appear that no Cobol Application processing was being run at the time of failure.

Do you know if Acuserver has any work to do if the are no requests to it for files?

How can it hit a memeory error or a memory access violation if it is not processing files?

The oldest source I have easy access to is for version 6.2. But 6.1 is probably not too different.

AcuServer actually runs in a 3-second loop when it is waiting for requests. So if it is not getting any requests to do something, every 3 seconds it will do a little cleanup and checking, and then go wait for requests again. That cleanup and checking code could have a bug or memory leak or who knows, causing the crash. Or perhaps between 6 and 7 AM, something happens to the machine that causes this code to fail somehow.

We do have trace statements in the code in that loop. You can force them to show by doing two things:

1. Set DEAD-CLIENT-TIMEOUT to a small value (2) in your configuration file.
2. Make sure that the trace level has bit 1 set (has a value of 2, 3, 6, 7, 10, 11, 14, or 15, for example)

After doing this, you should see in the trace every three seconds:

It would be interesting to see how soon after that line we get the crash. Is it immediately after? Or immediately before? Somewhere in the middle?

Note that this is a very old version of AcuServer, and without some sort of special support contract, we won't be able to solve this problem if it turns out to be a bug. You might be well served upgrading to a later version.

Thanks rzack

Is it possible that adding these settings could actually help prevent acuserver crashing?

The reason I ask is twice now when acuserver has been restarted with these config entries added the expected crash did not take place.

It happened the following day but by then the acuserver.log was too huge unmanageable.  Asked them to try again another day and again the crash did not materialize and this is after weeks and weeks of failures.

Changing settings could have an effect on the outcome. You are laying out memory differently by changing settings, which means that any memory error could move. I have seen it before. But it's unusual.

There is another setting you can try in order to limit the size of the log file. Set MAX-ERROR-LINES to the line limit you desire, and AcuServe will rewrite the log from the beginning after writing that many lines. So you get a wrapping log file, rather than a long linear one. Again, this may change behavior (that memory layout thing), but it will make any log file manageable in size.

Please find attached a video recording of process explorer session where Acuserver just dies...

It was suggested that I get the client to do a process explorer session when the acuserver crashes.

They provided a video of this but I don't know what I am looking for .  I can see acuserver running and then the line showing acuserver turns red and drops out.

Please confirm that I manage to upload the mp4