Hi Alex,

Which product and version are you using and under what OS?

If you rebuild to a new file do the duplicated records remain in the new file?
Are you using Fileshare or is this a file which resides locally or on a network drive?

Just grasping here but If it happens every 3 months could it be that some quarterly report is being run that perhaps isn't run at other times?

Chris,

I am using MF Visual COBOL 5.0 with Windows 10 and Windows Server 2012. The file resides on a network drive that all users access. When this problem occurred back in March, I rewrote all the records, excluding duplicates, to a new file because I thought the old file was corrupted. I then ran the rebuild on the new file to make sure that the duplicates were not copied over given that the reubuild give me a size number of the file. The rebuild is a bat file with the command: @"C:\\Program Files (x86)\\Micro Focus\\Visual COBOL\\bin64\\REBUILD.EXE" L:\\tps\\PROD\\FILES\\TPSFGNCK.DAT /i. .

I, too, thought that it may have to due with quarterly reports but we do not run quarterly reports and even if we did, then problem should have occurred at the end of June. There is nothing we are doing different than what we have been doing for the past 10 years. I am going through all my programs that write to the file again to see if I miss something but I am at a complete loss as to why this problem began occurring this year. I am unable to replicate this problem in our test system which makes it near impossible to identity the root cause. 

A few questions:

Do you still have the file with duplicated records in it?

Can you run rebuild directly on the file containing the duplicate records and rebuild it to a new file? I would like to know whether or not the new file still contains the duplicate records or if rebuild has deleted them.

In your select statement, do you allow for duplicates on your keys including the primary record key? If not, then it makes no sense that this could happen programatically.

Are you using file compression?
What are the compiler directives and extfh.cfg options that you use for this file?

Just so you are aware, VC 5.0 is very old and is no longer officially supported. 

what is the reason to use idxformat-8? is the file so big? do you use also datacompress?

what is when you use idxformat-4? here you have data and index in separate files.

how big is your file?

can you publish the select code and the file description?

Netexpress 5.1 is the latest netepxress version and now we must use visual cobol, the latest version is 9.0

Chris,

Yes, I still have the file with the duplicate records. Here is what I did:

I ran the rebuild utility on the file which was 201,541 KB. After the rebuild, the size decreases slightly to 201,366 KB. I then took all the records from the rebuilt file and wrote them to a new file. I then checked to see if the new file had duplicates and they in fact had duplicates. The rebuild did not delete duplicate records. Our program allows for duplicate records on this file (it is the only file in our database for which our programs allow duplicates). We are not using any DATACOMPRESS Compiler directives or any other kind of file compressor.  Here is what the extfh.cfg looks like:

[L:\\\\tps\\\\PROD\\\\FILES\\\\XFH-DEFAULT]

IDXFORMAT=3

[L:\\\\tps\\\\PROD\\\\FILES\\\\tpselfax.dat]

IDXFORMAT=8

FILEMAXSIZE=8

[L:\\\\tps\\\\PROD\\\\FILES\\\\receive.dat]

IDXFORMAT=8

FILEMAXSIZE=8

All of our files with the exception of two, both of which are not the file in question, are set to format 3. I just realized that I stated in the question that the file in question (TPSFGNCK.DAT) is an IDXFORMAT-8. TPSFGNCK.DAT , which is the file that is giving me problems was originally created in NetEpxress which means that it was an IDXFORMAT-3. Back in March of this year, however, when this problem first occurred, I rewrote the records from TPSFGNCK.DAT to a new data file in Visual Studio which means that the new TPSFGNCK.DAT file is now an IDXFORMAT-8. However, I did not give the TPSFGNCK.DAT file the directive of IDFFORMAT=8 and FILEMAXSIZE=8 in the extfh.cfg. But this problem first occurred when TPSFGNCK.DAT was an IDXFORMAT-3.

We will be upgrading to VC 9 in a month or so.

Greiner,

We use Visual COBOL 5 which automatically creates data files as IDXFORMAT-8. The file that is giving me problem is 103,458 KB. There programs that write to this file have the roughly the same code which as follows:

WRITE-FGNCK-FILE-ROUTINE.

MOVE F-PRIME TO FILE-KEY. 
MOVE F-WRITE TO FILE-ACTION.
CALL TPSIO026 USING FILE-REQUEST TPS-FGNCK-REC. 
IF IT-WAS-A-DUPLICATE-KEY 
      IF FGN-CHECK-RECON-PENDING 
           GO TO WRITE-FGN-CHECK-FILE-EXIT 
       END-IF 
 END-IF. 



IF IT-WAS-A-DUPLICATE-KEY
      if FGN-CHECK-NUMBER = zeros
         if FGN-CHECK-AMOUNT = zeros
             if FGN-CHECK-DATE = zeros
                move f-prime to file-key
                 move f-rewrite to file-action
                 call tpsio026 using file-request tps-fgnck-rec
                 if not a-successful-operation
                     move ' fgncke ' to file-name
                     move 'tps2010-wr2' to file-text
                     perform file-error thru file-error-exit
                     go to tps2010-common-exit
                  end-if
             END-IF
          END-IF
        END-IF
 END-IF.

 IF IT-WAS-A-DUPLICATE-KEY 
      IF FGN-CHECK-NUMBER = ZEROS

          ADD 1 TO FGN-CHECK-TIE-BREAKER 
          GO TO WRITE-FGN-CHECK-FILE 
       END-IF 
 END-IF. 

if it-was-a-duplicate-key
    if tps3020-calling
        add 1 to fgn-check-tie-breaker
         go to write-fgn-check-file
     end-if

end-if.

WRITE-FGNCK-FILE-ROUTINE-EXIT.

I am not sure about this entry in your extfh.cfg file:
L:\\\\tps\\\\PROD\\\\FILES\\\\XFH-DEFAULT]
Do you have an actual file with this name?

If this is to set default behavior then the tag should simply be:

[XFH-DEFAULT]

Since the file isnt really getting corrupted and you can still access all the records just fine I would think that these records are actually being written to the file using program logic.

You could set up a CTF trace to trace the RTS and file handler components but since this happens so infrequently it might be hard to catch and the tracing might be a performance hit. 

I would recommend that you wait until you upgrade to 9.0 before trying to troubleshoot this problem as at least then you will be on a supported version and we can investigate the issue further.

Chris,

I was informed that the extfh.cfg should be in the directory where the applications reside. The application resides in L:\\Mifocus3SP2,  but the data files reside in L:\\\\tps\\\\PROD\\\\FILES. I was therefore told by Fano from Micro Focus that I should therefore include the path of the data files in the extfh.cfg since the extfh.cfg resides in a different directory than the data files. If this is not correct, then please let me know.

I will continue parsing through the program logic to see if anything stands out. Hopefully this will not happen again for a couple of months. We will, hopefully, be running the newest version of VC by September. 

The path name should be specifed on the individual filename entries such as:

[L:\\\\tps\\\\PROD\\\\FILES\\\\tpselfax.dat]
IDXFORMAT"8"

but not on the [XFH-DEFAULT] tag.
The entries under [XFH-DEFAULT] apply to all files regardless of their location on disk.

[XFH-DEFAULT]
IDXFORMAT"8"

If you wish to use a common set of configuration options for all the files in a specific directory you can use the [FOLDER] tag:

[FOLDER]:L:\\\\tps\\\\PROD\\\\FILES
IDXFORMAT"8"

Please see the product documentation here:

Chris,

Hopefully, I a understanding you correctly. I modified the file to look like this:

[XFH-DEFAULT]
IDXFORMAT=3
[L:\\\\tps\\\\PROD\\\\FILES\\\\tpselfax.dat]
IDXFORMAT=8
FILEMAXSIZE=8
[L:\\\\tps\\\\PROD\\\\FILES\\\\receive.dat]
IDXFORMAT=8
FILEMAXSIZE=8

All the data files, with the exception of tpselfax.dat and receive.dat, are to BE handled as IDXFORMAT=3. If this still is no correct then please let me know. 

Yes, that looks correct now.