Skip to main content

Edit - can't reply for some reason. I am on UD 8 myself but they are on 7.3 with no upgrade coming anytime soon, so wholefile is out. File has gone all the way down to zero records several times and we're still at ~2M modulo. I have come to the conclusion that merging isn't a thing dynamic files do on this release, so am working this problem from another angle. These files don't have an optimal config, they've got a bunch of random stuff flowing through them.

Unidata 7.3.7 Windows

I need help understanding merging and dynamic files.  

I have a file that I think should be merging, but it never does. Disk space is not a concern, I just need to get the modulo down to something reasonable.

Normal state for this file is a low recordcount, but periodic updates push huge amounts of data to it temporarily. Converting it to static or changing it to a DIR-type file are both not options. (Yes, this situation isn't ideal. No, I don't have control over it.)

From Jonathan Smith's excellent articles, I know REBUILD.FILE, CLEAR.FILE, and RESIZE will trigger a merge. I need this to happen when users are on, though.

My theory is that whatever event triggers a merge only happens at times when there is also a condition that prevents the merge.  I'm not opposed to writing something to "encourage" the merge if this is true, I am just not sure what that thing would be. My alternate theory is, I completely misunderstand how merging works.

Things I know that prevent a file from merging which do not seem to be the case here:

  • Records in last group
  • Active SELECT against the file
  • Active COUNT operation on the file
  • Combined load factor above MERGE.LOAD
  • Having a min modulo higher than current modulo

What do I have to make happen to get this file to merge?

Here's the info for one of the more ridiculous files:

Number of groups in file (modulo)     = 2311239                                 
Minimum groups of file                = 3                                       
Hash type = 0, blocksize = 512                                                  
Split load = 60, Merge load = 40                                                
Split/Merge type = KEYONLY                                                      
                                                                                
   Group     Keys      Key Loads       Percent                                  
=================================================                               
    0         0               0              0                                  
    1         0               0              0                                  
    2         0               0              0                                  
    3         0               0              0                                  

(etc. - nothing in last group)

                      
Number of groups in file (modulo)     = 2311239                                 
Dynamic hashing, hash type            = 0                                       
Split/Merge type                      = KEYONLY                                 
Block size                            = 512                                     
File has 86 groups in level one overflow.                                       
Number of records                     = 126                                     
Total number of bytes                 = 38274                                   
                                                                                
Average number of records per group   = 0.0                                     
Standard deviation from average       = 0.0                                     
Average number of bytes per group     = 0.0                                     
Standard deviation from average       = 2.6                                     
                                                                                
Average number of bytes in a record   = 303.8                                   
Average number of bytes in record ID  = 30.0                                    
Standard deviation from average       = 191.5                                   
Minimum number of bytes in a record   = 110                                     
Maximum number of bytes in a record   = 853                                     
                                                                                
Minimum number of fields in a record  = 21                                      
Maximum number of fields in a record  = 21                                      
Average number of fields per record   = 21.0                                    

Standard deviation from average       = 0.0                                     
File has 1 over files, 2 prime files                                            

Thanks very much for any help.

Edit - can't reply for some reason. I am on UD 8 myself but they are on 7.3 with no upgrade coming anytime soon, so wholefile is out. File has gone all the way down to zero records several times and we're still at ~2M modulo. I have come to the conclusion that merging isn't a thing dynamic files do on this release, so am working this problem from another angle. These files don't have an optimal config, they've got a bunch of random stuff flowing through them.

Unidata 7.3.7 Windows

I need help understanding merging and dynamic files.  

I have a file that I think should be merging, but it never does. Disk space is not a concern, I just need to get the modulo down to something reasonable.

Normal state for this file is a low recordcount, but periodic updates push huge amounts of data to it temporarily. Converting it to static or changing it to a DIR-type file are both not options. (Yes, this situation isn't ideal. No, I don't have control over it.)

From Jonathan Smith's excellent articles, I know REBUILD.FILE, CLEAR.FILE, and RESIZE will trigger a merge. I need this to happen when users are on, though.

My theory is that whatever event triggers a merge only happens at times when there is also a condition that prevents the merge.  I'm not opposed to writing something to "encourage" the merge if this is true, I am just not sure what that thing would be. My alternate theory is, I completely misunderstand how merging works.

Things I know that prevent a file from merging which do not seem to be the case here:

  • Records in last group
  • Active SELECT against the file
  • Active COUNT operation on the file
  • Combined load factor above MERGE.LOAD
  • Having a min modulo higher than current modulo

What do I have to make happen to get this file to merge?

Here's the info for one of the more ridiculous files:

Number of groups in file (modulo)     = 2311239                                 
Minimum groups of file                = 3                                       
Hash type = 0, blocksize = 512                                                  
Split load = 60, Merge load = 40                                                
Split/Merge type = KEYONLY                                                      
                                                                                
   Group     Keys      Key Loads       Percent                                  
=================================================                               
    0         0               0              0                                  
    1         0               0              0                                  
    2         0               0              0                                  
    3         0               0              0                                  

(etc. - nothing in last group)

                      
Number of groups in file (modulo)     = 2311239                                 
Dynamic hashing, hash type            = 0                                       
Split/Merge type                      = KEYONLY                                 
Block size                            = 512                                     
File has 86 groups in level one overflow.                                       
Number of records                     = 126                                     
Total number of bytes                 = 38274                                   
                                                                                
Average number of records per group   = 0.0                                     
Standard deviation from average       = 0.0                                     
Average number of bytes per group     = 0.0                                     
Standard deviation from average       = 2.6                                     
                                                                                
Average number of bytes in a record   = 303.8                                   
Average number of bytes in record ID  = 30.0                                    
Standard deviation from average       = 191.5                                   
Minimum number of bytes in a record   = 110                                     
Maximum number of bytes in a record   = 853                                     
                                                                                
Minimum number of fields in a record  = 21                                      
Maximum number of fields in a record  = 21                                      
Average number of fields per record   = 21.0                                    

Standard deviation from average       = 0.0                                     
File has 1 over files, 2 prime files                                            

Thanks very much for any help.

Elaine,

I'm eagerly awaiting an answer to your question,  although I'm not a UniData user. The forum had a lot of talk about  Dynamic files recently. So, I've been reading the UniData documentation in the interest of separation and modulo calculations and overall writing style as compared to Universe documentation.

It looks like UniData has a lot of options when it comes to Dynamic files. So, all I can really say is the blocksize of 512 looks way to small. I would try 4096 for the block size and experimenting with a minimum modulo to avoid hitting any massive overflow conditions. As I'm sure you know, changing anything would require some sort of resize. Hopefully, finding the optimal config for the file will keep it fast, out of overflow, and splitting and merging properly.

Below is documentation that I find pretty interesting.

The UniData_CommandsRefGuide_V737.pdf on pages 1-119 through 1-121 look like a good place to start under normal circumstances.

The UniData_WindowsAdminGuide_V737.pdf on page 11-12, Dynamic Files and Disk Space, states:
 "However, if you remove all records from a dynamic file by using a select list, the file might not return to its minimum modulo. Depending on the order in which UniData removes records, some groups resulting from earlier splits might not become eligible for merging, even though they do not contain any records."

The UniData_WindowsAdminGuide_V737.pdf on page page A-8 has info on the GRP_FREE_BLK config setting:
 "Pertains to dynamic files only. The number of free blocks that are kept in the free block list at the group level. If more blocks are freed, they are kept at the file level."

Good Luck on your UniData journey,

Mike 

Edit - can't reply for some reason. I am on UD 8 myself but they are on 7.3 with no upgrade coming anytime soon, so wholefile is out. File has gone all the way down to zero records several times and we're still at ~2M modulo. I have come to the conclusion that merging isn't a thing dynamic files do on this release, so am working this problem from another angle. These files don't have an optimal config, they've got a bunch of random stuff flowing through them.

Unidata 7.3.7 Windows

I need help understanding merging and dynamic files.  

I have a file that I think should be merging, but it never does. Disk space is not a concern, I just need to get the modulo down to something reasonable.

Normal state for this file is a low recordcount, but periodic updates push huge amounts of data to it temporarily. Converting it to static or changing it to a DIR-type file are both not options. (Yes, this situation isn't ideal. No, I don't have control over it.)

From Jonathan Smith's excellent articles, I know REBUILD.FILE, CLEAR.FILE, and RESIZE will trigger a merge. I need this to happen when users are on, though.

My theory is that whatever event triggers a merge only happens at times when there is also a condition that prevents the merge.  I'm not opposed to writing something to "encourage" the merge if this is true, I am just not sure what that thing would be. My alternate theory is, I completely misunderstand how merging works.

Things I know that prevent a file from merging which do not seem to be the case here:

  • Records in last group
  • Active SELECT against the file
  • Active COUNT operation on the file
  • Combined load factor above MERGE.LOAD
  • Having a min modulo higher than current modulo

What do I have to make happen to get this file to merge?

Here's the info for one of the more ridiculous files:

Number of groups in file (modulo)     = 2311239                                 
Minimum groups of file                = 3                                       
Hash type = 0, blocksize = 512                                                  
Split load = 60, Merge load = 40                                                
Split/Merge type = KEYONLY                                                      
                                                                                
   Group     Keys      Key Loads       Percent                                  
=================================================                               
    0         0               0              0                                  
    1         0               0              0                                  
    2         0               0              0                                  
    3         0               0              0                                  

(etc. - nothing in last group)

                      
Number of groups in file (modulo)     = 2311239                                 
Dynamic hashing, hash type            = 0                                       
Split/Merge type                      = KEYONLY                                 
Block size                            = 512                                     
File has 86 groups in level one overflow.                                       
Number of records                     = 126                                     
Total number of bytes                 = 38274                                   
                                                                                
Average number of records per group   = 0.0                                     
Standard deviation from average       = 0.0                                     
Average number of bytes per group     = 0.0                                     
Standard deviation from average       = 2.6                                     
                                                                                
Average number of bytes in a record   = 303.8                                   
Average number of bytes in record ID  = 30.0                                    
Standard deviation from average       = 191.5                                   
Minimum number of bytes in a record   = 110                                     
Maximum number of bytes in a record   = 853                                     
                                                                                
Minimum number of fields in a record  = 21                                      
Maximum number of fields in a record  = 21                                      
Average number of fields per record   = 21.0                                    

Standard deviation from average       = 0.0                                     
File has 1 over files, 2 prime files                                            

Thanks very much for any help.

Elaine,

As you observe, just because a file is below the merge threshold, does not mean it will merge.  

I also notice that you are using KEYONLY, which could cause a lot of level one overflow, more so with the default 60/40 split/merge load settings.

Key Only will only split a group when the keys take up more the Split value, 60% by default with the 512 Block size  

So, in your case the file will not split until more than ~300 bytes are filled with IDs, 

Based on Average id, that would be 10 items.  Which would have ~3,000 bytes of data and be in level one overflow.

I suggest that you upgrade to the latest version of UniData, since it has the WHOLEFILE type for splitting/merging.

Note that the WHOLEFILE was first available in UniData 8.1.0, yet that release, like the version you are on is EOLS

I suggest you upgrade to the latest release:

Release Build OS
Release		 PLC
 Date
8.2.4.3003  64 
 		 Windows 11
Windows 10 (Creators Update Supported)
2022
2019
2016
2012 (R2)
 		 GA 29 Mar 2023 GA 
17 Aug 2025 EOM 
17 Aug 2026 EOS 
17 Aug 2028 EOLS 

 

Terms & ConditionsAccessibility statement