Rocket U2 | UniVerse & UniData

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

Could you clarify this a bit?  What do you mean by "Master Data" vs "item files". In Universe you can have one copy of the file with Q pointers in all 8 accounts pointing to that one copy. If there's data specific to the account, each of your 8 accounts can have it's own copy of that file.

Having 8 files with one index sounds like you're going to have a lot of mismatches. I must not be understanding the question.

------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US

Original Message:
Sent: 09-26-2024 15:47
From: Thomas Ludwig
Subject: one index for multiple files?

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

Thanks JOE for thinking about it.

There ar 8 physical Data-Files with the same dict in 8 Accounts.

about 60% of the data are identical in all 8 data-files.

Only Fields are in the index that are identical in all 8 Files

best regards

Thomas

Could you clarify this a bit?  What do you mean by "Master Data" vs "item files". In Universe you can have one copy of the file with Q pointers in all 8 accounts pointing to that one copy. If there's data specific to the account, each of your 8 accounts can have it's own copy of that file.

Having 8 files with one index sounds like you're going to have a lot of mismatches. I must not be understanding the question.

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

Thanks JOE for thinking about it.

There ar 8 physical Data-Files with the same dict in 8 Accounts.

about 60% of the data are identical in all 8 data-files.
The keys and number of Entries in all 8 data-files are also identical.

Only Fields are in the index that are identical in all 8 Files

best regards

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account

Original Message:
Sent: 09-26-2024 15:54
From: Joe Goldthwaite
Subject: one index for multiple files?

Could you clarify this a bit?  What do you mean by "Master Data" vs "item files". In Universe you can have one copy of the file with Q pointers in all 8 accounts pointing to that one copy. If there's data specific to the account, each of your 8 accounts can have it's own copy of that file.

Having 8 files with one index sounds like you're going to have a lot of mismatches. I must not be understanding the question.

------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US

Original Message:
Sent: 09-26-2024 15:47
From: Thomas Ludwig
Subject: one index for multiple files?

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

Thomas,

I would worry about how you are handling the writes to he 8 files, if/wen locks are involved.  What would happen if one of the files did not get updated?  I can see a situation where the index would return an item based on data that would not be in the version of the file that failed to write.

As for the performance, I think I see why it is faster.

While I am not 100% sure, I expect that the index would only be updated by the first file that was updated.  So, 9 writes for each update, 1 write for each of the 8 files and one index write, is approximately half the writes (16) if each file had its own index. 

I understand your concern with changing code that works and the effort may not be worth the gain, but maintaining one file for the 60% that is identical, and a secondary file that uses the same id for the unique data in each of the 8 accounts may provide better performance.  The bulk import of the common data would be updating 2 files not 9 or 16.  

------------------------------
Mike Rajkowski
MultiValue Product Evangelist
Rocket Internal - All Brands
US
------------------------------

Original Message:
Sent: 09-26-2024 15:58
From: Thomas Ludwig
Subject: one index for multiple files?

Thanks JOE for thinking about it.

There ar 8 physical Data-Files with the same dict in 8 Accounts.

about 60% of the data are identical in all 8 data-files.
The keys and number of Entries in all 8 data-files are also identical.

Only Fields are in the index that are identical in all 8 Files

best regards

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account

Original Message:
Sent: 09-26-2024 15:54
From: Joe Goldthwaite
Subject: one index for multiple files?

Could you clarify this a bit?  What do you mean by "Master Data" vs "item files". In Universe you can have one copy of the file with Q pointers in all 8 accounts pointing to that one copy. If there's data specific to the account, each of your 8 accounts can have it's own copy of that file.

Having 8 files with one index sounds like you're going to have a lot of mismatches. I must not be understanding the question.

------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US

Original Message:
Sent: 09-26-2024 15:47
From: Thomas Ludwig
Subject: one index for multiple files?

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

Thomas,

I would worry about how you are handling the writes to he 8 files, if/wen locks are involved.  What would happen if one of the files did not get updated?  I can see a situation where the index would return an item based on data that would not be in the version of the file that failed to write.

As for the performance, I think I see why it is faster.

While I am not 100% sure, I expect that the index would only be updated by the first file that was updated.  So, 9 writes for each update, 1 write for each of the 8 files and one index write, is approximately half the writes (16) if each file had its own index. 

I understand your concern with changing code that works and the effort may not be worth the gain, but maintaining one file for the 60% that is identical, and a secondary file that uses the same id for the unique data in each of the 8 accounts may provide better performance.  The bulk import of the common data would be updating 2 files not 9 or 16.  

------------------------------
Mike Rajkowski
MultiValue Product Evangelist
Rocket Internal - All Brands
US

Original Message:
Sent: 09-26-2024 15:58
From: Thomas Ludwig
Subject: one index for multiple files?

Thanks JOE for thinking about it.

There ar 8 physical Data-Files with the same dict in 8 Accounts.

about 60% of the data are identical in all 8 data-files.
The keys and number of Entries in all 8 data-files are also identical.

Only Fields are in the index that are identical in all 8 Files

best regards

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account

Original Message:
Sent: 09-26-2024 15:54
From: Joe Goldthwaite
Subject: one index for multiple files?

Could you clarify this a bit?  What do you mean by "Master Data" vs "item files". In Universe you can have one copy of the file with Q pointers in all 8 accounts pointing to that one copy. If there's data specific to the account, each of your 8 accounts can have it's own copy of that file.

Having 8 files with one index sounds like you're going to have a lot of mismatches. I must not be understanding the question.

------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US

Original Message:
Sent: 09-26-2024 15:47
From: Thomas Ludwig
Subject: one index for multiple files?

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

</http:></https:></http:></mailto:sambeard@micahtek.com>

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

Hi Thomas,

Have you had a look at creating a distributed file ? Where the 8 files would be partfiles of a Distributed file.  Once you create the index on the header file you can use it on the header file (across all 8)  or on the individual part files.

Regards,

Greg

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

Hi Greg,

this is a very good poin. I have never heard or thought about distributed files. By reading the  UV-Documentation it turns out that it could be worth to investigagte further, although i think the algorythm could be a bit tricky because i cannot split on a complete record. Instead i have to split the record itself based on the field-positions.

but i would defnetly investigate this and try to create one in a test-envornment.

best regards

Hi Thomas,

Have you had a look at creating a distributed file ? Where the 8 files would be partfiles of a Distributed file.  Once you create the index on the header file you can use it on the header file (across all 8)  or on the individual part files.

Regards,

Greg

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

Hi Greg,

this is a very good poin. I have never heard or thought about distributed files. By reading the  UV-Documentation it turns out that it could be worth to investigagte further, although i think the algorythm could be a bit tricky because i cannot split on a complete record. Instead i have to split the record itself based on the field-positions.

but i would defnetly investigate this and try to create one in a test-envornment.

best regards

Hi Thomas,

Have you had a look at creating a distributed file ? Where the 8 files would be partfiles of a Distributed file.  Once you create the index on the header file you can use it on the header file (across all 8)  or on the individual part files.

Regards,

Greg

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

If the accounts are on the same machine, a pointer from each account to one item file with one index might work. 

If speed is all you want, file resizing could give you benefits as well. Optimized file parameters can change the world.

------------------------------
Mike Bojaczko
PROGRAMMER ANALYST
US

Original Message:
Sent: 09-26-2024 15:47
From: Thomas Ludwig
Subject: one index for multiple files?

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

Thank you Mike, 

speed was not the initial-reason for investigating this the intial reason was that there was one index for two files for a while because of an error by an account-duplication  and it worked without problems.

The item's-File currently has about 1.700.000 Items and works well with file-type 30.

But my thought was "if it is possible to speed up without to much effort -> why not."

I think i keep the system like it is at the moment but learned a lot because of this thread especially about distributed files.

best regards

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account

Original Message:
Sent: 09-27-2024 06:09
From: Mike Bojaczko
Subject: one index for multiple files?

If the accounts are on the same machine, a pointer from each account to one item file with one index might work. 

If speed is all you want, file resizing could give you benefits as well. Optimized file parameters can change the world.

------------------------------
Mike Bojaczko
PROGRAMMER ANALYST
US

Original Message:
Sent: 09-26-2024 15:47
From: Thomas Ludwig
Subject: one index for multiple files?

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

------------------------------
Mike Bojaczko
PROGRAMMER ANALYST
US

Original Message:
Sent: 09-27-2024 08:40
From: Thomas Ludwig
Subject: one index for multiple files?

Thank you Mike, 

speed was not the initial-reason for investigating this the intial reason was that there was one index for two files for a while because of an error by an account-duplication  and it worked without problems.

The item's-File currently has about 1.700.000 Items and works well with file-type 30.

But my thought was "if it is possible to speed up without to much effort -> why not."

I think i keep the system like it is at the moment but learned a lot because of this thread especially about distributed files.

best regards

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account

Original Message:
Sent: 09-27-2024 06:09
From: Mike Bojaczko
Subject: one index for multiple files?

If the accounts are on the same machine, a pointer from each account to one item file with one index might work. 

If speed is all you want, file resizing could give you benefits as well. Optimized file parameters can change the world.

------------------------------
Mike Bojaczko
PROGRAMMER ANALYST
US

Original Message:
Sent: 09-26-2024 15:47
From: Thomas Ludwig
Subject: one index for multiple files?

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

Sorry Mark, I accidentally replied private...

I wonder of the @WHO could be used in the DF algorithm or one of the attributes returned from the Basic STATUS statement like the I-node or the Absolute pathname.

------------------------------
Mike Bojaczko
PROGRAMMER ANALYST
US

Original Message:
Sent: 09-26-2024 15:47
From: Thomas Ludwig
Subject: one index for multiple files?

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------

Sorry Mark, I accidentally replied private...

I wonder of the @WHO could be used in the DF algorithm or one of the attributes returned from the Basic STATUS statement like the I-node or the Absolute pathname.

------------------------------
Mike Bojaczko
PROGRAMMER ANALYST
US

Original Message:
Sent: 09-26-2024 15:47
From: Thomas Ludwig
Subject: one index for multiple files?

Our Universe application consists of 8 accounts for 8 branches. In all accounts, there is a file for the items. The item file is identical for the master data in all accounts, only the transaction data differs. When writing, changes to the file, master data are distributed across all accounts to keep them consistent.

Although this approach is certainly not optimal, we do not want to change it because the programming is not easy to modify.

I conducted a test, and used: SET.INDEX ITEM TO H:\ALPHA\INDEXES\I_ITEM to set the target path for the index of all 8 item files to the same destination.

During a bulk import into the item file, where all 8 files are updated, switching to a single index file, approximately double the speed is achieved.

My question now is: Is there anything that speaks against using a single index file for multiple data files, as long as all indexed fields are identical for all data files?

greetings

Thomas

------------------------------
Thomas Ludwig
System Builder Developer
Rocket Forum Shared Account
------------------------------