diseli said Why not use the TXT-driver? I did a quick test and loading a 1GB profiling log-file does not seem to be problem. I can also loop through the 3.8+ million lines. It probably is also a good idea to use discard (after analyzing a profiling line) in order to safe/release memory (when not needed anymore). Hope this helps. Regards, Daniel  

Hi Daniel How could I open an arbitrary text file? Ingo

istiller said

diseli said Why not use the TXT-driver? I did a quick test and loading a 1GB profiling log-file does not seem to be problem. I can also loop through the 3.8+ million lines. It probably is also a good idea to use discard (after analyzing a profiling line) in order to safe/release memory (when not needed anymore). Hope this helps. Regards, Daniel  

Hi Daniel How could I open an arbitrary text file? Ingo  

Hi Ingo, I was asked already once to solve this kind of issue and just rediscovered it this morning in my archives! We solved this way: 1) Front-end let user choose its file 2) File chosen is verified solving variances like CR vs LF vs CRLF as line terminator 3) Verified file is copied where ASN file address it 4) Application is loading the copied file 5) Copied file is renamed/deleted after loading This initial implementation was single user only and file size was limited. In a later stage we added more functionalities: 2bis) Splitting verified file in chunks when too large 3) Verified file is copied where ASN file address a set of them to enable multiuser usage 4bis) Checking if a further chunk should be loaded Hope it helps. Gianni

Hi Gianni, nice solution to still have a choice of the file but with the comfort of TXT. IIRC, once there was a problem that one couldn't close "$TXT" the workaround was to implement the universal close without parameter close this way, the TXT input file was no longer locked and you could process the next file

Hi Uli, Oh YES...I remember that issue! I remember also another tricky situation, again about file locking: when a TXT file was generated embedding record types in it (1=document header, 2=document details), an application should open same TXT file more than once, while using functional subtypes to read embedded tables. ASN file was pointing two model tables to the same physical TXT file: the second open returned an error! I remember a /nolock to be added to assignment file to avoid getting error but as of today I do not see it in current documentation. We are differently young in these scenarios... :-) :-) :-) Gianni

gianni said Hi Ingo, I was asked already once to solve this kind of issue and just rediscovered it this morning in my archives! We solved this way: 1) Front-end let user choose its file 2) File chosen is verified solving variances like CR vs LF vs CRLF as line terminator 3) Verified file is copied where ASN file address it 4) Application is loading the copied file 5) Copied file is renamed/deleted after loading This initial implementation was single user only and file size was limited. In a later stage we added more functionalities: 2bis) Splitting verified file in chunks when too large 3) Verified file is copied where ASN file address a set of them to enable multiuser usage 4bis) Checking if a further chunk should be loaded Hope it helps. Gianni  

Hi Gianni That's the workaround I also thought about :-) @UnifAce: Let there be a function $tableproperties() Then a topic "TABLE=<tablename>" resp. "FILE=<filename>" to open other files then defined in model Ingo

Why not use the TXT-driver? I did a quick test and loading a 1GB profiling log-file does not seem to be problem. I can also loop through the 3.8+ million lines. It probably is also a good idea to use discard (after analyzing a profiling line) in order to safe/release memory (when not needed anymore). Hope this helps. Regards, Daniel

Perhaps a matter of your memory ?? In most cases where you process large files line by line, using the good old (since 1996) USEQREAD.DLL would be a real booster. because it empowers linear processing speed and minimal memory consumption (only for 1 record) See the old uniface listserver entry from https://groups.google.com/forum/#!topic/comp.soft-sys.app-builder.uniface/-WBbREJqco4

Hi Ingo, I made some stress test with a simple form loading from a profiling log file, duplicating in memory and dumping to another file with 6.644.800 lines sizing 1.280MB!; Uniface (32bit) was using about 3GB as maximum. The issue we are facing with could be related more probably to other objects in memory or DBMS driver. I am currently doing this small test using SQLLite on U9.7.04.G422 plain; which DBMS are you using? Gianni

diseli said Why not use the TXT-driver? I did a quick test and loading a 1GB profiling log-file does not seem to be problem. I can also loop through the 3.8+ million lines. ... Regards, Daniel  

Hi Daniel, very interesting! Could you provide your quick test configuration? UPDATE: No need for this...I just made it! Using U9704.G422 (32bit) I've retrieved a profiling log file sized 1.724GB containing 8863744 rows and looped into it forward and backward. Maximum RAM usage for Uniface process while looping was more than 2.8GB. I am guessing using Uniface 32bit platform (RAM costrained!) maximum profiling log file manageble could be around 10 millions rows...not bad! The only thing I would like to be explained from ULab is the huge overhead Uniface is applying: 1.724GB requires more than 2.8GB to be managed: it is more than 60% overhead. Could I ask for a rough explanation? Thanks in advance. Regards, Gianni

Hi Ingo, I am currently doing some stress test on this same subject. I can confirm the maximum profiling log size I was able to load is a little bit more than 120MB, about 630k profiling lines in my case. Regards, Gianni