Rocket U2 | UniVerse & UniData

 View Only
Back to discussions
Expand all | Collapse all

Reading a file without a SELECT list?

  • 1.  Reading a file without a SELECT list?

    Posted 05-24-2023 12:14
    Edited by Scott Johnston 05-24-2023 13:35

    I finished my contract last week, so back on the market. I landed an interview with a "recognizable company" and went straight to the final round. 

    During the interview one of the panel asked me, "How do you read a file without a select list?" I paused for a moment wondering why anyone would need to do that?  I thought for a moment and came to the conclusion, it may be possible, but I cannot think of how it could be done. I responded to him, "In the twenty years I've worked in multivalue, I have never seen any code that would do that. But I don't ever recall ever having the need". 

    Is it possible?  Does anyone have an example?

    Added:  This person meant without using SELECT anywhere. At TCL or internally in a program. He claims he has a way, I doubt him. 



    ------------------------------
    Scott Johnston
    Sr Software Engineer, Architect & Developer
    Dover DE US
    ------------------------------



  • 2.  RE: Reading a file without a SELECT list?

    ROCKETEER
    Posted 05-24-2023 12:30

    Scott,

    A SELECT statement against a file variable does not create a SELECT list, where a SELECT file at TCL / ECL does. It process the file group by group (aka hash order) and starts straight away without any delay, there are other limitations around doing this around but that's anthoer discussion.

    OPEN "CUSTOMER" TO F.CUSTOMER ELSE ...
    SELECT F.CUSTOMER TO 5
    EOF = @FALSE
    LOOP
    READNEXT K.CUSTOMER FROM 5 ELSE EOF = @TRUE
    UNTIL EOF DO
    ...
    REPEAT



    ------------------------------
    Jonathan Smith
    UniData ATS
    Rocket Support
    ------------------------------

    Original Message


  • 3.  RE: Reading a file without a SELECT list?

    Posted 05-24-2023 13:33

    That's still creates a selection in memory and you are USING a SELECT statement.  This persons claim question was how can you read a file without using a SELECT statement, which always creates a list.  

    I'll edit that post to make it clearer.



    ------------------------------
    Scott Johnston
    Sr Architect &n Developer
    Sonna Solutions
    Dover DE US
    ------------------------------

    Original Message


  • 4.  RE: Reading a file without a SELECT list?

    ROCKETEER
    Posted 05-24-2023 16:23
    Edited by Jonathan Smith 05-24-2023 16:29

    Scott,

    If you want the simplist way I can think of without a SELECT statement. Create an Index on the file of @ID. Use the SETINDEX command to position the program at the first index key in the index and then use the READFWD command. READFWD also has READFWDU and READFWDL to lock records as well.

    OPEN "CUSTOMER" TO F.CUSTOMER ELSE STOP
    SETINDEX "@ID" ON F.CUSTOMER
    EOF = @FALSE
    LOOP
       READFWD R.CUSTOMER FROM F.CUSTOMER ELSE EOF = @TRUE
    UNTIL EOF DO
       CRT @ID : " " : R.CUSTOMER
    REPEAT
    END

    Your original question as per the subject was without a SELECT list. The SELECT statement in Basic of a file variable, simply tells UniData or UniVerse to read through the file group by group and record by record as they are stored in the file. It does not create a list or a selection in memory, it simply sets a pointer to keep track of pointer to the next record (or record id in the id list in UniData)  to process. In UniVerse it follows the forward pointer to next record in the group to get the next item id, in UniData it just goes through the list of item ids stored in the first part of the file (the UniData file structure is different to UniVerse in that it is not a doubly linked list, the top part of the group contains a list of item ids with pointers to where the data is in the second part of the group).

    So based on the original description of not using a SELECT list, the SELECT of file variable does not generate a list and in terms of how it proceses it simply updates a pointer to the next item id after the READNEXT is executed.

    I do have utiltiy programs that use commands commands such as OSBREAD in UniData and READBLK in UniVerse to go through a file block by block, these don't require any SELECT statement just an understanding of the internal file structures.

    There are some low level tools (fileview in UniData for example)  which can display the records in each group of the file, so that would be one very complicated way of doing it, as are the OSBREAD and READBLK methods above without knowing the internals of the file structures.

    I'll be interested to see if anyone has a simple of way of doing this is Basic without using a SELECT statement of any type and not utilizing SETINDEX and READFWD.




    ------------------------------
    Jonathan Smith
    UniData ATS
    Rocket Support
    ------------------------------

    Original Message


  • 5.  RE: Reading a file without a SELECT list?

    Posted 05-24-2023 17:39

    It might me more accurate to say that when selecting a file variable, the file itself *is* the list, and it's processed in logical order (the order in which the records are 'physically' laid out on the disk) , whereas, when a TCL select is executed, a separate list must be generated, usually in a hidden, temporary file.  But not all flavors retain the entire list in RAM.  In fact, I doubt any do.  UniData, for example, splits big lists up into 32K chunks, and loads/processes them individually so as to save RAM.  You can see this by saving the list and editing the resulting items in the savedlists file.

    This is why the preferred method for reading host OS files sequentially (at least on Pick) used to be to select them as as savedlists, and then get the list.  No way would it load the entire file into RAM.  It would just take the next item, just like a readseq does now.

    An interesting experiment would be to build a dynamic list in a program, and then process that list, deleting the top attribute after each loop.  Then compare this to using the TCL select, and then the BASIC select.



    ------------------------------
    Marcus Rhodes
    marcus1@thinqware.com
    ------------------------------

    Original Message


  • 6.  RE: Reading a file without a SELECT list?

    ROCKETEER
    Posted 05-24-2023 18:15

    Hi Marcus,

    The spliting of savedlists into 32K chuck files was changed in UniData 8.1.0 back in 2014. A udtconfig item SINGLE_SAVEDLIST was added with the default value of 1. Any savedlist saved or created with this set is created as a single item without chunking. The 32K limit was based on a very old and obselete value of the largest record that could be read into UniData. So from 8.1.0, the whole list without chunking is loaded into memory as an array.

    UniData also has an internal 'remove pointer' when it comes to processing arrays at an attribute level, which matches the speed of the REMOVE statement / function (this was extended to multivalue and subvalues in 8.2.1 in 2020). You can process through huge arrays accessing them attribute or attribute via a FOR NEXT loop or REMOVE statement. Deleting the top attribute requires using memory based operations around copying, so I would expect processing an array attribute by attribute using a FOR ... NEXT or REMOVE would be really quick.

    Please also see my earlier post on how 'SELECT filevariable' works and a way to do the processing without using a SELECT statement at all using SETINDEX and READFWD.


    Thanks,



    ------------------------------
    Jonathan Smith
    UniData ATS
    Rocket Support
    ------------------------------

    Original Message


  • 7.  RE: Reading a file without a SELECT list?

    Posted 05-25-2023 08:11

    Scott,

    It is possible and in some cases the only way to perform some tasks. The simple case is when you process a file of all cardinal integers. You only need the primary key staring with 1 and process until there are no more records. No select was required. In a B-TREE indexing system you have a root key and then the other index and leaf keys are all link listed to the root record. No select required. Very fast and efficient.

    As Pick BASIC programmers we see the world through multivalued glasses. There are other programming methodologies that exist outside of Pick.

    Jon



    ------------------------------
    Jon Kristofferson
    Pick Programmer
    Snap-on Credit LLC
    Libertyville IL US
    ------------------------------

    Original Message


  • 8.  RE: Reading a file without a SELECT list?

    PARTNER
    Posted 05-25-2023 09:59

    Jon,

    Are you meaning that with foreknowledge of a file's key structure (necessarily simple) and/or indexes, then you can bypass the need to select and go straight to reading?  The original post seemed to be looking for a general method that would always work.  I honestly can't think of anything that doesn't require ahead-of-time setup/knowledge, like "these are the only possible keys" or "this index has been created" or "this is how the data is stored on disc".  I agree that different situations lend themselves to different approaches, not all of which are equally efficient.  I think we can all agree on that.

    We only have the original post as a reference; we don't know how the actual interview question was asked, or if the asker might have clarified their question with enough prodding.  But if we take the original post at face value, then I'm not sure there's a general method that is actually better/more generally applicable than defaulting to a select list of one form or another.  That's just my point of view though.

    Thanks,



    ------------------------------
    Tyrel Marak
    Technical Support Manager
    Aptron Corporation
    Florham Park NJ US
    ------------------------------

    Original Message


  • 9.  RE: Reading a file without a SELECT list?

    PARTNER
    Posted 05-25-2023 09:26

    Hi Scott:

    It could also be someone who doesn't completely understand the internals and thought that if you did a simple LIST ... it's a way of reading records without an 'actual' SELECT.  It could have also been referring to the FORMLIST Basic command to turn a dynamic array into 'select' list.

    Did he ever provide an answer?

    Sam



    ------------------------------
    Samuel A Powell
    President/Developer
    Advanced Transportation Systems Inc
    Colorado Springs CO US
    ------------------------------

    Original Message


  • 10.  RE: Reading a file without a SELECT list?
    Best Answer

    ROCKETEER
    Posted 05-25-2023 10:02
    Edited by Jonathan Smith 05-25-2023 10:38

    @Samuel A Powell  @Jon Kristofferson @Tyrel Marak @Ian McGowan It is possible to do it and I showed how in UniData yesterday but you might of missed the posts, so I suspect the guy asking the question did in fact know a lot about programming.

    The simplist way I can think of without a SELECT statement. Create and Build an Index on the file using @ID (Try this in the demo account). Use the SETINDEX command to position the program at the first index key in the index and then use the READFWD command. READFWD also has READFWDU and READFWDL to lock records as well. BSCAN in UniVerse would allow you to follow a similar logic.


    OPEN "CUSTOMER" TO F.CUSTOMER ELSE STOP
    SETINDEX "@ID" ON F.CUSTOMER
    EOF = @FALSE
    LOOP
       READFWD R.CUSTOMER FROM F.CUSTOMER ELSE EOF = @TRUE
    UNTIL EOF DO
       CRT @ID : " " : R.CUSTOMER
    REPEAT
    END

    Regards,



    ------------------------------
    Jonathan Smith
    UniData ATS
    Rocket Support
    ------------------------------

    Original Message


  • 11.  RE: Reading a file without a SELECT list?

    Posted 05-25-2023 10:32

    If you have a list of keys from some other source, or you know the structure of the keys (maybe they are ascending integers < 999,999), then you can read the records in without EXECUTE \SELECT..\, or GET.LIST or the SELECT command.  Otherwise, we're going to need you to call them back and find out what their answer is  :-)  It seems like one of those interview gotcha questions to me - my money is on the Basic SELECT command, which I don't think of as creating a select list, but just traversing the file.



    ------------------------------
    Ian McGowan
    Principal Consultant
    Rocket Forum Shared Account
    ------------------------------

    Original Message


  • 12.  RE: Reading a file without a SELECT list?

    ROCKETEER
    Posted 05-25-2023 10:58

    Scott,

    There are several good answers on the thread, but I see the issue with the question.  I believe you would need more information, to properly answer.

    My first thought is what type of file?  
    We may assume the interviewer was discussing a MultiValue file, but in fact it could be sequential, where no SELECT is needed to read the entire file.
    We know nothing about the keys, as suggested in another post, and if there is an ID counter used in the file, no select is needed.
    Lastly, I agree with Jonathan, if there is an index on the id, setting the index, and reading through the file is a good method to avoid a large SELECT list, and is what I expect the interviewer is doing.



    ------------------------------
    Mike Rajkowski
    MultiValue Product Evangelist
    Rocket Internal - All Brands
    US
    ------------------------------

    Original Message


  • 13.  RE: Reading a file without a SELECT list?

    Posted 05-26-2023 08:39

    Are over thinking things, a READ doesn't need a select, I wrote a lot of applications that didn't contain a single select. The other way is using READSEQ.



    ------------------------------
    Leslie Sherlock Hewkin
    Business Analyst
    Rocket Forum Shared Account
    ------------------------------

    Original Message


  • 14.  RE: Reading a file without a SELECT list?

    Posted 05-26-2023 10:35

    You don't need to select a file to anything

    OPEN N.FILE TO F.FILE ELSE dosomethingelse
    SELECT F.FILE
    READNEXT K.FILE

    there is no select LIST created, the select only opens a pointer to the start of the file and the pointer is pointing to where it's traversing to next
    no LIST at all



    ------------------------------
    Will Johnson
    Systems Analyst
    Rocket Forum Shared Account
    ------------------------------

    Original Message


  • 15.  RE: Reading a file without a SELECT list?

    ROCKETEER
    Posted 05-26-2023 11:02

    Will,

    I believe that is a grey area, the documentation defines what is returned from the SELECT statement as a active select list of all record IDs in a file.  Yet it also acts differently from the SELECT done at ECL/TCL, in that it processes one group at a time.  




    ------------------------------
    Mike Rajkowski
    MultiValue Product Evangelist
    Rocket Internal - All Brands
    US
    ------------------------------

    Original Message


  • 16.  RE: Reading a file without a SELECT list?

    Posted 05-26-2023 11:10

    I will stipulate to the idea that it returns the first group.  I personally found that out by accident, 30 years ago when I parsed the string returned.  I didn't know it had been documented since then.

    So yes this is a list, so I suppose I should modify my statement to say that there is a pointer to the frame start, and a text string of the first groups contents.  Then when that group is exhausted, it triggers the pointer to move forward one group.

    I still think this is what the interviewer was going for however.



    ------------------------------
    Will Johnson
    Systems Analyst
    Rocket Forum Shared Account
    ------------------------------

    Original Message