Skip to main content

I have a statistics program that builds an array with <1,POS>= sku and <2,POS>=qty.  The output would often have 8-9k SKUs.

When I dump it out to a sequential file, I want to output the data as  a sinlge row for each sku in the format of SKU:<delimiter>:QTY

I know I can do this programatically using a for...next loop, but wondered if ther was a quicker/built-in way to do this?

Nelson



------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

I have a statistics program that builds an array with <1,POS>= sku and <2,POS>=qty.  The output would often have 8-9k SKUs.

When I dump it out to a sequential file, I want to output the data as  a sinlge row for each sku in the format of SKU:<delimiter>:QTY

I know I can do this programatically using a for...next loop, but wondered if ther was a quicker/built-in way to do this?

Nelson



------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

Hi Nelson,

It can be done but probably not as simply as a for next loop.  There's a CATS function that will concatenate to dynamic arrays  and return a new dynamic array with the results.  It's not that simple though if you want to concatenate a new delimiter.  Here's an example that does it two ways. Take a look and see which one you think is easier to understand and maintain.

      ARRAY = 'A1':@VM:'A2':@VM:'A3':@VM:'A4'
      ARRAY<2> = 'B1':@VM:'B2':@VM:'B3':@VM:'B4'

      LN = DCOUNT(ARRAY<1>, @VM)

      * PUT THEM TOGETHER WITH A FOR NEXT LOOP
      LOOP.RESULT = ''
      FOR LINE.NO = 1 TO LN
         LOOP.RESULT<1,LINE.NO> = ARRAY<1,LINE.NO>:'-':ARRAY<2,LINE.NO>
      NEXT

      PRINT LOOP.RESULT

      * DO THE SAME THING WITH THE CATS FUNCTION

      * CATS CONACTENATES THE VALUES FROM ONE ARRAY TO ANOTHER ARRAY AND RETURNS THE RESULTING ARRAY
      * IN ORDER TO USE IT, WE NEED TO HAVE A CORRECTLY SIZED ARRAY OF THE DELMITERS SO THAT WE
      * CAN CONACTENATE IT WITH THE SECOND ARRAY
      DELIMS = STR('-':@VM, LN)

      * WE'RE ALWAYS GOING TO END UP WITH AN EXTRA ELEMENT AT THE END SO WE NEED TO STRIP THAT OFF
      DELIMS = FIELD(DELIMS, @VM, 1, LN)

      * NOW WE CAN BUILD A NEW ARRAY BY CONCACTENATING OUR DELIMITERS WITH ARRAY<2>
      DELIM.ARRAY = CATS(DELIMS, ARRAY<2>)

      * FINALLY WE CAN BUILD THE NEW ARRAY WITH ARRAY<1> AND DELIM.ARRAY
      RESULT = CATS(ARRAY<1>, DELIM.ARRAY)
      
      PRINT RESULT

      STOP

If you're trying to get the best performance, CATS is faster. To loop through each of those sections 999,999 times took the first method 2,510 milliseconds. The second took 960 milliseconds.  For most real world cases it's better to go with understandable.



------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US
------------------------------

Hi Nelson,

It can be done but probably not as simply as a for next loop.  There's a CATS function that will concatenate to dynamic arrays  and return a new dynamic array with the results.  It's not that simple though if you want to concatenate a new delimiter.  Here's an example that does it two ways. Take a look and see which one you think is easier to understand and maintain.

      ARRAY = 'A1':@VM:'A2':@VM:'A3':@VM:'A4'
      ARRAY<2> = 'B1':@VM:'B2':@VM:'B3':@VM:'B4'

      LN = DCOUNT(ARRAY<1>, @VM)

      * PUT THEM TOGETHER WITH A FOR NEXT LOOP
      LOOP.RESULT = ''
      FOR LINE.NO = 1 TO LN
         LOOP.RESULT<1,LINE.NO> = ARRAY<1,LINE.NO>:'-':ARRAY<2,LINE.NO>
      NEXT

      PRINT LOOP.RESULT

      * DO THE SAME THING WITH THE CATS FUNCTION

      * CATS CONACTENATES THE VALUES FROM ONE ARRAY TO ANOTHER ARRAY AND RETURNS THE RESULTING ARRAY
      * IN ORDER TO USE IT, WE NEED TO HAVE A CORRECTLY SIZED ARRAY OF THE DELMITERS SO THAT WE
      * CAN CONACTENATE IT WITH THE SECOND ARRAY
      DELIMS = STR('-':@VM, LN)

      * WE'RE ALWAYS GOING TO END UP WITH AN EXTRA ELEMENT AT THE END SO WE NEED TO STRIP THAT OFF
      DELIMS = FIELD(DELIMS, @VM, 1, LN)

      * NOW WE CAN BUILD A NEW ARRAY BY CONCACTENATING OUR DELIMITERS WITH ARRAY<2>
      DELIM.ARRAY = CATS(DELIMS, ARRAY<2>)

      * FINALLY WE CAN BUILD THE NEW ARRAY WITH ARRAY<1> AND DELIM.ARRAY
      RESULT = CATS(ARRAY<1>, DELIM.ARRAY)
      
      PRINT RESULT

      STOP

If you're trying to get the best performance, CATS is faster. To loop through each of those sections 999,999 times took the first method 2,510 milliseconds. The second took 960 milliseconds.  For most real world cases it's better to go with understandable.



------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US
------------------------------

Oops, I forgot to show the output.

A1-B1ýA2-B2ýA3-B3ýA4-B4
A1-B1ýA2-B2ýA3-B3ýA4-B4



------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US
------------------------------

Hi Nelson,

It can be done but probably not as simply as a for next loop.  There's a CATS function that will concatenate to dynamic arrays  and return a new dynamic array with the results.  It's not that simple though if you want to concatenate a new delimiter.  Here's an example that does it two ways. Take a look and see which one you think is easier to understand and maintain.

      ARRAY = 'A1':@VM:'A2':@VM:'A3':@VM:'A4'
      ARRAY<2> = 'B1':@VM:'B2':@VM:'B3':@VM:'B4'

      LN = DCOUNT(ARRAY<1>, @VM)

      * PUT THEM TOGETHER WITH A FOR NEXT LOOP
      LOOP.RESULT = ''
      FOR LINE.NO = 1 TO LN
         LOOP.RESULT<1,LINE.NO> = ARRAY<1,LINE.NO>:'-':ARRAY<2,LINE.NO>
      NEXT

      PRINT LOOP.RESULT

      * DO THE SAME THING WITH THE CATS FUNCTION

      * CATS CONACTENATES THE VALUES FROM ONE ARRAY TO ANOTHER ARRAY AND RETURNS THE RESULTING ARRAY
      * IN ORDER TO USE IT, WE NEED TO HAVE A CORRECTLY SIZED ARRAY OF THE DELMITERS SO THAT WE
      * CAN CONACTENATE IT WITH THE SECOND ARRAY
      DELIMS = STR('-':@VM, LN)

      * WE'RE ALWAYS GOING TO END UP WITH AN EXTRA ELEMENT AT THE END SO WE NEED TO STRIP THAT OFF
      DELIMS = FIELD(DELIMS, @VM, 1, LN)

      * NOW WE CAN BUILD A NEW ARRAY BY CONCACTENATING OUR DELIMITERS WITH ARRAY<2>
      DELIM.ARRAY = CATS(DELIMS, ARRAY<2>)

      * FINALLY WE CAN BUILD THE NEW ARRAY WITH ARRAY<1> AND DELIM.ARRAY
      RESULT = CATS(ARRAY<1>, DELIM.ARRAY)
      
      PRINT RESULT

      STOP

If you're trying to get the best performance, CATS is faster. To loop through each of those sections 999,999 times took the first method 2,510 milliseconds. The second took 960 milliseconds.  For most real world cases it's better to go with understandable.



------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US
------------------------------

Well $%@#@,

You ca use the SPLICE function:

      PRINT SPLICE(ARRAY<1>, '-', ARRAY<2>)

It's a bit simpler.



------------------------------
Joe Goldthwaite
Consultant
Phoenix AZ US
------------------------------

I have a statistics program that builds an array with <1,POS>= sku and <2,POS>=qty.  The output would often have 8-9k SKUs.

When I dump it out to a sequential file, I want to output the data as  a sinlge row for each sku in the format of SKU:<delimiter>:QTY

I know I can do this programatically using a for...next loop, but wondered if ther was a quicker/built-in way to do this?

Nelson



------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

 I created a subroutine to work like Excel to transpose an array.  I've attached it for reference.  My first pass I used Splice but it was slow on larger data sets.  I hope this helps.

Edit:

I don't know why uploading did not seem to attach the file, so I've added it here.

SUBROUTINE TRANSPOSE.ARRAY(ITEM)
* ------------------------------------------------------------------------
*  Name        : TRANSPOSE.ARRAY
*  Description : Transpose an array
*  Programmer  : Joseph von Arx
*  Date        : 7/20/2020
*  ------------------------------------------------------------------------
*
*  Notes
*  -----
*
*  Modification History
*  Date        Who  Description
*  ----------  ---  -----------------------
*  03/20/2023  JLVA Rewrote code for optimization and efficiency for larger arrays
*
* ------------------------------------------------------------------------
    $OPTIONS DEFAULT
    FMAX = COUNT(ITEM, @AM)+1
    DIM WORK(FMAX)
    MAT WORK = ''
    MATPARSE WORK FROM ITEM USING @AM
!***
!* MAKE SURE ALL FIELDS HAVE SAME NUMBER OF MULTIVALUES
!***
    VMAX = COUNT(WORK(1), @VM)+1
    FOR FMC = 2 TO FMAX
        VMC = COUNT(WORK(FMC), @VM)+1
        IF VMC GT VMAX THEN VMAX = VMC
    NEXT FMC
!***
!* Extract the elements and load them into an array.
!* This should be faster than <> extraction on large Items.
!***
    DIM ARRAY(VMAX)
    MAT ARRAY = ''
    DIM VALARRAY(VMAX)
    FOR II = 1 TO FMAX
        LINE = WORK(II)
        MAT VALARRAY = ''
        MATPARSE VALARRAY FROM LINE USING @VM
        FOR JJ = 1 TO VMAX
            IF II # 1 THEN ARRAY(JJ) := @VM
            VAL = VALARRAY(JJ)
            ARRAY(JJ) := VAL
        NEXT JJ
    NEXT II
    MAT VALARRAY = ''
    MAT WORK = ''
!***
!* Done.
!***
    MATBUILD ITEM FROM ARRAY USING @AM
    ICNT = DCOUNT(ITEM, @AM)
    IF ICNT LT VMAX THEN
        ITEM := STR(@AM, (VMAX - ICNT))
    END
*
    RETURN
END



------------------------------
Joseph von Arx
Software Developer
Data Management Associates Inc DMA
Cincinnati OH US
------------------------------


 I created a subroutine to work like Excel to transpose an array.  I've attached it for reference.  My first pass I used Splice but it was slow on larger data sets.  I hope this helps.

Edit:

I don't know why uploading did not seem to attach the file, so I've added it here.

SUBROUTINE TRANSPOSE.ARRAY(ITEM)
* ------------------------------------------------------------------------
*  Name        : TRANSPOSE.ARRAY
*  Description : Transpose an array
*  Programmer  : Joseph von Arx
*  Date        : 7/20/2020
*  ------------------------------------------------------------------------
*
*  Notes
*  -----
*
*  Modification History
*  Date        Who  Description
*  ----------  ---  -----------------------
*  03/20/2023  JLVA Rewrote code for optimization and efficiency for larger arrays
*
* ------------------------------------------------------------------------
    $OPTIONS DEFAULT
    FMAX = COUNT(ITEM, @AM)+1
    DIM WORK(FMAX)
    MAT WORK = ''
    MATPARSE WORK FROM ITEM USING @AM
!***
!* MAKE SURE ALL FIELDS HAVE SAME NUMBER OF MULTIVALUES
!***
    VMAX = COUNT(WORK(1), @VM)+1
    FOR FMC = 2 TO FMAX
        VMC = COUNT(WORK(FMC), @VM)+1
        IF VMC GT VMAX THEN VMAX = VMC
    NEXT FMC
!***
!* Extract the elements and load them into an array.
!* This should be faster than <> extraction on large Items.
!***
    DIM ARRAY(VMAX)
    MAT ARRAY = ''
    DIM VALARRAY(VMAX)
    FOR II = 1 TO FMAX
        LINE = WORK(II)
        MAT VALARRAY = ''
        MATPARSE VALARRAY FROM LINE USING @VM
        FOR JJ = 1 TO VMAX
            IF II # 1 THEN ARRAY(JJ) := @VM
            VAL = VALARRAY(JJ)
            ARRAY(JJ) := VAL
        NEXT JJ
    NEXT II
    MAT VALARRAY = ''
    MAT WORK = ''
!***
!* Done.
!***
    MATBUILD ITEM FROM ARRAY USING @AM
    ICNT = DCOUNT(ITEM, @AM)
    IF ICNT LT VMAX THEN
        ITEM := STR(@AM, (VMAX - ICNT))
    END
*
    RETURN
END



------------------------------
Joseph von Arx
Software Developer
Data Management Associates Inc DMA
Cincinnati OH US
------------------------------

Coming to this late but always worth remembering for UniVerse:

MATPARSE is quick but so is iterating through fields (not values) in a dynamic array because of the hint pointer in a dynamic array.

Often just rewriting:

For i = 1 To NumberOfValues

    ... MyArray<1, i>

    ... MyArray<2,i>

Next

to promote the values:

MyField1 = Convert(@VM,@FM, MyArray<1>)
MyField2 = Convert(@VM, @FM, MyArray<2>)

and then iterating at the field level (so it doesn't go back to the start to position the extract) can be so much quicker.

Also, time taken to build the array in the first place - you said you have a statistics program that builds the array with two sets of values. 

One option is to use UDOs, yes they are pig-ugly to code (!) but I've found can be a lot quicker than using large arrays in Basic, which makes sense when you think about the memory allocation underneath. Not knowing the context of what you are doing to create the arrays in the first place, that might be quicker still to build and render out if each sku is a UDO property added to an array. Same for a python dictionary of course.



------------------------------
Brian Leach
Director
Brian Leach Consulting
Chipping Norton GB
------------------------------

I have a statistics program that builds an array with <1,POS>= sku and <2,POS>=qty.  The output would often have 8-9k SKUs.

When I dump it out to a sequential file, I want to output the data as  a sinlge row for each sku in the format of SKU:<delimiter>:QTY

I know I can do this programatically using a for...next loop, but wondered if ther was a quicker/built-in way to do this?

Nelson



------------------------------
Nelson Schroth
president
C3CompleteShop LLC
Harrison OH US
------------------------------

You can also use python to convert and write your csv (or xlsx if that's the end game). I haven't tested the transform.

#  Write to "PP/ConvertToCSV.py"
import pandas as pd
def write_csv(mvList,fileName):
   UVAM = '\\uf8fe'; UVVM = '\\uf8fd'
   pd.read_csv(mvList.replace(UVAM,'\\n'),UVVM)
   pd.T.to_csv(fileName, header=False, index=False) # T to transform

* write to BP CSV.TEST, compile and run 
program CSV.TEST

   mvList      = 'name':@vm:'address':@vm:'notes':@vm:'numbers'
   mvList<2>   = 'Miss "January" Jones':@vm:'666 Mockingjay way, Parramatta NSW 2200'
   mvList<2,3> = 'These are some notes':char(10):'with, arbitrarily, a line break.':char(10):'Third line.'
   mvList<2,4> = 123.45
   moduleName  = 'ConvertToCSV'
   funcName    = 'write_csv'
   outName     = 'output.csv'

   pyresult = PyCallFunction(moduleName, funcName, mvList, outName)

   crt 'Done ':@pyexceptionmsg
end



------------------------------
Stuart Boydell
Technical Specialist
Meier Business Systems PTY LTD
Carnegie Vic AU
------------------------------