Building a Uniface application

Hi - We are currently using Uniface v10.3 and are looking at ways to speed up the build time for the application. Our build process typically runs for a couple of hours and we are looking to cut down this time to a few minutes, if possible. We are currently doing the build using a powershell command and it's being executed as a single thread process.
We are looking at options to run the build as a multi-thread process but I didn't find any reference in the documentation on whether/how this can be achieved. I would also like to know why the "full" build takes such a long time and if there are ways to cut it down e.g. caching.

Can anyone provide details on the sub-steps involved in a Uniface build - e.g. dependency gathering, compilation of individual components, packaging etc? Depending on which process takes maximum time, we can focus on reducing it?

Any help will be appreciated.

Thanks
Vaibhav

------------------------------
Vaibhav Kolhe
Manager
Accenture Limited Liability Partnership
Mumbai IN
------------------------------

Okay, first question is what are you outputting to? If it's a single uar file, then you cannot run multi threaded, as each compile locks the uar file and 'competing' compiles will fail to merge one or other compiled component. 
If you are running the compile output to a folder structure (we're talking about $resources_output here) then you can run the compile in bits. 

Here is the section of our 'full compile' batch file which deals with breaking down /all into the subsections.

"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile.asn /obj /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /con /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /sig /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /frm /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /svc /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /rpt /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /usp /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /dsp /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /esv /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /ssv /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /ceo /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /dtd /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /app /war
"c:\\apps\\compuware\\uniface\\common\\bin\\urm.exe" move resource\\main_system.uar:/frm/* resource/main_forms.uar:/frm/*
"c:\\apps\\compuware\\uniface\\common\\bin\\urm.exe" move resource\\main_system.uar:/svc/* resource/main_svc.uar:/svc/*

Our RESOURCES_OUTPUT is set to main_system.uar, and after all the compiles are done, we then use the urm to split the output into  main_forms and main_svc (leaving all the other stuff in main_system).  The difference between the to assignment files (compile & compile2) is that one is set to create a new log file, and the other to append to it, which gives us a single file to check for errors. Again, multi-threaded you would want to create multiple files and stitch them back together afterward. 
You could, if you were compiling to a folder rather than a uar, run all the sections separately, and include patterns in the component compile to split them up further (e.g.  "C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /frm /war "A*" to compile only the forms starting with A.)

The above is the suggested order from the /all help section. And I would definitely run /obj on its own separately before going multi threaded so that global objects are available. 

Regards, 
Iain

------------------------------
Iain Sharp
Head of Technical Services
Pci Systems Ltd
Sheffield GB
------------------------------

Okay, first question is what are you outputting to? If it's a single uar file, then you cannot run multi threaded, as each compile locks the uar file and 'competing' compiles will fail to merge one or other compiled component. 
If you are running the compile output to a folder structure (we're talking about $resources_output here) then you can run the compile in bits. 

Here is the section of our 'full compile' batch file which deals with breaking down /all into the subsections.

"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile.asn /obj /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /con /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /sig /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /frm /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /svc /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /rpt /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /usp /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /dsp /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /esv /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /ssv /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /ceo /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /dtd /war
"C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /app /war
"c:\\apps\\compuware\\uniface\\common\\bin\\urm.exe" move resource\\main_system.uar:/frm/* resource/main_forms.uar:/frm/*
"c:\\apps\\compuware\\uniface\\common\\bin\\urm.exe" move resource\\main_system.uar:/svc/* resource/main_svc.uar:/svc/*

Our RESOURCES_OUTPUT is set to main_system.uar, and after all the compiles are done, we then use the urm to split the output into  main_forms and main_svc (leaving all the other stuff in main_system).  The difference between the to assignment files (compile & compile2) is that one is set to create a new log file, and the other to append to it, which gives us a single file to check for errors. Again, multi-threaded you would want to create multiple files and stitch them back together afterward. 
You could, if you were compiling to a folder rather than a uar, run all the sections separately, and include patterns in the component compile to split them up further (e.g.  "C:\\apps\\compuware\\uniface\\common\\bin\\idf.exe" /ini=asns\\usys.ini /asn=asns\\pci_idf_test_compile2.asn /frm /war "A*" to compile only the forms starting with A.)

The above is the suggested order from the /all help section. And I would definitely run /obj on its own separately before going multi threaded so that global objects are available. 

Regards, 
Iain

------------------------------
Iain Sharp
Head of Technical Services
Pci Systems Ltd
Sheffield GB
------------------------------

Hi - Thanks for your response.

1. We are outputting to a single UAR file, however, this was the way it was being done until now. But if we need to change it to make the process quicker, would be open to suggestions.
2. Regarding compiling to a folder, I am not completely following your explanation. Do you have a reference to the Uniface documentation regarding this? I didn't quite understand the split of main_forms & main_svc. I assume this relates to forms and services components? 
3. I would like to know if this has been implemented by anyone and how much time savings, did they achieve via this.

------------------------------
Vaibhav Kolhe
Manager
Accenture Limited Liability Partnership
Mumbai IN
------------------------------

Hi - Thanks for your response.

1. We are outputting to a single UAR file, however, this was the way it was being done until now. But if we need to change it to make the process quicker, would be open to suggestions.
2. Regarding compiling to a folder, I am not completely following your explanation. Do you have a reference to the Uniface documentation regarding this? I didn't quite understand the split of main_forms & main_svc. I assume this relates to forms and services components? 
3. I would like to know if this has been implemented by anyone and how much time savings, did they achieve via this.

------------------------------
Vaibhav Kolhe
Manager
Accenture Limited Liability Partnership
Mumbai IN
------------------------------

In your IDE assignment file will be a line $RESOURCES_OUTPUT=somethingorother.uar. If you change this line to $RESOURCES_OUTPUT=afolderpath (not a uar)  then uniface will create subfolders and files inside that folder rather than in the uar (which is basically just a zip folder).  
Here is a link to the help page for resources_output
As a standard filesystem folder, it is not locked to single use write, and can therefore be written by more than one process at once. 
This is the only way I can see that you could have more than one compile running against the same output. 
You can then use 'standard' zip utilities to reconfigure the separate files and folders into a UAR (we just basically highlight the frm,svc etc. folders and use 'send to compressed folder' from windows). 

Another alternative would be to have a copy of the assignment file for every separate compile being done, all with different UAR outputs, and then use the urm move/copy command in a similar manner to that shown at the end of my run to recombine them into one (or more) UAR at the end. 

I have no idea what time savings you would make, it entirely depends on the types and distributions of the objects you build, and whether you further break down (say) /frm into 26 separate compiles all run simultaneously.  (/frm "a*", /frm "b*" etc.). Each compile is (probably) going to interfere with the others, and uniface is prone to running the core it's on to 100%, so it probably doesn't make sense to run more compiles at once than you have CPU cores to run them on. 

As far as I can tell, the compiler compiles each object separately, and there are no 'phases' you can control to speed this up, (no caching, pretty spotty identification of modification times, no cascade of modification times (so changing a template (modeled) program, or field template for that matter and running the compile using /aft does (or at least did) not compile the components dependant on it. ))

I have written stuff into our repository using triggers to try and track what objects have changed when, but I've not got 100% coverage, if you use version control, you may be able to use it to track changed sources, and from there build a list of objects to recompile so that you are not building the whole system all at once.

Or you can use the information above to recompile the whole system, but in separate threads, and then combine the output into as many (or few) UAR files as you want to deploy. 

Short version, there's no out of the box "make my full compile faster", or "only compile changed objects".  Some of the stuff above may help you to tune what you are doing. 

Regards, 
Iain

------------------------------
Iain Sharp
Head of Technical Services
Pci Systems Ltd
Sheffield GB
------------------------------

Hi - We are currently using Uniface v10.3 and are looking at ways to speed up the build time for the application. Our build process typically runs for a couple of hours and we are looking to cut down this time to a few minutes, if possible. We are currently doing the build using a powershell command and it's being executed as a single thread process.
We are looking at options to run the build as a multi-thread process but I didn't find any reference in the documentation on whether/how this can be achieved. I would also like to know why the "full" build takes such a long time and if there are ways to cut it down e.g. caching.

Can anyone provide details on the sub-steps involved in a Uniface build - e.g. dependency gathering, compilation of individual components, packaging etc? Depending on which process takes maximum time, we can focus on reducing it?

Any help will be appreciated.

Thanks
Vaibhav

------------------------------
Vaibhav Kolhe
Manager
Accenture Limited Liability Partnership
Mumbai IN
------------------------------

Hi

First of all, why do you compile the whole application? :-)
Build a simple helper component which for example loop over all froms in UCFORM (filter by a date) and with the UDE-statement you can compile them one by one

Second, how to avoid locking of UAR

We use the filesystem for the resource output
$RESOURCES_OUTPUT = \\\\server\\path\\app_version\\resources_output
UnifAce create then sub directory frm,svc,prc,... where it stores the output

To get an UAR out of this, just use any ZIP-tool and ZIP the whole directory
But be aware, the the directory 'resources_output' is not part of the patches in the ZIP-File but only \\frm,\\svc,\\prc,...
Then rename this ZIP to "AppSources.uar"

If you want to deliever only diff-patches, just include only the necessary 'components'
"AppSources_patch_nnn.uar"

At customer site use this syntax:
[RESOURCES]
\\\\customer_server\\path\\application\\resources\\AppSources_patch_nnn.uar
\\\\customer_server\\path\\application\\resources\\AppSources.uar

That's it :-)

Ingo

PS: We use this also internaly for our develepment- and test-environments. And it works :-)

------------------------------
Ingo Stiller
Aareon Deutschland GmbH
------------------------------

Hi

First of all, why do you compile the whole application? :-)
Build a simple helper component which for example loop over all froms in UCFORM (filter by a date) and with the UDE-statement you can compile them one by one

Second, how to avoid locking of UAR

We use the filesystem for the resource output
$RESOURCES_OUTPUT = \\\\server\\path\\app_version\\resources_output
UnifAce create then sub directory frm,svc,prc,... where it stores the output

To get an UAR out of this, just use any ZIP-tool and ZIP the whole directory
But be aware, the the directory 'resources_output' is not part of the patches in the ZIP-File but only \\frm,\\svc,\\prc,...
Then rename this ZIP to "AppSources.uar"

If you want to deliever only diff-patches, just include only the necessary 'components'
"AppSources_patch_nnn.uar"

At customer site use this syntax:
[RESOURCES]
\\\\customer_server\\path\\application\\resources\\AppSources_patch_nnn.uar
\\\\customer_server\\path\\application\\resources\\AppSources.uar

That's it :-)

Ingo

PS: We use this also internaly for our develepment- and test-environments. And it works :-)

------------------------------
Ingo Stiller
Aareon Deutschland GmbH
------------------------------

That's what I was saying about part compiles, but it isn't as simple as looping uform (ucform?)  if you rely on templates, or modeled code in entities, or messages, or include proc, or global proc. Some of these no longer have modification dates stored, meaning you can't find the changed ones directly from uniface IDE, which is why I manage an entity using triggers to detect modifications in the source not covered by internal modification dates. 
We do a full compile of the development setup every night, because it's impossible to guarantee I've caught every modification which needs to be compiled into the components. 

But the patching thing, yes exactly. 

Regards, 
Iain

------------------------------
Iain Sharp
Head of Technical Services
Pci Systems Ltd
Sheffield GB
------------------------------

That's what I was saying about part compiles, but it isn't as simple as looping uform (ucform?)  if you rely on templates, or modeled code in entities, or messages, or include proc, or global proc. Some of these no longer have modification dates stored, meaning you can't find the changed ones directly from uniface IDE, which is why I manage an entity using triggers to detect modifications in the source not covered by internal modification dates. 
We do a full compile of the development setup every night, because it's impossible to guarantee I've caught every modification which needs to be compiled into the components. 

But the patching thing, yes exactly. 

Regards, 
Iain

------------------------------
Iain Sharp
Head of Technical Services
Pci Systems Ltd
Sheffield GB
------------------------------

Hi Ian

I know, it's not only UCFORM :-)

But to compile the components cost most of the time.
So use on instance of IDE to compile components:
  If you are sure that only a couple of components have changed, just comile them.
  If you changed a table definition, use UXGROUP to compile all components depending on that table/entity
  If you changed them basic for your components (includes&co) compile all components
     But only components or even only specific component types (frm,esc,ssv,...)
     and you can divide this by a retrieve profile like
       first IDE:       label='<N'
       second IDE: label='>=N'
      (or even more groups)

Use other instances of IDE for compiling global things like menues, signatures,variables,...

With the resource_output into filesystem there is no interlocking :-)

Btw: don't forget

  to analyze the modells involed
  to compile signatures (components and entities (occurence/collections))

Ingo

------------------------------
Ingo Stiller
Aareon Deutschland GmbH
------------------------------