Passing record sets from client/server

Hi Iain,

In a C/S - what would be the benefit in doing this?  Locking is gone, update concurrency is gone (unless you implement $occcrc everywhere), unless you're very careful stepped hitlists are gone - I could go on...

XML is expensive - I totally get your point re DTD's - however, could you distribute the DTD 'source' from DICT with your application and store in a runtime db table - with a $DBMS_FIRST setting?  Everything comes at a cost I guess..

I guess Uniface sales must love you due to the Uniface Application servers you'd need to purchase on the way through as well...  😉

Only saying this because we're running 4500 users across some 300+ office servers - to a central, umpteen TB Oracle 12 server.  Just the mere thought of running server based data retrival makes me shiver...

Regards,

Knut

Well, we already do this, we just do it using putlistitems/getlistitems, which does NOT carry the modification status of fields or occurrences. 

The XML functions (xmlload/xmlsave) and retrieve/reconnect do (mostly) allow for this and are the same therefore as optimistic locking. 

We are therefore already licensed for application servers. 

We'd have to rewrite the whole system to connect the UI direct to the database I was looking to make the current data set passing more efficient and functional. 

Iain

All this because when we were looking at languages > 15years ago Uniface was sold as an N-tier object oriented development language. What we have now is the 15year old result of us trying to beat uniface 7/8 into something resembling either OO or N-tier... 

I hear you....

I presume you added $occcrc to the entities as a non-dbms field, and include that field in all back'n forth between client and server?  At the time of reconnect - rather than having to compare each field - the crc check would be much faster to do...

and as for the dtd mapping - if it's stored in the db at least you don't have to update the dol/urr whenever the dtd changes...

Not sure $occcrc existed when we built the original system. 

Using xmlsave/xmload correctly with retrieve/reconnect is SUPPOSED to do all that for you within the uniface code. I do believe it works with crc values within the XML as well as whether the client THINKS it's modified the data. 

componenttostruct/reconnecttags is supposed to do the same thing within structs. But then you can't pass a struct from one machine to another so you have to do structtojson (or structtoxml) to give you something which will pass. I was looking to see what the current approved 'standard' should be. 

We stopped using dol/urr a long time ago, We're all about the uar now. 🙂 Way better for patching. The DTD is, therefore, a separate file now and easy to manage on that front. It's the maintenance of the thing, every field in the entity has to be explicitly added manually. If it isn't then it's not included in the passed data. If it's not included then the reconnect wipes the data from the database entry. So two programs with two DTDs (different structures) can therefore add data and take it away, because a developer missed the update  on a DTD. 

Oh, and it's not possible (easy) to search DTDs to do a validity check that it's not been missed. And the only thing you can be sure about with manual documentation is that it's out of date/wrong.... 

Hi guys,

Thanks for start up and giving fire to this subject!

Gianni

So I've tried structtojson and structoxml and there is no sign of the reconnecttags in the strings they output. As such the retrieve/reconnect fails to reset the modification status. 

Much as I'd like to say I added fire to this discussion, it does seem I'm screaming in the wind here. I can only presume that no-one is actually using Uniface in the method they suggest as it's just broken... 

Iain

I can see the $tags in the created struct, But they are not included in the XML (or JSON), there doesn't appear to be a method for including them. 

I am wondering if I can do a pass through the struct, copying the contents of a node's $tags, to a non tag struct, then converting then changing them back when using the struct at the other end. This is getting very complex though. Although, done properly, it would be reusable, flexible code which would still cope with new fields, so still better than using DTDs. 

All in all, I do feel like I'm beating my head against a wall... 

Iain

This is not really broken, but rather works by design.

Before raising your blood pressure, please continue reading...

When you do componentToStruct/reconnectTags and then structToComponent then this work without any further changes.

If you, however, convert the Struct to either XML or JSON then you need (as it works now) to move the reconnect tags before you can do the conversion. For XML you would create XML attributes that hold the reconnect info. For JSON you need to do something similar. On the way back you need to reverse the process. Granted, this requires a bit of work, but it's possible to create a generic routine (one for XML and one for JSON) that does this.

I'm sure that I have a possible sample routine somewhere lying around. I'll have a look when I get some spare time.

And in the meantime you could have a look at (e.g.) this Struct example:

> Scripting Application Behavior > ProcScript Syntax > Structs > Example: Looping Over Structs

I've already used this numerous times if I needed to manipulate a Struct in a generic way.

I hope this helps.

Daniel

Here's the code for moving/restoring the reconnect info for XML:

;----------------------------------------------------------------
entry moveReconnectData
;----------------------------------------------------------------
params
   struct p_struct      : IN
endparams

variables
   numeric i
   numeric n
endvariables

   ;If this struct has more than 1 member then loop over them
   ;calling moveReconnectData recursively on each
   if (p_struct->$collSize > 1)

       n = p_struct->$collSize
       for i = 1 to n
          call moveReconnectData(p_struct{i})
       endfor

       return 0

   endif

   ;This struct only has 1 member so process it.
   ;First, move the u_type tag into an attribute
   p_struct->$tags->u_type->$parent = p_struct->$tags->$parent
   p_struct->u_type->$index = 1
   p_struct->*{1}->$tags->xmlClass = "attribute"

   ;Now see what type it is, component, entity, occurrence
   selectcase p_struct->u_type
       case "component"

          ;This is the component node, so call this function
          ;recursively on all the entity nodes under it
          call moveReconnectData(p_struct->*)

       case "entity"

          ;This is an entity node, so call this function
          ;recursively on all the occurrence nodes
          ;under it (If they exist)
          if (p_struct->$membercount > 0)
             call moveReconnectData(p_struct->*)
          endif

       case "occurrence"

          ;This is an occurrence node, so just call this function
          ;recursively on all the entity or field nodes under it.
          call moveReconnectData(p_struct->*)

          ;Move reconnect data into attributes
          p_struct->$tags->u_status->$parent = p_struct->$tags->$parent
          p_struct->u_status->$index = 1
          p_struct->*{1}->$tags->xmlClass = "attribute"
          p_struct->$tags->u_crc->$parent = p_struct->$tags->$parent
          p_struct->u_crc->$index = 1
          p_struct->*{1}->$tags->xmlClass = "attribute"
          p_struct->$tags->u_id->$parent = p_struct->$tags->$parent
          p_struct->u_id->$index = 1
          p_struct->*{1}->$tags->xmlClass = "attribute"

       case "field"

          ;Fields will always be leaves, so no need to do anything here

   endselectcase

   return 0

end ;moveReconnectData


;----------------------------------------------------------------
entry restoreReconnectData
;----------------------------------------------------------------
params
   struct p_struct      : IN
endparams

variables
   numeric i
   numeric n
endvariables

   ;Clear out tags added by xmltostruct
   p_struct->$tags->*->$parent = ""

   ;If this struct has more than 1 member then loop over them
   ;calling restoreReconnectData recursively on each
   if (p_struct->$collSize > 1)

       n = p_struct->$collSize
       for i = 1 to n
          call restoreReconnectData(p_struct{i})
       endfor

       return 0

   endif

   ;This struct only has 1 member so process it.First see what
   ;type it is, component, entity, occurrence
   p_struct->"u_type"->$tags->*->$parent = ""
   p_struct->"u_type"->$parent = p_struct->$tags

   selectcase p_struct->$tags->u_type
       case "component"

          ;This is the component node, so call this function
          ;recursively on all the entity nodes under it
          call restoreReconnectData(p_struct->*)

       case "entity"

          ;This is an entity node, so call this function
          ;recursively on all the occurrence nodes
          ;under it (If they exist)
          if (p_struct->$membercount > 0)
             call restoreReconnectData(p_struct->*)
          endif

       case "occurrence"

          ;This is an occurrence node, so just call this function
          ;recursively on all the entity or field nodes under it.
          call restoreReconnectData(p_struct->*)

          ;Move reconnect data into tags
          p_struct->u_status->$tags->*->$parent = ""
          p_struct->u_status->$parent = p_struct->$tags
          p_struct->$tags->u_status->$index = 2

          p_struct->u_crc->$tags->*->$parent = ""
          p_struct->u_crc->$parent = p_struct->$tags
          p_struct->$tags->u_crc->$index = 2

          p_struct->u_id->$tags->*->$parent = ""
          p_struct->u_id->$parent = p_struct->$tags
          p_struct->$tags->u_id->$index = 2

       case "field"

          ;Fields will always be leaves, so no need to do anything here

   endselectcase

   return 0

end ;restoreReconnectData

Okay, it's not broken against the design. But the design is broken. 🙂 

What I am looking for is a simple, repeatable way to design an interface from the application server to the client, to pass complex datasets including their modification statuses etc., preferably in less than (say 3 ) lines of code. Which works in every component, without the developer having to manually maintain lists of fields or lose data. 

So xmlsave is broken because DTD's are (almost by definition) out of date.

componenttostruct is broken because the system doesn't pass structs between the application server and the client (which it could do, using XML, JSON or something else). 

componenttostruct/structtoXXXX is broken because the tags are not included (either automatically or via a switch) in either the xml or json. 

I agree that if I faff about and write code to pass through the struct and 'de-tag' the tags, I can get it to work and that once I've written that code (which I spent all afternoon doing and now works, an example to follow) I can re-use that. 

My point is, and it's one I've made to Uli a lot, is that WE SHOULDN'T HAVE TO. I would like uniface to post the definitive method of passing data from client to server and vice versa whilst maintaining the modification statuses, and if it's xmlsave/xmlload, then I'm telling you that silently wiping my data because I forgot to include it in the DTD is broken..... 🙂 

A change of structtoXXXX/reconnectags or even just including the reconnecttags from the struct in the JSON/XML because they are there, (and why would they be there if you didn't want them?) would make it not so broken. 

It's been what, 10 years no since we got retrieve/reconnect? Isn't it time it was useful instead of the usual, hack it till it works? 

BTW componenttostruct v_struct, "ENTITYNAME" does not provide decent output for structtocomponent without hacking that about as well. Also, no simple mapping, so how do you go from FRM component subtypes to SSV component subtypes and vice versa. 

Regards, 

Iain

My versions of the tags routines. (Which are proof against the uniface adding or changing the tags names/contents. )

; $tags to struct nodes. 
entry lp_flatten_tags
	params
		struct p_struct : IN
	endparams
	variables
		numeric v_counter
		string v_name
	endvariables
	
	if(p_struct->$tags->$istags & p_struct->$tags->$membercount > 0 )
		p_struct->keep_tags = $newstruct
		for v_counter = 1 to p_struct->$tags->$membercount
			v_name = p_struct->$tags->*{v_counter}->$name
			p_struct->keep_tags->"%%v_name%%%" = p_struct->$tags->*{v_counter}
		endfor
	endif
	for v_counter = 1 to p_struct->$membercount
		if(p_struct->*{v_counter}->$name != "keep_tags")
			call lp_flatten_tags(p_struct->*{v_counter})
		endif
	endfor
	
	return(0)
end

and

; struct nodes back to $tags.
entry lp_replace_tags
	params
		struct p_struct : IN
	endparams    
	variables
		numeric v_counter
		string v_name
	endvariables
	
	if(p_struct->keep_tags->$membercount > 0)
		for v_counter = 1 to p_struct->keep_tags->$membercount
			v_name = p_struct->keep_tags->*{v_counter}->$name
			p_struct->$tags->"%%v_name%%%" = p_struct->keep_tags->*{v_counter}
		endfor
		p_struct->keep_tags->$parent = ""
	endif
	for v_counter = 1 to p_struct->$membercount
		if(p_struct->*{v_counter}->$membercount > 0)
			call lp_replace_tags(p_struct->*{v_counter})
		endif
	endfor
	return(0)
end

Oh, and while I'm here, why do I have to know the implementation layout before I can code something which just works? 

If I am running all the services in the client, without an application server, then I can pass the struct as a parameter. 

If I move the services to the application server as an implementation decision, then whilst this will compile without issue, it will fail at implementation because you can't pass a struct across the divide. 

Why doesn't the parameter handler within Uniface work out whether the struct needs to be converted, and convert it itself into something whcih doesn't lose information. I appreciate that some function would be lost, (changing the struct in one not changing it in the other) but it's light years better than just failing.... 

Still cheerful, 

Iain

Are your reading glasses broken (as well)? 😉

> Scripting Application Behavior > ProcScript Syntax > Structs > Struct Parameters

"Structs are passed by value when struct parameters are declared in a public operation, or when the byVal qualifier is used in the declaration."

"Passing Structs by value is the only way to exchange information between components that are running in different processes, but it is fundamentally different from passing Structs by reference, because a copy of the Struct is made."

And yes, you need to be aware of the consequences when running services remotely. It is of course advisable to use ByVal wherever possible, since it's faster (because just a memory pointer is passed and not a copy of the Struct).

I agree that there (still) is room for improvement how Uniface "handles" Struct's. But this does not mean that something is broken. It's just working as advertised. Maybe your expectations are broken? 😜

No hard feelings and stay cheerful. 🙂

Daniel

Here's some code for doing the sub-type mapping:

;----------------------------------------------------------------
entry doMapping
;----------------------------------------------------------------
	params
		struct p_struct      : IN
	endparams
	
	variables
		numeric i, n
	endvariables
	
	;If this struct has more than 1 member then loop over them
	;calling doMapping recursively on each
	if (p_struct->$collSize > 1)
		
		n = p_struct->$collSize
		for i = 1 to n
			if (p_struct{i}->$tags->u_type != "field")
				call doMapping(p_struct{i})
			endif
		endfor
		
		return 0
		
	endif
	
	;Now see what type it is, component, entity, occurrence
	selectcase p_struct->$tags->u_type
		case "component"
			
			;This is the component node, so call this function
			;recursively on all the entity nodes under it
			call doMapping(p_struct->*)
			
		case "entity"
			
			;This is an entity node, so call this function
			;recursively on all the occurrence nodes
			;under it (If they exist)
			if (p_struct->$membercount > 0)
				call doMapping(p_struct->*)
			endif
			if (p_struct->$name != "")
				p_struct->$name = $item(p_struct->$name, "")
			else
				p_struct->$name = $valuepart($itemnr(1, ""))
			endif
			
		case "occurrence"
			
			;This is an occurrence node, so just call this function
			;recursively on all the entity or field nodes under it.
			call doMapping(p_struct->*)
			
	endselectcase
	
	return 0
	
end ;doMapping

]] >

The required define for the mapping could look like this (this would be for a SSV component:

#define  myStructMapping = ENT1FRM.MODEL1=ENT1SSV.MODEL1·;ENT2FRM.MODEL1=ENT2SSV.MODEL1

No rocket science and done in about 15 minutes...

Hmm, I’m only used to public and partner in operations in web stuff. Not come across it in ‘normal’ operations. 

If that works, I can get rid of all this rubbish, and it turns out that is what I am looking for, and all the rest of this faff above is without merit. 

I am still surprised there isn‘t just an example somewhere of how one SHOULD do intercomponent communication, so we can work on it, maybe we should GIT an example together and crowd source some standards.... 

So, simply declaring the operation as public does not, as the documentation would suggest, allow passing byVal by default, the declaration has to be specific. No (real) issue there. 

However, in attempting to write a routine to build the mapping automatically, I have encountered a lack of information. 

$entinfo(XXX, "SUPERTYPE") returns the ultimate supertype of the component subtype. Is there possibly a hidden function (or compile-time constant even) which returns the PARENT value from ucgroup? So If I have multiple functional subtypes,  painted in the same component (as component subtypes of the functional subtype) then I can work out the functional subtype name? Otherwise using $entinfo will return me the same mapping for all (in this case 7) uses of the same table. 

I know you are manually maintaining the mapping, and I can think of a way to manually put it in the collection operations of each entity in the model, and then build that from a loop through outerentities etc, 

Or I can manually maintain information the system should just know in every component I want to use it in.... 

So, after all this mucking about, the answer (it appears) is to declare the struct parameters as byVal in the operation. There is then no need for the XML or Json transforms, or flattening the tags. 

At that point, 

componenttostruct/reconnectflags/firetriggers v_struct, "entityname" 

can pass its data to the client. 

In the client, I use the procedures below (global procs to minimise compiled code). To 'map' the component subtypes together.  The component variables are structs (for speed).

1. Setup mappings (cp_mapssvtofrm called as cp_map_ssvtofrm( "",$frmmapping$). Mapping needs to be INOUT or else it wipes it leaving the top recursion, not sure why. 

   params
   	string p_entity : IN
   	struct p_mapping : INOUT
   endparams
   
   variables
   	string v_frm_ent, v_ssv_ent, v_inners, v_sub, v_output
   endvariables
   
   if(p_entity = "")
   	v_inners = $componentinfo($instancename,"OUTERENTITIES")
   else
   	v_inners = $entinfo(p_entity,"INNER")
   endif
   forlist v_frm_ent in v_inners
   	if($scan(v_frm_ent,".") > 0)
   		v_frm_ent = v_frm_ent[1,$scan(v_frm_ent,".")-1]
   	endif
   	v_ssv_ent = v_frm_ent[1,$length(v_frm_ent)-3]
   	v_ssv_ent = $concat(v_ssv_ent,"SSV")
   	p_mapping->"%%v_SSV_ent%%%" = v_FRM_ent
   	if($entinfo(v_frm_ent,"INNER") != "")
   		call cp_map_ssvtofrm(v_frm_ent, p_mapping)
   	endif
   endfor
   return 0

2. Post load to form, change struct data before structocomponent. (This one mostly nicked from Daniel's code.) (cp_mapentstruct called as cp_mapentstruct(v_struct, $frmmapping$). Mapping doesn't need to be INOUT as the struct is only read. 

   params
   	struct p_struct      : IN
   	struct p_mapping : IN
   endparams
   
   variables
   	numeric i, n
   	string v_name
   endvariables
   
   ;If this struct has more than 1 member then loop over them
   ;calling cp_mapentstruct recursively on each
   if (p_struct->$collSize > 1)
   	
   	n = p_struct->$collSize
   	for i = 1 to n
   		if (p_struct{i}->$tags->u_type != "field")
   			call cp_mapentstruct(p_struct{i}, p_mapping)
   		endif
   	endfor
   	
   	return 0
   	
   endif
   
   ;Now see what type it is, component, entity, occurrence
   selectcase p_struct->$tags->u_type
   	case "component"
   		
   		;This is the component node, so call this function
   		;recursively on all the entity nodes under it
   		call cp_mapentstruct(p_struct->*, p_mapping)
   		
   	case "entity"
   		
   		;This is an entity node, so call this function
   		;recursively on all the occurrence nodes
   		;under it (If they exist)
   		if (p_struct->$membercount > 0)
   			call cp_mapentstruct(p_struct->*, p_mapping)
   		endif
   		v_name = p_struct->$name
   		v_name = v_name[1,$scan(v_name,".")-1]
   		if(p_mapping->"%%v_name%%%" != "")
   			p_struct->$name = $replace(p_struct->$name,1,v_name,p_mapping->"%%v_name%%%",-1)
   		endif
   		;            if (p_struct->$name != "")
   		;                p_struct->$name = $item(p_struct->$name, "")
   		;            else
   		;                p_struct->$name = $valuepart($itemnr(1, ""))
   		;            endif
   		
   	case "occurrence"
   		
   		;This is an occurrence node, so just call this function
   		;recursively on all the entity or field nodes under it.
   		call cp_mapentstruct(p_struct->*, p_mapping)
   		
   endselectcase
   
   return 0
   
   end ;cp_mapentstruct]] >

There's some faffing about in here, as OUTERENTITIES gives the entname.model, but INNER only gives the entname, and there's no way (AFAIK) to get the model from the entity name. So we have to strip the model from the outers to make them consistent, and then strip the model from the name of the struct node to compare to the mapping. But it IS self-maintaining based on the painted entities and doesn't require developer input.