Status: Needs Clarification
During the compile/link process of any given program, a requirement has come up to extract the list of all copybooks (component name and type) from the ChangeMan ZMF package. Looking at #VARLIST, it was found that the PDSTGTBL ISPF table is available, which is the Package Driven Stage Table.
When one does a ")DOT PDSTGTBL" to find all copybooks (using ")SEL &STGLIKE EQ C"), it does destruction of values across various variables used in the compile/link skeletons. For example, COBOL program ABC123.SRC may be staged with certain user options, etc. When ISPF file tailoring begins, values are set as follows:
- &CMPNAME is "ABC123"
- &CMPTYPE is "SRC"
- &LNGNAME is "COBOL"
- etc.
The issue is that once the whole PDSTGTBL ISPF table is looped through via ")DOT PDSTGTBL", the various variables (&CMPNAME, &CMPNAME, &LNGNAME, &PROC, &USEROP1, &CUSR011, etc.) are set to the last entry / component within PDSTGTBL.
In order to effectively to perform a ")DOT PDSTGTBL" without worry about ISPF variable value destruction, one must save the values of all variables used in staging ISPF file tailoring that the PDSTGTBL table references, then use the PDSTGTBL ISPF table, and then restore all the variables with the values associated with the target component being staged. Since PDSTGTBL contains 59 fields, there around 59 variables that must have their values saved before PDSTGTBL is used, and after PDSTGTBL is used these variables will need to be restored.
As a specific example, we have a package that contains the following components:
STAGE: UAS 000123 Components Row 1 to 2 of 2
Command ===> Scroll ===> PAGE
Name + Type Status Changed Procname User Request
TEST00 SRC ACTIVE 20240814 194657 CMNCOBEP STD50
TEST00C CPY ACTIVE 20240515 141949 STD50
******************************* Bottom of data ********************************
When TEST00.SRC is staged, here is part of the CMNWRITE job step:
//*)IM CMN$$WRT
//WRITE EXEC PGM=CMNWRITE, *** PARSE/EXPAND COMPONENT TEST00
// COND=(4,LT),
// PARM=('SUBSYS=I,USER=STD50',
// '')
...
CMP=TEST00.SRC
LNG=COBOL
PKG=UAS 000123
Some time after the CMN$$WRT skeleton, the PDSTGTBL ISPF table is processed via a "Do Table."
In the FAILURE job step, one can see that a the copybook component (TEST00C.CPY) is referenced instead of the target component that was staged (TEST00.SRC):
//FAILURE EXEC PGM=CMNBATCH, *** NOTIFY USER PROCESS HAS FAILED
// COND=(EVEN,(0,EQ,CHKCOND)),
// PARM='SUBSYS=I,USER=STD50'
...
//SYSIN DD *
UAS #Ä LIB=CPY
UAS #Ä CNM=TEST00C
UAS #Ä FUN=90,TID=STD50
Rather, in the FAILURE job step, it should be "LIB=SRC" and "CNM=TEST00".
In any case, the above shows an example of how the PDSTGTBL ISPF table will destruct the values of &CMPNAME, &CMPTYPE, and many others.
As implied, it is true that it's op to the ZMF admin person customizing the ChangeMan ZMF product to ensure that these ISPF variables are saved and restored; however, there are quite a few variables that need to be saved and restored. Additionally, what if more same-named variables are added to the staging file tailoring process and the PDSTGTBL ISPF table in a future ZMF version? It would make an ZMF upgrade a little more error-prone to ensure that these new ISPF variables are saved and restored.
It would seem that a simple (and sensible) solution would be to have the PDSTGTBL ISPF table contain the target component being staged to be the last entry in the PDSTGTBL table. This way, even though processing various rows of PDSTGTBL will cause destruction of various &CMPNAME, &CMPTYPE, etc. variables, when the last entry within PDSTGTBL is processed, it will effectively restore the ISPF file tailoring variables associated with the target component being staged. Thus, there would be no need to save and restore a multitude of ISPF variables to ensure that staging ISPF file tailoring continues to generate JCL reliably.
-- Ron Palaniuk
#ZMF#ReduceCustomizations#BuildHi Ron,
Can you give me some detail on how you want to use these copybook names ?
Can you not extract them using a service invocation or do they have to be present as ISPF variables during file tailoring ?
Cheers
Hi Ron,
Can you give me some detail on how you want to use these copybook names ?
Can you not extract them using a service invocation or do they have to be present as ISPF variables during file tailoring ?
Cheers
Hi Steve,
I apologize for the delay in responding to you.
So, the use case of capturing copybook names is that we are integrating ChangeMan ZMF with the Checkmarx security scanning tool. This tool requires a copy of the COBOL program as well as any copybooks that are being changed in the ZMF package. All of these components are shipped to Checkmarx using a REST API call, and then a result is returned by Checkmarx to either block the build or allow the build to continue to create a load module.
With respect to extracting the copybooks from a service invocation, I'm trying to get a better idea of what kind of service you were thinking of. Are you suggesting something like creating an XML service within the compile/link job that would extract the list of copybooks from the package, and from that list to build appropriate control cards (e.g. using the SORT utility) to bundle-up all the copybooks to be sent to Checkmarx? The other option could be to use the output from the CMNWRITE report to determine what copybook components and associated libraries within which they reside, and generate the "bundle" of copybooks to be used during the Checkmarx scan.
Though the above other ideas could work, and thereby avoid the need to use the PDSTGTBL ISPF table, I will say that using the PDSTGTBL during the compile/link file tailoring process still needs to be used very carefully because it is quite possible that component history would be corrupted. For example, the first program in the package might be a CICS program, and the last program in the package might be a batch program. If the first program is staged, the end result could result in the first program having its component history incorrectly saved as a batch program.
-- Ron Palaniuk
------------------------------
Ron Palaniuk
ISD Support Lead IV
Navy Federal Credit Union
Vienna VA US
------------------------------
Hi Steve,
I apologize for the delay in responding to you.
So, the use case of capturing copybook names is that we are integrating ChangeMan ZMF with the Checkmarx security scanning tool. This tool requires a copy of the COBOL program as well as any copybooks that are being changed in the ZMF package. All of these components are shipped to Checkmarx using a REST API call, and then a result is returned by Checkmarx to either block the build or allow the build to continue to create a load module.
With respect to extracting the copybooks from a service invocation, I'm trying to get a better idea of what kind of service you were thinking of. Are you suggesting something like creating an XML service within the compile/link job that would extract the list of copybooks from the package, and from that list to build appropriate control cards (e.g. using the SORT utility) to bundle-up all the copybooks to be sent to Checkmarx? The other option could be to use the output from the CMNWRITE report to determine what copybook components and associated libraries within which they reside, and generate the "bundle" of copybooks to be used during the Checkmarx scan.
Though the above other ideas could work, and thereby avoid the need to use the PDSTGTBL ISPF table, I will say that using the PDSTGTBL during the compile/link file tailoring process still needs to be used very carefully because it is quite possible that component history would be corrupted. For example, the first program in the package might be a CICS program, and the last program in the package might be a batch program. If the first program is staged, the end result could result in the first program having its component history incorrectly saved as a batch program.
-- Ron Palaniuk
------------------------------
Ron Palaniuk
ISD Support Lead IV
Navy Federal Credit Union
Vienna VA US
------------------------------
Ron, Do you only need the copybooks related to the program or all copybooks in the change package? You can get the list of copybooks in the package with an XML call the issue is which ones apply to a particular program. It gets more complicated with participating packages or ERO. You can get the list of component types in the package and use the files service XML calls to get the contents of copybook libraries. It is not straight forward but it could be done. We have done similar things for customers using REXX in the past, i would assume it could be done with webservices as well.
------------------------------
Tom Mavor
President
Self Registered
Land O Lakes FL US
------------------------------
Ron, Do you only need the copybooks related to the program or all copybooks in the change package? You can get the list of copybooks in the package with an XML call the issue is which ones apply to a particular program. It gets more complicated with participating packages or ERO. You can get the list of component types in the package and use the files service XML calls to get the contents of copybook libraries. It is not straight forward but it could be done. We have done similar things for customers using REXX in the past, i would assume it could be done with webservices as well.
------------------------------
Tom Mavor
President
Self Registered
Land O Lakes FL US
------------------------------
Hi Tom,
Thanks for your response and feedback. At this point, we are just getting all copybooks that reside in the package to keep things simpler as the initial iteration for setting up of the Checkmarx scanning process to be synchronous during the build process for build blocking / security gating purposes. (We already have a fair bit on our plate and doing a fair bit of "trailblazing" from our perspective in that we have ZMF using Co:Z Batch (as opposed to BPXBATCH) to prep everything that is required in USS (e.g. directories, files, Python virtual environment, etc.), call a Python script that does a REST API services call to Checkmarx, and then receive back the result of the scan back to ZMF.)
But yes, the better approach--which will be part of phase 2 of this effort--will be to only limit it to the copybooks that a given program is using, take into consideration participating packages, and even rescanning copybooks not being changed that are in baseline. (Doing rescanning is better done because new "scan rules" for vulnerabilities can come up in the future. Thus, a scan of code that yields no security vulnerability issues today could yield vulnerabilities tomorrow.)
True, an XML services call may be done during ISPF file tailoring to retrieve a list of copybooks associated with a given program or the copybooks in a given package; however, that is a fair bit of overhead especially when the list of copybooks in the package already exists and can be easily found in the PDSTGTBL ISPF table, which is already built by ZMF prior to invoking ISPF file tailoring.
However, this particular use-case of PDSTGTBL aside, the bigger "issue" or "concern" that I'm bringing-up is the usefulness of the PDSTGTBL ISPF table--especially when doing a Do-Table of PDSTGTBL results in destruction of almost all stand-alone, single-entry ISPF variables that are made available in ISPF just prior to invocation of ISPF file tailoring.
For example, if I need to use the PDSTGTBL, I need to save most of the ISPF variable names defined within PDSTGTBL before doing a Do-Table against PDSTGTBL, and then restore most of them. In my use case example, I've saving all 59 of them (I realize that there is a handful that I don't need to save):
)SET SVCOMP = &CMPNAME
)SET SVTYPE = &CMPTYPE
)SET SVREQ = &REQUEST
)SET SVMEMB = &MEMB
)SET SVTSOID = &TSOID
)SET SVSTGDTE = &STGDTE
)SET SVSTGTME = &STGTME
)SET SVLIBORG = &LIBORG
)SET SVDSN1 = &DSN1
)SET SVSTATUS = &STATUS
)SET SVLANG = &LNGNAME
)SET SVPROC = &PROC
)SET SVSTGLK = &STGLIKE
)SET SVTLODLT = &TLODLTP
)SET SVCMPIND = &CMPIND1
)SET SVCHKBAT = &CHKCBAT
)SET SVUSROP1 = &USEROP1
)SET SVUSROP2 = &USEROP2
)SET SVUTLIND = &UTILIND
)SET SVDPROC = &DPROC
)SET SVDB2PC = &DB2PC
)SET SVTSLDSC = &TSTLDSC
)SET SVUSR011 = &CUSR011
)SET SVUSR012 = &CUSR012
)SET SVUSR013 = &CUSR013
)SET SVUSR014 = &CUSR014
)SET SVUSR015 = &CUSR015
)SET SVUSR021 = &CUSR021
)SET SVUSR022 = &CUSR022
)SET SVUSR023 = &CUSR023
)SET SVUSR031 = &CUSR031
)SET SVUSR032 = &CUSR032
)SET SVUSR033 = &CUSR033
)SET SVUSR041 = &CUSR041
)SET SVUSR042 = &CUSR042
)SET SVUSR043 = &CUSR043
)SET SVUSR081 = &CUSR081
)SET SVUSR082 = &CUSR082
)SET SVUSR083 = &CUSR083
)SET SVUSR084 = &CUSR084
)SET SVUSR085 = &CUSR085
)SET SVUSR101 = &CUSR101
)SET SVUSR102 = &CUSR102
)SET SVUSR161 = &CUSR161
)SET SVUSR162 = &CUSR162
)SET SVUSR341 = &CUSR341
)SET SVUSR342 = &CUSR342
)SET SVUSR441 = &CUSR441
)SET SVUSR442 = &CUSR442
)SET SVUSR641 = &CUSR641
)SET SVUSR642 = &CUSR642
)SET SVUSR643 = &CUSR643
)SET SVUSR644 = &CUSR644
)SET SVUSR645 = &CUSR645
)SET SVUSR721 = &CUSR721
)SET SVUSR722 = &CUSR722
)SET SVUSR723 = &CUSR723
)SET SVUSR724 = &CUSR724
)SET SVUSR725 = &CUSR725
)CM
)DOT PDSTGTBL
)SEL &STGLIKE EQ C
)SETF USSCMD = &STR(cp -v -U -O c=ISO8859-1)
)SETF USSCMD = &STR(&USSCMD "//'&STGLIB..&CMPTYPE.(&CMPNAME.)'")
)SETF USSCMD = &STR(&USSCMD "&SRCDIR/&CMPNAME..&CMPTYPE..cbl")
)IM COZSTDIN
)ENDSEL &STGLIKE EQ C
)ENDDOT PDSTGTBL
)CM
)SET CMPNAME = &SVCOMP
)SET CMPTYPE = &SVTYPE
)SET REQUEST = &SVREQ
)SET MEMB = &SVMEMB
)SET TSOID = &SVTSOID
)SET STGDTE = &SVSTGDTE
)SET STGTME = &SVSTGTME
)SET LIBORG = &SVLIBORG
)SET DSN1 = &SVDSN1
)SET STATUS = &SVSTATUS
)SET LNGNAME = &SVLANG
)SET PROC = &SVPROC
)SET STGLIKE = &SVSTGLK
)SET TLODLTP = &SVTLODLT
)SET CMPIND1 = &SVCMPIND
)SET CHKCBAT = &SVCHKBAT
)SET USEROP1 = &SVUSROP1
)SET USEROP2 = &SVUSROP2
)SET UTILIND = &SVUTLIND
)SET DPROC = &SVDPROC
)SET DB2PC = &SVDB2PC
)SET TSTLDSC = &SVTSLDSC
)SET CUSR011 = &SVUSR011
)SET CUSR012 = &SVUSR012
)SET CUSR013 = &SVUSR013
)SET CUSR014 = &SVUSR014
)SET CUSR015 = &SVUSR015
)SET CUSR021 = &SVUSR021
)SET CUSR022 = &SVUSR022
)SET CUSR023 = &SVUSR023
)SET CUSR031 = &SVUSR031
)SET CUSR032 = &SVUSR032
)SET CUSR033 = &SVUSR033
)SET CUSR041 = &SVUSR041
)SET CUSR042 = &SVUSR042
)SET CUSR043 = &SVUSR043
)SET CUSR081 = &SVUSR081
)SET CUSR082 = &SVUSR082
)SET CUSR083 = &SVUSR083
)SET CUSR084 = &SVUSR084
)SET CUSR085 = &SVUSR085
)SET CUSR101 = &SVUSR101
)SET CUSR102 = &SVUSR102
)SET CUSR161 = &SVUSR161
)SET CUSR162 = &SVUSR162
)SET CUSR341 = &SVUSR341
)SET CUSR342 = &SVUSR342
)SET CUSR441 = &SVUSR441
)SET CUSR442 = &SVUSR442
)SET CUSR641 = &SVUSR641
)SET CUSR642 = &SVUSR642
)SET CUSR643 = &SVUSR643
)SET CUSR644 = &SVUSR644
)SET CUSR645 = &SVUSR645
)SET CUSR721 = &SVUSR721
)SET CUSR722 = &SVUSR722
)SET CUSR723 = &SVUSR723
)SET CUSR724 = &SVUSR724
)SET CUSR725 = &SVUSR725
If I don't do the above save-and-restore of ISPF variables, then it would result in getting component history from the last package component stored internally within ZMF and applying it to the program being changed. So, if the last program in the package stored in the master files is a batch Db2 program, and the program that I'm staging is a CICS non-Db2 program, then after the build process runs it will result in the metadata of the CICS non-Db2 program being staged changing to be a batch Db2 program, which is incorrect.
Yes, the above save-and-restore will avoid this issue, but one needs to keep an eye on this--especially with ZMF upgrades. If, for example, a new ISPF variable becomes available for component history metadata, and the same variable name is used as a single-entry variable as well as the variable within the PDSTGTBL ISPF table, and the appropriate "save-and-restore" isn't done, then you will get into a situation where bits of component history/metadata is "leaking" from one component into another. I realize that one should find this issue with testing, but we are talking about looking at potentially "obscure" metadata that, on the surface, won't seem like an issue or even interpreted as an issue. However, over time, this slow corruption of component history would be detected, and by then you are dealing a large effort in finding a way to correct all the incorrect bits of metadata over many components over variable packages that have accumulated over time.
My suggestion was to put the component being staged as the last entry within PDSTGTBL. This way, even after we loop through the PDSTGTBL ISPF table, this would result in the single-entry ISPF variables being restored automatically without having to put together--and maintain--the save-and-restore-the-variables code. I look at this as not only keeping customizations simpler, but also making ZMF upgrades less error prone.
I realize that this also happens within other ISPF tables. See LTYPETBL in CMN$$VAR, where &STGLIKE and &STGTLLT is being saved and then finally being restored at the end of CMN$$VAR. I suppose that one could argue the same should be done within LTYPETBL, where the last entry within LTYPETBL is associated with the component type being staged. However, there are far few variables to save-and-restore within LTYPETBL compared to PDSTGTBL. I guess in an ideal world, all ISPF tables that have share the same ISPF variable names as any single entry ISPF variables within a given ZMF function should result in effectively restoring the single-entry ISPF variable values back to what they should be by putting the last entry in each ISPF table to make that restoration happen. However, I could see this as being a rather large effort, and not even possible possible. In some cases there is no way to even make that happen because when you stage something, it may not relate to what the contents are within an ISPF table. (For example, the RPMLIBTB ISPF table for compile/link also uses &STGLIKE (which is a single-entry variable) and would destroy the contents of the single-entry &STGLIKE value. But since it is possible that the component type being staged under (e.g. "SRC") is not defined in a promotion level and thus not in the RPMLIBTB, then there is no way to "automatically restore" &STGLIKE without having to resort to the tried-tested-and-true save-and-restore method. Besides, I believe that certain ISPF tables have their records in a certain order for a reason, but we would not want to make such changes to the order because that would cause other issues.)
Anyway, just my thoughts...
-- Ron Palaniuk
------------------------------
Ron Palaniuk
ISD Support Lead IV
Navy Federal Credit Union
Vienna VA US
------------------------------
