Unable to start Batch region in Micro focus enterprise server

Microfocus enterprise server : 4.0 patch update #7 running on windows 2016 server.

Thanks. (By the way, it's "Micro Focus". Two words.)

The most common class of issues that cause this symptom ("Server Manager initialization failure") is probably permissions. On Windows that can include old processes from an earlier run that didn't shut down correctly, because Windows is finicky about releasing and reallocating named resources like files and IPC objects.

So here are some things to check:

• Look at the process list using Task Manager or Process Explorer or a similar tool for processes with names starting with "cas". If you have other regions running it can be tricky to tell them apart; here something like Process Explorer which will let you view the command line is helpful. If you find any old processes for the region you're trying to start, kill them.
• Look at the "System Directory" setting for the region in Enterprise Server Administration. Check to see:
  • Does that directory exist?
  • Do the permissions on it allow the user account you're trying to start ES with (see below) create files in it?
  • Are there files in it? If so, do they have permissions letting that account read, write, and delete them?
  • Is it on a network filesystem (e.g. on a mounted drive)? If so, there may be issues trying to start the region from Enterprise Server Administration, or from the command line if you don't have access to that filesystem.
  • If the directory exists, is there a console.log file there from your most recent failed attempt to start the region? If so, what messages are in it?
• Have permissions for system objects such as shared memory regions changed recently, perhaps due to configuring a local security policy or receiving a domain group policy update? You may want to try starting the region again, then check the Windows Security and Application event logs using the Event Viewer to see if there are any messages since you initiated region startup.

I believe the CASSI1001 message is issued quite early during region startup (a quick look at the code suggests it's only emitted if one of a few initialization routines fail), which would eliminate a number of other possible failure sources.

Regarding "the user account you're trying to start ES with": On Windows, the user account that Enterprise Server regions will run under depends on how they're started.

• If you start an Enterprise Server region from the command line using the casstart utility, it runs under the same account as the casstart command. Usually that's the account of whoever is signed on to that Windows desktop, though you can use the runas command or similar to change that.
• If you start a region from the Enterprise Server Administration web interface, it runs under the same account as the Micro Focus Directory Server service. By default, this is LOCAL_SYSTEM. I strongly discourage running regions as LOCAL_SYSTEM - that's a severe security hole - so if you prefer to start regions from the web interface I recommend configuring the Micro Focus Directory Server service to run under a different account without administrator privileges.

Let me know if any of that bears fruit. If not, we may have to do some more complex investigation.

Thank you so much for your analysis. Will get back to you incase of any queries.

Hi Mike,

We are getting continuously getting error in console log "XA Resource Manager Initialization failure" & Resource DB2P open failed messages in our environment. DB is running in Amazon Cloud. As per DBA no abnormalities found in database end . 

Please advice.

Hi Mike,

Are you referring to ODBC as the database client software which we need to verify.

I don't know what client-side software your XA resource is using. Regardless of whether it's ODBC (and if so, what ODBC driver it's using), you should check its configuration and diagnostics. ES is telling you that your XA resource module is reporting that it can't connect to the database.

Hi Mike,

Today we encountered issues in file allocation/deallocation continously. All the jobs were getting failed which didnt happen before. So as a workaround we were manually opening/closing files using casfile command. 

i am unable find any abnormal in the online region console logs. Issue didnt fix event after restarting the file share service. Need your guidance.

Hi Mike,

No worries, there was one hung transaction running for long time , by termintating it the issue got resolved.

Thanks

Hi Mike,

Is there a way to identify any file contention persists through logs or any other way as allocation/deallocation jobs keep failing intermittently.Since when we see the job log we are able to find message as file open/close failed but unable to identify the root cause. Kindly advice.

What sort of file contention? Dataset sharing among JCL jobs? Datasets used by both JCL and CICS? File locking in Fileshare? Physical file locking at the OS level?

In any case, I expect someone has put together a set of guidelines or tips for diagnosing these sorts of problems, but I wouldn't know where to find it, I'm afraid. This isn't really my area. Again, your best bet would be to open an incident with MF Customer Care. They can put you in touch with people who have dealt with production ES systems.

If I were diagnosing this sort of problem on one of my development systems, I would probably enable the appropriate tracing in the Consolidated Trace Facility (CTF) - see the product documentation for more information. But that's not always practical with production systems, or with intermittent problems on a busy system.

Hi Mike,

Thanks for your response., Even i am not sure whether its locking in CICS or within 2 JCL jobs. We couldnt observe any file locks today because no allocation / deallocation jobs failed. its really uncertain and the application support teamis blaming the microfocus admin team. They have done scheduling changes because of these failures.

Hi Mike,

After we installed microfocus enterprise server patch update 14, previously it was patch update 7(not removed) . after doing it we are having issues in online screen.  We can see the below conitnously in the console log.

Hi Mike,

What is the use of Microsoft procmon for handling file contention issues, if so how far

and also use of SCOM in Enterprise server.

Hi Mike,

How can we take cics screen usage for a program in  Microfocus? program name has been given by the requestor.

Thanks

Procmon can be useful for finding what processes are accessing a particular OS file. For Enterprise Server, and OS file might be a program object, or a compiled BMS mapset, or the "physical file" corresponding to a VSAM file, or a COBOL structured file, or a file that's opened as line-sequential, or a file opened with the COBOL library binary-file functions, and so on.

Often in Enterprise Server what's more useful is to know whether a file (OS physical file or cataloged file) is open by CICS, through a CICS FCT; and whether it's in use by any batch jobs. In those cases the best procmon can do is tell you that one or more SEP processes have the corresponding OS file open. The various Enterprise Server diagnostic and monitoring features will be more helpful.

If you are using Fileshare, either standalone or integrated in Enterprise Server, then a given file may be open by the Fileshare server process, and not by a SEP or standalone COBOL application. In that case procmon would only tell you what Fileshare server process has the file open.

In short, procmon may provide some information, but it often needs to be coordinated with other sources to be useful.

Regarding SCOM, I can only refer you to the product documentation. If you have specific questions about it someone else may be able to help. I have not used it myself.

Hi Mike,

Recently we have been facing outage where multiple DB jobs failing back to back with "DB2 open call failure", "XA interface initialization failure". But other Microfocus jobs execute fine.

In order to fix the outage immediately we used to recycle the batch region to stop these failures. After the region refresh, issue stops. 

This outage is occurring inspite of weekly maintenance activity like region recycle.

Even the odbc was rebuilt 3 months back but still we have this issue.

I'm going to refer you to my message of 27 January, above:

• This is a community forum, not a way to contact me personally.
• Please start a new thread for each new question. Don't reuse existing ones.
• For urgent problems, contact Micro Focus Customer Care and open a support incident.

Even if I worked on the XA connection feature (I don't), there's nothing I can do with a report of occasional failures. We'll need a proper diagnostic data collection demonstrating the problem, and that will have to be communicated to the appropriate development team. The way to do that is to open a support incident.

We are receiving lot of alerts in log-1.html file of the batch region stating

"CCI error 8 in send on conversation 37526 for listener Web Services and J2EE(5001) & CCITCP-0015E A CCITCP transport failure has occurred. (CCI 15, Ext 10054)

• Conversation 2675 closed by client at 172.21.153.131:54212 or failed due to broken network connection
• GkProcessConversation: Error sending or receiving data for conversation 2675 for listener "Web Services and J2EE": COMMS: communications component error

Need help.