Is there a way to stagger the timing of the regions starting up?

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

Probably the easiest way to do this is with startup scripts. What platform are you running on? What product version?

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

we're running on straight windows, single server, with multiple regions. What we've noticed is that when the auto start for a server re-boot executes, it appears that the regions all start at once completing for the same resources. When er manually start the application we don't see this problem. So if you've any examples of a startup script that would stagger the start up regions starting, that would be great. I'm not thoroughly convinced thaqt this is the answer, I cannot think of a more immediate solution. I've open a support ticket asking that same question but have not heard back. The other odd thing is that there are no error messages in any logs complaining about resource competition but I can't think of anything else it could be.

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

we're running on straight windows, single server, with multiple regions. What we've noticed is that when the auto start for a server re-boot executes, it appears that the regions all start at once completing for the same resources. When er manually start the application we don't see this problem. So if you've any examples of a startup script that would stagger the start up regions starting, that would be great. I'm not thoroughly convinced thaqt this is the answer, I cannot think of a more immediate solution. I've open a support ticket asking that same question but have not heard back. The other odd thing is that there are no error messages in any logs complaining about resource competition but I can't think of anything else it could be.

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

we're running on straight windows, single server, with multiple regions. What we've noticed is that when the auto start for a server re-boot executes, it appears that the regions all start at once completing for the same resources. When er manually start the application we don't see this problem. So if you've any examples of a startup script that would stagger the start up regions starting, that would be great. I'm not thoroughly convinced thaqt this is the answer, I cannot think of a more immediate solution. I've open a support ticket asking that same question but have not heard back. The other odd thing is that there are no error messages in any logs complaining about resource competition but I can't think of anything else it could be.

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

we are running windows XP , server 2003, ES 6.0.11063 WS1

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

we are running windows XP , server 2003, ES 6.0.11063 WS1

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

we are running windows XP , server 2003, ES 6.0.11063 WS1

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

I've already replied to Support internally on this topic, presumably in answer to your query, but I'll discuss it here too in case it's useful to other users.

First, a couple of comments. Avoid running ES in production on a non-Server-class version of Windows, such as XP. Non-Server Windows is deliberately crippled (for example, it has a fixed short TCP listen queue) to discourage it for this sort of use, and so trying to run large workloads under it may result in intermittent failures like clients failing to connect. In your particular case, you may see better behavior when trying to start several regions simultaneously under Server 2003 than under XP.

My second comment is that any ES issue - such as problems when a number of regions start simultaneously - should be opened with MF Customer Care as a product issue, and they should help you gather appropriate diagnostics. Looking into workarounds (like staggering server startup) is fine, but if there's a real problem here - and we have numerous customers who run several regions, and test configurations like that - it should be diagnosed and addressed. I don't know if you did open an incident for this problem, but I don't recall seeing one.


Now, on to the proposed workaround. To use a startup script for an ES region, edit the region in the MFDS GUI ("Enterprise Server Administration"). Go to the Scripts tab (under Properties). Put the text of your startup script in the Startup text area, click the Enabled checkbox, and hit OK.

When MFDS goes to start a region with a startup script, either in response to user action or automatically at system start, it writes the contents of the script to a temporary file and executes it. On Windows, it writes it out as a .cmd file; on Unix, it's executed as a regular interpreter (script) file.

A startup script needs to include the "casstart" command somewhere; that's what will actually start the region. When you run casstart manually from the command line you usually have to specify the region name, but in a startup script it should pick it up automatically (MFDS will set it in the script's environment before the script runs).

So a script could be just:

casstart

and that would basically do the same thing as having no script at all.

The next part we want is to put a delay of some sort in before the casstart command. Then we can vary that delay for each region. Windows XP doesn't come with a built-in delay command (unlike some later versions of Windows). You can get any number of free ones from Microsoft or third parties, but it's just as easy to hack one using the ping command:

rem Pause for 20 seconds
ping -n 21 127.0.0.1 > nul

rem Start the region
casstart

ping waits one second between each attempt, so if we have it ping 21 times, it will take about 20 seconds in all to run. The -n option tells it to send 21 pings, and we redirect the output so we don't have to see it (though with a startup script we probably wouldn't see it anyway).

Now we have a couple of options. We could use this script for each region, and vary the hard-coded pause time. That lets us decide how long we want between each region startup, and what order we want them starting in.

If we're feeling lazy, though, and don't want to have to set each region's startup script pause time individually, we could have the script wait for a random amount of time. For that we use a couple of other cmd features:

setlocal

rem Get a random number between 1 and 61, inclusive
set /a SleepTime=%RANDOM% % 61 1

rem Pause for one less than that many seconds (that is, 0 to 60 seconds)
ping -n %SleepTime% 127.0.0.1 > nul

rem Start the region
casstart

Then we can use the exact same script for all of our regions.

Another trick, to avoid having to edit all of the regions if we want to make a change, is to put the actual script in a .cmd file in some well-known location, e.g. c:\\es\\start-es.cmd, and then just have the ES startup scripts run that script:

call c:\\es\\start-es.cmd

Then we only ever have to change it in one place.

Note: I haven't actually tested these scripts. I believe they'll work, but if you run into problems let me know and I'll take a look.

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

Thanks Michael. That was the best description I've had yet to where I could understand it. I did have a ticket with support opened but there was a micommunication on why I needed a word doc renamed as a txt file and then posted to the ticket, so I decided to try the forum. You gave me the answers I was looking for. I already have a vbs script that tests the active ports, so basically I will take a version of that and use it to start the regions.
We use specific port numbers for our listeners so it's easy to determine which region is up or not. Issue solved.
Thanks again.

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

Thanks Michael. That was the best description I've had yet to where I could understand it. I did have a ticket with support opened but there was a micommunication on why I needed a word doc renamed as a txt file and then posted to the ticket, so I decided to try the forum. You gave me the answers I was looking for. I already have a vbs script that tests the active ports, so basically I will take a version of that and use it to start the regions.
We use specific port numbers for our listeners so it's easy to determine which region is up or not. Issue solved.
Thanks again.

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

i've written a startup script to handle all of our regions. The curious thing is that our "prod" region doesn't start first. We gave it lower port numbers figuring that was the creiteria for startups, but appearantly not. the script will aide in keeping the regions from starting (hopefully) at the same time. I'll let you know how it goes.

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

i've written a startup script to handle all of our regions. The curious thing is that our "prod" region doesn't start first. We gave it lower port numbers figuring that was the creiteria for startups, but appearantly not. the script will aide in keeping the regions from starting (hopefully) at the same time. I'll let you know how it goes.

[Migrated content. Thread originally posted on 25 February 2012]

Is there a way to stagger the timing of the regions starting up?
We've noticed a trend that when the server is bounced, the regions (3) all attempt to start at the same time.
at one of the regions will fail to start when this happens. I end up with orphaned tasks running and trying to
figure out which process/task goes to whatever IS up & running is a nightmare. What we've got are 2 cobol prgorammers turned dba's, network specialists etc over night who haven't a clue. To me the simplest solution
would be to alternate the startup of the regions. So tell me whats wrong with this thinking or direct me to
some knoweledge please. When I look for documentation I don't find anything, so we've probably got a problem there too. Any help is greatly appreciated.

#EnterpriseDeveloper

i've written a startup script to handle all of our regions. The curious thing is that our "prod" region doesn't start first. We gave it lower port numbers figuring that was the creiteria for startups, but appearantly not. the script will aide in keeping the regions from starting (hopefully) at the same time. I'll let you know how it goes.