Rocket OpenQM

Hi folks,

I have a performance issue with the large-scale system. I know that this discussion depends on resources, so please assume sufficient resources are available.

Let's think about a single read-only transaction performance by just 'run  <file name>'.  Assuming that elapsed time is 5 sec and CPU time is 1 sec.

Then I tried processing this query by concurrent 2 users by phantom, but each user of elapsed and CPU time was very much worse than single. Roughly, it's 10 times worse.

In reality, Is it impossible to get a benefit from parallel processing?

---- Sample basic program.

cnt = 300000

open "STOCK" to fp else abort

startTime = Time()
startTimeCpu = System(9)

for i=1 to cnt
    read rec from fp,"101" else abort
next i

endTime = Time()
endTimeCpu = System(9)

print "elapse      : ":(endTime - startTime)
print "elapse(cpu) : ":(endTimeCpu - startTimeCpu)/1000

end

-------------------------------------

------------------------------
Yukio Kozuka
Simount Corp
Bunkyo-ku JP
------------------------------

There is an overhead in starting the phantom process - so when you attempt to split work into two or more processes you're incurring some additional overhead in the starting of the phantom. Starting a phantom is the same as logging in a new user so any VOC LOGIN (or mater LOGIN in the QMSYS account) you may have is also being executed before it gets to do the work you have asked it.

Now - it is possible to get dramatically improved performance by using parallel phantoms - here's how we do it at a high level:

We have a single manager phantom that runs at system start up, and 5 worker phantoms (this is configurable). We then submit large processing requests to the manager phantom and it splits the work load across the workers. As work is completed it gathers the results.

This way we can have up to 5 workers handling a work request. Our process for this is a little more complex than I've described in that we can have fixed "workers" that start and never die (unless told to), and overflow phantoms that can be started and will die when finished with their contribution. When you submit work you can specify how many threads to use and we can specify how long a worker should stay alive without work before ending. This is highly configurable, and dependent on available licenses for phantoms.

I hope this helps you out.

We currently leverage file i/o for the phantoms to "talk" to each other, but we are also experimenting with sockets where each phantom can "chat" with the manager. This consumes more licenses as the phantoms are interactive, but it's blazingly quick, and reduces file I/O significantly.

Given the powerful object capability of QM and the built in Web API we stuck a vue S web page on top of this that enables us to monitor the phantoms in action, interact with the master to start / stop workers, and gather stats as to usage etc. It's pretty cool.

Hope this helps.

Justin

Hi folks,

I have a performance issue with the large-scale system. I know that this discussion depends on resources, so please assume sufficient resources are available.

Let's think about a single read-only transaction performance by just 'run  <file name>'.  Assuming that elapsed time is 5 sec and CPU time is 1 sec.

Then I tried processing this query by concurrent 2 users by phantom, but each user of elapsed and CPU time was very much worse than single. Roughly, it's 10 times worse.

In reality, Is it impossible to get a benefit from parallel processing?

---- Sample basic program.

cnt = 300000

open "STOCK" to fp else abort

startTime = Time()
startTimeCpu = System(9)

for i=1 to cnt
    read rec from fp,"101" else abort
next i

endTime = Time()
endTimeCpu = System(9)

print "elapse      : ":(endTime - startTime)
print "elapse(cpu) : ":(endTimeCpu - startTimeCpu)/1000

end

-------------------------------------

------------------------------
Yukio Kozuka
Simount Corp
Bunkyo-ku JP
------------------------------