Product: OpenFusion TAO
Version: ALL

Description:

Running out of file descriptors when using -ORBWaitStrategy rw

We are using the –ORBWaitStrategy rw option so that threads waiting for replies block until they get a reply. We have set the following configuration options:

static Client_Strategy_Factory "-ORBWaitStrategy rw -ORBTransportMuxStrategy exclusive -ORBConnectStrategy blocked"

static Resource_Factory "-ORBFlushingStrategy blocking"

When we use these above settings we see that the number of file descriptors used by our server process grows until it reaches the OS max (1024). The server will then not accept any more connections or make any more out calls.

Resolution:

When the 'RW' client connection handler policy is supplied you effectively lose the ability to detect that a client has gone until you try and read from or write to a connection. This means that the handle to a client that is no longer there will not be removed. (By default TAO will cache all connections forever).

This is not usually an issue for pure clients as their lifetime is usually less than the server lifetime. A server making call backs to short lived clients will eventually fill up its cache with connections that are no longer attached to clients.

You can manage the connection cache size by using the -ORBConnectionCacheMax to keep the number of handles in the cache down below the level where it causes problems. This should be used in conjunction with –ORBConnectionCachePurgePercentage.

When the ORBConnectionCacheMax value is reached then the next time a connection is made the ORBConnectionCachePurgePercentage will act to close the percentage of connections that it is set to.

If you are using TAO 1.4.1.1 or later you can also use the MT_NOUPCALL policy as an alternative to the "-ORBWaitStrategy rw" option. This option uses a client connection handler that participates in the leader-follower model. The leader follower model basically allows multiple threads to share a single reactor. One thread acts as the leader and runs the reactors event loop. The other threads in the pool are the followers. When an event occurs, such as a incoming client connection, the leader thread acts to process it and one of the follower threads becomes the new leader. When it finishes processing the event it rejoins the thread pool as a follower thread.

The MT_NOUPCALL policy does not allow the handling of nested upcalls within the waiting thread. With this strategy it is possible to run out of threads in the thread pool. More information on this options is available in the HTML documentation supplied with the TAO distribution.