Problem:

COMM_FAILURE and TCP Reset after connectionMaxIdle kicks in

Resolution:

• Product Name: Visibroker for Java
• Product Version: All
• Product Component: orb
• Platform/OS version: All
• JDK: Not Applicable
• Patch Level: All

Problem

The client is making a corba call to the object hosting in the server. The client establishes a TCP connection to the Server. The server has configured the orb to close the connection once the connectionMaxIdle setting is reached. The connection is left idle without any invocation.

Once the connectionMaxIdle setting has kicks in, the server orb will send the GIOP CloseConnection message to the client. Subsequently, the server TCP/IP stack will follows with a TCP FIN to close the connection. The client TCP/IP stack has received the TCP FIN and sends a TCP FIN ACK to terminate the TCP connection. The GIOP closeConnection message is delivered to the buffer in the client GIOP layer.

Visibroker orb adopts the approach of reading from the socket after writing to the socket. This approach is optimized for most of the scenarios used in networking.

When the client thread wakes up and proceed with the next method call, the client orb will write to the OutputStream without realizing the TCP connection is broken. Next, the client thread will read from the socket. After reading the GIOP CloseConnection message, the client realizes it is out of synch and closes the connection. A TCP reset is observed in the ethereal. An internal COMM_FAILURE is thrown within the client orb and the message will be re-delivered if the default VB_TRANSPARENT rebind policy is used. The client experiences minimal delay and the message will be delivered once the TCP socket is re-established.

Reference to Section 15.7.1 of CORBA 2.6 Specification

After receiving a CloseConnection message, an ORB must close the TCP/IP connection. After sending a CloseConnection, an ORB may close the TCP/IP connection immediately, or may delay closing the connection until it receives an indication that the other side has closed the connection. For maximum interoperability with ORBs using TCP implementations that do not properly implement orderly shutdown, an ORB may wish to only shutdown the sending side of the connection, and then read any incoming data until it receives an indication that the other side has also shutdown, at which point the TCP connection can be closed completely. Given TCP/IP’s flow control mechanism, it is possible to create deadlock situations between clients and servers if both sides of a connection send large amounts of data on a connection (or two different connections between the same processes) and do not read incoming data. Both processes may block on write operations, and never resume. It is the responsibility of both clients and servers to avoid creating deadlock by reading incoming messages and avoiding blocking when writing messages, by providing separate threads for reading and writing, or any other workable approach. ORBs are free to adopt any desired implementation strategy, but should provide robust behavior.

Solution

You can mitigate the effect by tuning the connectionMaxIdle time to a sufficient long period of time to meet most of your application demands on the TCP/IP stack. If the numbers of client connecting to the server is small and predictable, you can disable the connectionMaxIdle property by setting it to 0 and fix the listener port too.