Extra bytes of data added to payload output stream causes huge delay

Problem:

• Product Name: VisiBroker 
• Product Version: All
• Product Component: Interceptors
• Platform/OS Version: All

Extra bytes of data added to payload output stream causes huge delay

Resolution:

When extra bytes of data are added to the payload output streams passed to request interceptors, the performance of GIOP packet output is severely impaired, in some cases by a factor of 80 times. A simple test case shows that making 100 requests would normally take 700 ms, but with a lengthened payload it takes 22000 ms.

For instance, supposing an encryption algorithm is used to encrypt the payload (i.e. marshaled arguments and returned values) of CORBA method invocations in the implementation of client and server request interceptors. The encryption of payload may causes the payload length to grow. As a result, the performance drops significantly.

Analysis has shown that in the case with lengthened payload, ORB will send the requests in 2 chunks: 1 chunk with the original length of the payload, and one chunk with the added length.

Furthermore, it is found that by adding these parameters on **both** the client and server, the delay went away:

vbroker.ce.iiop.connection.tcpNoDelay=true
vbroker.se.iiop_tp.scm.iiop_tp.connection.tcpNoDelay=true

The test result is as follows:

1. Case 1: without tcpNoDelay param:
   lengthened payload -> total time 22000 ms
   normal payload -> total time 700 ms
   
   
2. Case 2: with tcpNoDelay param:
   lengthened payload -> total time 703 ms
   normal payload -> total time 718 ms

This seen behavior can be explained that the 2nd chunk of 1 byte was delayed by the TCP layer in order to fill the TCP payload before sending it. OptimizeIt profiling of the client-side also showed that bulk of the difference in the CPU time spent between the Lengthened vs. Non-Lengthened scenario is in:

java.net.SocketInputStream.read() (21481 ms vs 915 ms)
called by
com.inprise.vbroker.IIOP.Connection.read()
called by
com.inprise.vbroker.GIOP.InputStream.readFromConnection()
...

This signifies that the delay is in the TCP-level sending/receiving phase.

In the case where a system is deployed with Gatekeeper and using HIOP, since Gatekeeper should group the different chunks of one message into one chunk before sending it to server/ or client, there is no need to set those 2 tcpNoDelay properties on the Gatekeeper, as long as the client and server have these 2 tcpNoDelay properties.

In HIOP communication, the payload is wrapped in HTTP post messages, and that portion of code is not VisiBroker, but rather part of JVM. Therefore there is no VisiBroker option that can control the behavior of the portion of the code that does HIOP sending.

By instrumenting a sniffer and checking the IP packets between the tunneling applet and the gatekeeper for both modified and un-modified payload scenarios, One will not see any difference in the way the IP packets were sent. But doing the same for a IIOP client and server shows that there is an extra packet sent from client to server, and vice versa. This is the case even though in both HIOP and IIOP client's debug outputs show an extra "writing byte ...." occurred. This confirms the fact that HIOP does not exhibit the same TCP/IP behavior as IIOP. Therefore there is no need for this tcpNoDelay property.

Author: Henry Hwang