Problem:

Product: VisiBroker 8.0.
Patches: Service Pack 3 & below
Component: Security
Platform: All

Can you explain how VisiBroker server process perform SSL handshake request from the client?

Resolution:

VisiBroker for C & Java leverages on 2 different security providers for secure connection (SSL) establishment.

VisiBroker for Java ORB

In VisiBroker for Java ORB, the SocketListener thread will establish the TCP/IP connection before spawning off a new worker thread to perform the SSL handshake.  VisiBroker for Java ORB can accept and process SSL handshake with different clients simultaneously.
This is an example of the worker thread in the midst of performing SSL handshake.
 
"VBJ ThreadPool Worker id=0 se=iiop_tp scm=ssl orb=11c8a71" daemon prio=10 tid=0x006ad4c0 nid=0x15 runnable [0xede7e000..0xede7f9f0]
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
        at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:723)
        - locked <0xf1ed6700> (a java.lang.Object)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl .java:1030)
        - locked <0xf1ed6790> (a java.lang.Object)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:678
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
        - locked <0xf1ed67c8> (a com.sun.net.ssl.internal.ssl.AppInputStream)
        at com.borland.security.csiv2.SSLConnection.read(SSLConnection.java:781)
 
If there are 3 clients connecting to the VisiBroker Java server process, there will be 3 worker threads.

VisiBroker for C ORB

In VisiBroker for C ORB, it has a SocketListener thread which performs both an establishment of the TCP/IP connection and SSL handshake.  VisiBroker for C server ORB accept a SSL connection at a time.
If a rogue client hangs during the SSL handshake, it creates a denial of service for the VisiBroker for C server process. 
VisiBroker 8.5 introduces a new property: vbroker.security.server.ssl.handshakeTimeout for both Java & C ORB to prevent the client from hogging server resources & denial of service respectively.

Steps to reproduce the issue:

1.  Compile the bank_ssl examples in /examples/vbroker/security/bank_ssl.
2.  Include this properties: vbroker.se.iiop_tp.scm.ssl.listener.port=50000 for easier reproduction with SSH.
3.  Include this properties: vbroker.security.server.ssl.handshakeTimeout=0 to simulate a hang on the SocketListener thread.
4.  Follow the steps in the example to start the SecureServer.
4.  Start the putty software, use SSH and specify the port to be 50000
5.  Start the SecureClient to invoke the SecureServer.
6.  For Java SecureClient, the request is not impacted by the hang on the SocketListener and processed by the SecureServer.
7.  For C SecureClient, the request hangs pending the release of the SocketListener thread by the SSH.
8.  Use kill -3 for Java SecureServer & pstack for C SecureServer process on Solaris to examine the threads.