Problem

  • Product Name: VisiBroker
  • Platform/OS version: All
The investigation progress for an issue is often delayed due to lack of crucial information to proceed with the next step of investigation. At times, the issue is not reproducible when the process is re-started. The run-time information is lost.

Resolution

The following table illustrates the most common scenarios our customers have encountered with their application. The suggested information to collect can assist in the progress of the issue.

Scenario
Purpose
Information to collect
Priority for Scenario
Very Slow/Hung Application at all times
Establish whether application is either slow or hung

·         Capture 3 successive thread dumps or stack at an interval of 20 seconds each

1
Understand whether the network is slow

·         Start snoop at both the sending & receiving hosts

2
Slow Intermittent  Response
Eliminate IPv4 versus IPv6 & DNS lookup configuration issue

·         /etc/hosts

·         nsswitch.conf (http://en.wikipedia.org/wiki/Nsswitch.conf)

1
Narrow the delay is due to JVM performing Garbage Collection.  (For Java process only)

·         Enable Java Garbage Collection debug

·         Enable VisiBroker debug to allow correlation of time.

2
High CPU
Understand the duration of the specific thread which are utilizing a lot of CPU

·         Use specific OS tool to capture the utilization for all the threads in the process.

·         Capture thread dump or stack.

·         Repeat both points at an interval of 20 seconds for 5 times.

1
Core Dump
Understand the threads where it cores and the no of threads during core

·         Use GDB, MDB, DBX or ADB to capture where and all threads during core

1
Memory Leak / Growing Memory
Isolate static memory initialization (C process only)

·         Run the test or Invoke the server for 10 mins and rest the process for 1 hour.

·         Observe & provide the memory utilization (real & VmRSS) every 10 mins for 1 hour.

1
Isolate and understand where the memory leak lies.  (C process only)

·         Choose a VisiBroker example which mirrors the application to run the profiling test.

·         Use either IBM Purify or Valgrind profiler

2
Isolate and understand where the memory leak lies.  (Java  process only)

·         Generate a Java heap dump

1
Unable to connect / COMM_FAILURE
Understand whether a network path exist

·         Start snoop at both the sending & receiving hosts

1
Narrow the SSL handshake (For Java process only)

·         Enable Java Security debug

2
 
The table shown below illustrates the common tool used to collect the information.  It is neither comprehensive nor attempt to cover all Operating Systems (OS) and tools.

Information to collect
C ORB
Java ORB
Thread Dump / Stack Information

·         pstack (Solaris)

·         Refer to GDB/MDB/DBX or ADB for specific instruction

·         kill -3

Snoop / Wireshark

·         dumpcap -i -f "host x.x.x.x" -a duration: -w output.pcap (Wireshark command)

 

·         startsrc -s iptrace -a "-i -s HostA -d HostB -p ./iptrace.log" (AIX)
Garbage Collection (Java only)

·         Not Applicable

·         verbose:gc

·         XX: PrintGCDetails (Oracle© JVM only)
CPU utilization

·         prstat –L (Solaris)
Memory Leak / Growing Memory

·         Refer to IBM Purify or Valgrind manual

·         vbj -J-agentlib:hprof=heap=dump -J-agentlib:hprof=format=b Server (Oracle© JVM only)
SSL Handshake

·         Not Applicable

·         javax.net.debug=ssl