Product: OpenFusion Notification Service

Version: ALL

Description:

Why use Custom Filters?

Resolution:

This can occur if the number of events produced exceeds the maximum rate that the events can be consumed, then the memory usage will increase until a limit is reached and the service runs out of memory.

This occurs because events (as with all Java objects) require memory. Once a channel has passed an event to all proxy suppliers, and all the proxy suppliers have delivered the event, the event is no longer required and will eventually be garbage collected. At any one time there is only one "copy" of the event; the proxy suppliers all reference this one instance. If the EventReliablity is Persistent then the events will also be written to a persistent store until they are delivered. If a consumer is unavailable for any reason then the Notification Service will receive an exception. If the proxy supplier has ConnectionReliability set to Persistent then the proxy will retry delivery. It will still retain a reference to the event, so even if all the other proxies have finished with it then that event remains in memory until it can be delivered. If a large number of consumers are unavailable or a large number of events are being sent then the memory can be used up rapidly.

You can configure the Notification Service using QoS properties to prevent this from occurring.

Firstly you need to increase the memory available to the Notification Service process. To make the memory available to the process, the -Xmx flag should be added to the JVM Flags property for the NotificationService. For example, -Xmx150m will set the maximum java heap size to 150Mb. The maximum and initial JVM heap sizes can be set using the following flags. -Xms set initial Java heap size -Xmx set maximum Java heap size For example, -Xmx150m will set the maximum java heap size to 150Mb. These should be added to the JVM Flags option found on the Java Tag of the Notification Service in the Adminstration Manager. When you are allocating memory size to the Notification Service you need to take into consideration the number of events that you will be sending and receiving. Each event will occupy a certain amount of memory space. If you know roughly how much memory each event occupies and the event rate you can work out how much memory you will need. You need to make sure that you allocate sufficient memory to the JVM to deal with these events.

Secondly you need to use QoS properties to prevent the service from running out of memory during times of event burst. The setting you choose to use will depend on your architecture. The properties you can use are as follows: The QoS setting MaxQueueLength and MaxEventsPerConsumer can be used to control the memory usage of the Notification Service.

MaxQueueLength defines the maximum size of the master queue for an event channel. The master queue is where the Notification Service stores all events before it forwards them to any proxies. This value needs to be set to a greater value that the MaxEventsPerConsumer QoS property. This prevents a consumer that is slow to consume events or a consumer that has been suspended for an extended amount of time causing the events to be rejected from the master queue. The maximum size of the master queue is the accumulative size of all the proxy queues. Normally the size of the master queue is smaller than the accumulative size of all proxy queues because there is typically an overlap in the events received by different consumers. The RejectNewEvents QoS can be set to define what happens to events when the queue is full.

MaxEventsPerConsumer defines the maximum number of events that a proxy will queue on behalf of the connected consumer. This setting can be set to prevent a single consumer from exhausting the master queue. The default value is zero which means there is no queue limit size. This property is typically used when the incoming event rate exceeds the capabilities of the Notification Service for extended amounts of time. It is also used when the proxy queue represents periodic updates that will be available in the shape of a new event at a later time. Limiting the queue size also limits the resources required by the Notification Service.

RejectNewEvents indicates whether events should be rejected or discarded, according to the DiscardPolicy setting, when the maximum queue size is exhausted i. e when the MaxQueueLength is reached.
This can be set to

• TRUE which causes new events received by the event channel to be rejected when the maximum queue size is reached. The supplier will receive an IMP_LIMIT system exception when it attempts to deliver events to the channel. No pull operations will be issued by the event channel on the pull suppliers until the queue size has been reduced. This guarantees the Notification Service will not drop any events because it refuses to accept any more.
• FALSE which causes new events received by the event channel to be discarded when the MaxQueueLength is reached.

The DiscardPolicy QoS property can be set to tell the Notification Service how to deal with discarded events. Suppliers can still deliver events to the event channel but some may be discarded. DiscardPolicy indicates the order in which events are discarded from event queues. This is only used if the RejectNewEvents QoS is set to FALSE. The default setting for this property is AnyOrder which means that events are discarded in the same order as the OrderPolicy QoS.

The other settings are PriorityOrder, DeadlineOrder, FifiOrder and LifoOrder. Events are discarded from the master queue when the value of MaxQueueLength is reached. An event that is discarded from the master queue will never reach a consumer and it will appear to the consumer that the event was not delivered to the event channel. Events are discarded from proxy queues when the value of the MaxEventsPerConsumer QoS is reached.

Alternatively you can to manage the queue based on the current memory usage of the machine on which the Notification Service is running as well rather than on the current number of events in the queue. The QoS properties that allow you to do this are MaxMemoryUsage and MaxMemoryUsagePolicy.

MaxMemoryUsage - This is used to set a value for the maximum amount of memory (in bytes) that the Notification Service should attempt not to exceed. The behaviour of the Notification Service when this limit is reached depends on the value of the property MaxMemoryUsagePolicy.

MaxMemoryUsagePolicy - This is used to control the behaviour of the Notification Service when the value of MaxMemoryUsage is exceeded. It may hold three different values:

• Purge Events - if this value is set, then the MaxMemoryUsage is treated as a soft limit. Whenever an event is received that takes the memory usage above MaxMemoryUsage then that event will be added to the internal queue of the appropriate event channel as normal. Then, in a manner that mirrors discard behaviour, the event at the back of the queue will have its data purged from memory. If the event has best-effort delivery semantics then it is effectively discarded and all used memory should be available for recovery by the garbage collector. However in the case of a persistent event a placeholder will remain in memory so the data can be reloaded from the persistent store when required. Hence in the case of a persistent event not all of the used memory is freed so total memory usage will continue to increase. The rate of increase will be greatly reduced however, making this an appropriate policy for dealing with bursts of event delivery.
• DiscardEvents - if this value is set, then MaxMemoryUsage is treated as a fairly hard limit. Whenever an event is received that takes memory usage above MaxMemoryUsage, an event is discarded according to the current discard policy. Note that since events vary in size, the memory usage may still grow since the new event may be larger than that which is discarded
• RejectEvents - if this value is set, then MaxMemoryUsage is treated as a hard limit. Whenever an event is received that takes memory usage above MaxMemoryUsage, an org.omg.CORBA.IMP_LIMIT exception is thrown. Please note that if you decide to use these settings then you need to set the MaxMemoryUsage value at a value lower than the total memory that you have given to the JVM.