In a cluster with more than one server node, the typical JMS usage involves internal cluster communication with the central JMS server node.
The central JMS node can crash or become unavailable for a number of reasons - OutOfMemoryError, HotSpot error, hardware failure, network failure. In case the central node crashes or connection to it is lost, the JMS resources on other server nodes become unavailable and the applications are notified (according to the JMS specification) through all registered standard listeners (javax.jms.ExceptionListener). Upon receipt of such event, applications should normally recreate the JMS resources (connection, session, and so on) and the communication is automatically redirected to another running server node that becomes the central JMS (server) node. The old connections and resources are invalidated and all attempts to use them will result in the exceptions specified in the symptom section above.
Since the failover of the JMS central node does not happen immediately, applications that use JMS resources should perform the recovery in a cycle waiting for a certain time before each attempt to reestablish the JMS connections. There is a short period of time before the reinitialization, during which the resources are not available.
If an application does not wait for a certain time (for example, 1 second) before the subsequent reconnection attempt, it is possible to create an infinite loop (an exception is thrown, then the application tries to reestablish the JMS connection and the same exception is thrown again) and occupy all the CPU resources, thus not letting the JMS recovery procedure complete for a reasonable time.
The exceptions in the symptom section can also appear if there are pending requests when the J2EE Engine is being shut down. In this case, the exceptions appear because the JMS service has already been shut down as part of the shutdown procedure before the completion of all user requests. In such cases, the exceptions can be ignored. To distinguish this case, you can compare the time when the exception occurred with the timestamp of the logs for the server shutdown in <server>/log/system/server.X.log
Solution
If the exceptions appear sporadically, when the whole cluster is being stopped or when a node was restarted and everything else regarding the application using JMS is working fine, ignore the exceptions.
If the application is malfunctioning, the exceptions appear in the trace files and the timestamp of the exceptions do not coincide with shutdown or restart of the cluster, you should check with the application vendor (be it SAP or third party) that the reconnect mechanism has been implemented properly (an exception listener must have been implemented and the reconnection attempts should be performed in a loop with pauses between subsequent reconnection attempts) or if there are other reasons for the failure of the reconnection attempt (additional related errors/exceptions are present in the trace files).
A manual workaround when the application does not implement a JMS recovery mechanism would be to restart the application. Thus, the JMS connections used by the application will be initialized with a valid value at application startup.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.