My BO server is getting hanged and I am not able to access the CMC

[Former Member]

Dear Experts,

My BO server gets hanged and then automatically recovers after 10-15 minutes every hour. During this period I am not able to open my CMC or the BI Launch Pad. What could be the reason for this abnormal behavior of the server.

Server Details:

BO 4.1 SP4 Patch 3

OS : Windows 2012 sever

Data base : SQL Anywhere server located on same box.

The server has a RAM size of 64 GB and at any given point of time not more than 32 GB of RAM is being utilized . The CMS logs are not giving any specific reason for the error. We have already cleared the log files and the Cache of Tomcat and BO server still the problem persists. We have also stopped auditing.

Please suggest some steps for the problem.

[denis_konovalov]

1. is your server sized properly ? (note - memory size and usage does not mean good sizing)
2. Do you have auditing enabled ? -if yes, what is the activity right before and during the "hanging" ? (which reports are being refreshed/exported/scheduled etc..)
3. What do the tomcat logs say ?
4. What does Event viewer shows ?
5. Which queries does CMS runs during "hanging" ?
6. Is the hanging at web tier or backend ? (you can test this by deploying tomcat+BOE webapps on a separate machine and testing if login from there works or by using client tools while server is "hanged" ?
7. What reporting are you using ? Do report processing servers (or any others) change their PID's from before/after "hanging" ?

[Former Member]

Thanks for the prompt reply Denis. Following are the answers to your questions:

1. I had not done the sizing of the server. Can you guide me as to how can I know if the server is properly sized or not ?

2.No we have disabled auditing because SAP suggested to do so.

3.Following is the excerpt from the Tomcat log page at the time error occurred today :

unable to process the recovery steps
com.businessobjects.common.CdzClusterMgmt.ClusterFacadeException: IDL:CdzClusterMgmt/ClusterFacadeException:1.0
at com.businessobjects.common.CdzClusterMgmt.ClusterFacadeExceptionHelper.read(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.idl.OCA.OCAcdz.WICDZServer._WICDZUserSessionStub.openSession(_WICDZUserSessionStub.java:69)
at sun.reflect.GeneratedMethodAccessor364.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.crystaldecisions.enterprise.ocaframework.ManagedService.invoke(ManagedService.java:356)
at com.crystaldecisions.enterprise.ocaframework.idl.OCA.OCAcdz.WICDZServer._WICDZUserSessionProxy.openSession(_WICDZUserSessionProxy.java:783)
at com.businessobjects.webi.proxies.WebiUserSessionStateContext.recoverFromSessionSpec(WebiUserSessionStateContext.java:56)
at com.businessobjects.webi.proxies.WebiUserSessionStateContext.recoverState(WebiUserSessionStateContext.java:42)
at com.businessobjects.corba.generic.container.proxy.AbstractWorkSessionProxy.recover(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.ManagedService.invoke(ManagedService.java:544)
at com.crystaldecisions.enterprise.ocaframework.idl.OCA.OCAcdz.WICDZServer._WICDZUserSessionProxy.closeSession(_WICDZUserSessionProxy.java:280)
at com.businessobjects.rebean.wi.newserver.exec.WebiUserSessionExecutor.closeSession(WebiUserSessionExecutor.java:37)
at com.businessobjects.sdk.core.server.internal.session.CloseSessionCommand.execute(CloseSessionCommand.java:53)
at com.businessobjects.sdk.core.server.internal.corba.CorbaServerImpl.doProcess(CorbaServerImpl.java:79)
at com.businessobjects.sdk.core.server.internal.AbstractServer.processIt(AbstractServer.java:171)
at com.businessobjects.sdk.core.server.internal.AbstractServer.process(AbstractServer.java:133)
at com.businessobjects.rebean.wi.impl.engine.ReportEngineContext.destroy(ReportEngineContext.java:322)
at com.businessobjects.sdk.core.internal.service.ComposedContext.destroy(ComposedContext.java:64)
at com.businessobjects.sdk.core.internal.service.CoreContext.destroy(CoreContext.java:251)
at com.sap.webi.client.toolkit.LoginInfo.free(LoginInfo.java:363)
at com.sap.webi.client.toolkit.SessionContext.free(SessionContext.java:72)
at com.sap.webi.client.toolkit.SessionContext.logoff(SessionContext.java:110)
at com.sap.webi.client.toolkit.SessionContextManager.releaseSessionContext(SessionContextManager.java:56)
at com.sap.webi.client.toolkit.SessionContextManager.releaseSessionContext(SessionContextManager.java:154)
at com.businessobjects.adv_ivcdzview.CleanupWebSession.destroySessionContext(Unknown Source)
at com.businessobjects.adv_ivcdzview.CleanupWebSession.dispose(Unknown Source)
at com.businessobjects.bip.core.web.context.SessionCleanupListener$DisposalEntry.disposeIfNeeded(SessionCleanupListener.java:122)
at com.businessobjects.bip.core.web.context.SessionCleanupListener.sessionDestroyed(SessionCleanupListener.java:68)
at com.businessobjects.http.servlet.internal.ListenerRegistrationManager.sessionDestroyed(ListenerRegistrationManager.java:110)
at com.businessobjects.http.servlet.internal.BundlePathAwareServiceHandler.sessionDestroyed(BundlePathAwareServiceHandler.java:410)
at com.businessobjects.http.servlet.internal.ProxyServlet.sessionDestroyed(ProxyServlet.java:588)
at com.businessobjects.servletbridge.listener.ProxyApplicationLifeCycleListener.sessionDestroyed(ProxyApplicationLifeCycleListener.java:38)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:806)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:742)
at org.apache.catalina.session.StandardSession.invalidate(StandardSession.java:1253)
at org.apache.catalina.session.StandardSessionFacade.invalidate(StandardSessionFacade.java:190)
at com.businessobjects.pinger.TimeoutManagerFilter.doFilter(TimeoutManagerFilter.java:147)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1002)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:761)

4.Generally there is no error showing in the Event Viewer at the time of Server Hang. Although sometimes there are messages like:

"The Desktop Window Manager has exited with code (0xd00002fe)"

For rest of the three questions I will talk to my BO Consultants and get back to you. From the above information can you help me in any way ?

[denis_konovalov]

For #1 - https://archive.sap.com/documents/docs/DOC-33183

For #2 - why would SAP suggest that ? Auditing is cornerstone of proper managing of the BOE environment.

For #3 - this is webi related. maybe someone is trying to run or export huge webi report that hangs tomcat.

For #4 - time to call microsoft 😉 - your windows is breaking.