Why Wacs Server is in Failed state while health state shows Unkonwn value

[former_member530803]

We are on BO 4.2 sp4

In CMC - Monitoring (or CMC - Servers) the WACS server shows Failed state. However if I go to watchlist the wacs.health state shows Unknown Value. When I try to edit that watch and set the caution rule it actually evaluates to Healty ? but the tracking shows Unknown value??

The weirdest thing is the server seems to work fine. I have same settings on this server as my test box and all is ok there.

I di try restarting the whole server and did not help.

P.S. some other on watchlist items like Web Intelligence Services watch is in failed status while all the watched metrics are showing as healthy?

I am completely confused why is this happening.

Alex

[denis_konovalov]

1. If service (wacs) has failed status, then its monitoring status will be unknown, since it can't report its status.
You need to investigate what is wrong with WACS. Enable trace on it, review its properties in CMC (it might show which subservice is failing) etc...
2. Same with webi - need to drill down to see what is failing.

[former_member530803]

I did high trace on WACS and

I see my servername.WebApplicationContainerServer_gc.log

21.973: [GC (Allocation Failure) [PSYoungGen: 131584K->11662K(153088K)] 131584K->11670K(502784K), 0.0486872 secs] [Times: user=0.06 sys=0.02, real=0.05 secs]
34.371: [GC (Metadata GC Threshold) [PSYoungGen: 54488K->10989K(153088K)] 54496K->11021K(502784K), 0.0128751 secs] [Times: user=0.11 sys=0.00, real=0.02 secs]
34.384: [Full GC (Metadata GC Threshold) [PSYoungGen: 10989K->0K(153088K)] [ParOldGen: 32K->10800K(349696K)] 11021K->10800K(502784K), [Metaspace: 20281K->20

.and in my servername.WebApplicationContainerServer_trace.glf

687102EB3795411DB331B6084614D63A3f7e3|2017 12 15 07:15:40.197|-0500|Information| |==| | |wacs_servername.WebApplicationContainerServer| 3228| 39|Registration-CMS| ||||||||||||||||||||com.businessobjects.framework.servers.platform.adapters.ebus.registration.RegistrationBean||CMS has existing binding. No need to re-register.
|687102EB3795411DB331B6084614D63A3f7e4|2017 12 15 07:15:40.197|-0500| |Warning| | | |wacs_servername.WebApplicationContainerServer| 3228| 39|Registration-CMS| ||||||||||||||||||||com.businessobjects.framework.servers.platform.adapters.ebus.registration.RegistrationBean||Check registration again in 60 seconds.
|687102EB3795411DB331B6084614D63A3f7e5|2017 12 15 07:15:44.384|-0500|Debug| |<<| | |wacs_servername.WebApplicationContainerServer| 3228|4631790|JMX client heartbeat 9| ||||||||||||||||||||com.businessobjects.sdk.monitoring.boeiiop.BOEJMXConnector||[checkConnection] BOE communicator admin checking connection
|687102EB3795411DB331B6084614D63A3f7e6|2017 12 15 07:15:44.384|-0500|Debug| |<<| | |wacs_servername.WebApplicationContainerServer| 3228|4631789|JMX client heartbeat 7| ||||||||||||||||||||com.businessobjects.sdk.monitoring.boeiiop.BOEJMXConnector||[checkConnection] BOE communicator admin checking connection
|687102EB3795411DB331B6084614D63A3f7e7|2017 12 15 07:15:44.494|-0500|Debug| |<<| | |wacs_servername.WebApplicationContainerServer| 3228|4631791|JMX client heartbeat 8| ||||||||||||||||||||com.businessobjects.sdk.monitoring.boeiiop.BOEJMXConnector||[checkConnection] BOE communicator admin checking connection
|687102EB3795411DB331B6084614D63A3f7e8|2017 12 15 07:15:44.494|-0500|Debug| |<<| | |wacs_servername.WebApplicationContainerServer| 3228|4631787|JMX client heartbeat 6| ||||||||||||||||||||com.businessobjects.sdk.monitoring.boeiiop.BOEJMXConnector||[checkConnection] BOE communicator admin checking connection
|687102EB3795411DB331B6084614D63A3f7e9|2017 12 15 07:15:57.213|-0500|Information| |==| | |wacs_servername.WebApplicationContainerServer| 3228|6441533|ORBacus:ThreadPerRequest:Dispatcher|{|382|1|8|1|BIlaunchpad.WebApp|SRVRNTA38:1456:42.220692:1|cms_SRVRNTA38.CentralManagementServer.queryEx3|localhost:3472:4768.19794111:1|._non_existent|SRVRNTA38:3228:6441533.5529562:1|CsAdDDGik0IJv.nj2uRbI9835e12|||||||||||START INCOMING CALL Incoming: FROM [cms_servername.CentralManagementServer.queryEx3#localhost:3472:4768.19794111:1] TO [._non_existent#SRVRNTA38:3228:6441533.5529562:1]
|6

.No real errors just a warning and information.

the only other trace I can see is:

com.crystaldecisions.thirdparty.org.omg.CORBA.BAD_PARAM: invalid service context ID: 4094 minor code: 0x4f4d0014 completed: No
at com.crystaldecisions.thirdparty.com.ooc.PortableInterceptor.RequestInfo_impl.getServiceContext(RequestInfo_impl.java:141)
at com.crystaldecisions.thirdparty.com.ooc.PortableInterceptor.RequestInfo_impl.get_request_service_context(RequestInfo_impl.java:472)
at com.crystaldecisions.enterprise.ocaframework.TenantClientRequestInterceptor.setCategoryFromServiceContext(TenantClientRequestInterceptor.java:75)
at com.crystaldecisions.enterprise.ocaframework.TenantServerRequestInterceptor.receive_request(TenantServerRequestInterceptor.java:21)
at com.crystaldecisions.thirdparty.com.ooc.PortableInterceptor.ServerRequestInfo_impl._OB_request(ServerRequestInfo_impl.java:369)
at com.crystaldecisions.thirdparty.com.ooc.OB.PIManager.serverReceiveRequest(PIManager.java:526)
at com.crystaldecisions.thirdparty.com.ooc.OB.PIUpcall.postUnmarshal(PIUpcall.java:128)
at com.crystaldecisions.thirdparty.com.ooc.OBPortableServer.ServantDispatcher.dispatchBase(ServantDispatcher.java:164)
at com.crystaldecisions.thirdparty.com.ooc.OBPortableServer.ServantDispatcher.dispatch(ServantDispatcher.java:190)
at com.crystaldecisions.thirdparty.com.ooc.OBPortableServer.POA_impl._do_OB_dispatch(POA_impl.java:1980)
at com.crystaldecisions.thirdparty.com.ooc.OBPortableServer.POA_impl._OB_dispatch(POA_impl.java:1916)
at com.crystaldecisions.thirdparty.com.ooc.OB.DispatchRequest_impl.invoke(DispatchRequest_impl.java:122)
at com.crystaldecisions.thirdparty.com.ooc.OB.DispatchThreadPerRequest_impl$Dispatcher.run(DispatchStrategyFactory_impl.java:87)

Any ideas are appreciated.

[denis_konovalov]

WebApplicationContainerServer_gc.log

is not a trace log, its garbage collection log - shows heap memory management stuff.

.glf is the log to see. You need to trace start up of the wacs.

Is this happening after an install or upgrade or you just noticed it ?

[former_member530803]

Right, I did put the lines form trace.glf in the post above. It starts on line #9.

This started after the upgrade from 4.1 sp6 to 4.2 sp4 (about two months ago). I did the same upgrade on my two different test servers and they still work fine.

Users are logged in creating/running reports no issues there - but something has to be wrong I just need to figure out what.

I will try restarting the WACS and see what gets logged then.

[denis_konovalov]

if thing with wacs started after upgrade - read the upgrade logs from installdata dir.
Maybe there were some issues related to wacs there.

Most workflows related to reports do not involve wacs, unless you use restful webservices .

[former_member530803]

Fiorified BI Launchpad requires restful web service, correct?

[denis_konovalov]

yes, it does.

[former_member530803]

That was the main reason we upgraded to sp4.

I am looking into install dir and nothing about WACS errors. I will keep looking.

[denis_konovalov]

setupengine.log might not have obvious wacs related items, but if it has errors/fails - they might explain why its not working after upgrade

[former_member530803]

I just found something interesting (after restarting WACS):

trc file: "wacs_servername.WebApplicationContainerServer_ncs.trc", trc level: 1, release: "720"
---------------------------------------------------

[Thr 12396] Fri Dec 15 10:22:50 2017
[Thr 12396] ***LOG Q0I=> NiPGetHostByName: '$SMDAgentHost' not found: getaddrinfo [ninti.c 897]
[Thr 12396] *** ERROR => Main agent failed to be initilized due to incorrect host setting! [ncsmtdatasen 20]
[Thr 12396] *** ERROR => Main agent failed to be initilized due to incorrect host setting! [ncsmtdatasen 20]
[Thr 12396] NCS trace timer is disabled since the interval is set to 0
[Thr 12396] NCS data timer is disabled since the interval is set to 0
[Thr 12396] NCS library version 2.3.12 (unicode) loaded
[Thr 12396] NCS_ProcInit API invoked
[Thr 12396] NCS configuration info begin----------------
[Thr 12396] NCS configuration loaded from file D:/Program Files/SAP BusinessObjects/SAP BusinessObjects Enterprise XI 4.0/conf/servername/
[Thr 12396] Current NCS Configuration:
[Thr 12396] _socket_recv_timeout=1000;
[Thr 12396] _socket_send_timeout=500;
[Thr 12396] agent_host=$SMDAgentHost;
[Thr 12396] agent_port=$SMDAgentPort;
[Thr 12396] agent_type=2;
[Thr 12396] auto_transaction_trace_threshold=600;
[Thr 12396] cache_section_count=2;
[Thr 12396] context_map_bucket=13;
[Thr 12396] datasending_interval=0;
[Thr 12396] datasending_method=socket;
[Thr 12396] empty_timeout=300;
[Thr 12396] instrument_level_threshold=0;
[Thr 12396] lib_trace_level=1;
[Thr 12396] logdir=D:/Program Files/SAP BusinessObjects/SAP BusinessObjects Enterprise XI 4.0/logging/;
[Thr 12396] max_components_per_promotedstack=50;
[Thr 12396] max_components_per_transaction=5000;
[Thr 12396] perf_metric_scope=0;
[Thr 12396] skip_times_below_threshold=500;
[Thr 12396] system_name=SBOP ENTERPRISE\servername_6400;
[Thr 12396] trace_enabled_via_e2e=1;
[Thr 12396] trace_enabled_via_em=0;
[Thr 12396] trace_preview_timeout=120;
[Thr 12396] tracecheck_interval=0;
[Thr 12396] tracesize_threshold=1000000;
[Thr 12396] zombie_check_interval=300;
[Thr 12396] zombie_timeout=86400;
[Thr 12396] NCS configuration info end----------------

[denis_konovalov]

this is related to integration with solman or sapagent and usually not relevant.