Skip to Content
0

BO 4.1: Connection Server and Web Intelligence Processing server FAILED !

Feb 21, 2017 at 01:52 PM

1.5k

avatar image

Hi experts !

We are facing a very strange issue.

Our productive system consists of 9 hosts:

  • 4 hosting Apache Tomcat 7.0.59 (web application servers),
  • 4 hosting SIA (4 nodes cluster SAP BI 4.1 SP7 patch 3) and
  • a machine hosting MySQL 5.6.30 as our cms-repository and audit repository.

All hosts run Linux Red Hat 6.6.

After a certain point, on SIA number 1, there is no way to get Web Intelligence Processing Server and Connection Server to a running state: CS goes into “failed” state after some attempts to start (we can see it going into “starting” and then “initializing” state). WIPS is neither trying to start: he stays in “stopped” state and, trying to start it manually, it goes immediately in “failed” state.

Nothing useful found in logs.

Luckily, node 2,3 and 4 still working correctly.

In last days we tried to do everything we can:

  • Restarted SIA (“stopservers” then “startservers”);
  • Rebooted host;
  • Restarted whole cluster (all 4 SIA nodes);
  • Deleted & recreated CS and WIPS on node 1;
  • Checked note 1616130 (“<inst_dir>/setup/boconfig.cfg” file is ok);
  • Created another node on the same host;
  • Performed a REPAIR of the node

Nothing has changed the situation.

Anyone has an idea to resolve our issue ?

10 |10000 characters needed characters left characters exceeded
* Please Login or Register to Answer, Follow or Comment.

6 Answers

Best Answer
Megh Vora Feb 22, 2017 at 01:01 PM
2

Hi Andrea,

Below is what I am seeing in the logs:

|connectionserver_pme2easn01v.ConnectionServer_new|28636|47761983538944||{|0|91|0|3|-|-|-|-|-|-||||||||||||[CS] ServerInitialization::InitManager::Initialize |e6b3b35e-5c3e-21b7-d633-8fa3e44e84aa|2017 02 22 08:47:21:837|+0100|Error| |>>|E| |connectionserver_pme2easn01v.ConnectionServer_new|28636|47761983538944|| |0|91|0|3|-|-|-|-|-|-||||||||||||[CS] Unhandled exception raised during CS initialization. init_srv.cpp:95:static void ServerInitialization::InitManager::Initialize(Initialization::Mode, int, char**): TraceLog message 298 |3ee909a6-29da-073b-3fb5-b1e62ea5e8e1|2017 02 22 08:47:21:837|+0100|Error| |<<|E|X|connectionserver_pme2easn01v.ConnectionServer_new|28636|47761983538944||}|0|91|0|3|-|-|-|-|-|-||||||||||||[CS] ServerInitialization::InitManager::Initialize: 0.035 |b068153b-ab3f-8868-c392-7904a8a2946a|2017 02 22 08:47:21:837|+0100|Error| |>>|E| |connectionserver_pme2easn01v.ConnectionServer_new|28636|47761983538944|| |0|92|0|2|-|-|-|-|-|-||||||||||||[CS] The CS initialization failed within the CS server. Aborting.


===Next Step===

I would recommend you to go for the boconfig.cfg troubleshooting.

Before performing below steps stop all the BI services and kill any stuck processes for WebI Processing Server and Connection Server

1. Locate the boconfig.cfg file e.g. /home/bi4/BI4/setup/boconfig.cfg and copy the boconfig.cfg file from a working BI4 system and make sure you amend the paths and filenames to match your current system. Refer to SAP KBA: 2358425 - ConnectionServer will not start

2. Then remove tmp/bo_config.mutex. Refer SAP KBA: 2331564

3. Start the servers.

Though you did not run out of disk space I would recommend for the above steps.

Let me know how it goes.

Thanks,

Megh

Show 2 Share
10 |10000 characters needed characters left characters exceeded

IT SEEMS TO WORK !

we have to do some tests but... now Connection Server and Web Intelligence Processing Server are going into RUNNING STATE !

Thank you very much MEGH !!!

1

That's great!!

Thank you,

Megh

0
Andrea Pancotti Feb 22, 2017 at 08:00 AM
0

Hi Megh

thank you again for your support !

At the link below you can find CONNECTION SERVER log wrote in starting phase (log level set to "high"):

https://drive.google.com/drive/folders/0BwohRDrNMJxednBYdjlJSWhXcUE?usp=sharing

(it was impossibile to attach it to this message because of forum limitations).

Unfortunately WIPS is writing NO LOGS ! It goes to "failed" state without writing anything ! Its level is set to "high" as well.

THANK YOU VERY MUCH !!!

Share
10 |10000 characters needed characters left characters exceeded
Megh Vora Feb 21, 2017 at 02:17 PM
0

Hello Andrea,

Do you remember of any changes made recently?

Make sure that the disk space is sufficient.

Also check the Auditing folder on Node 1, What is the size of Auditing folder? Do you see many files with size more than 0 KB?.

If yes then stop the SIA and try to rename the current Auditing folder to Auditing_old and then again try to start the SIA. See if you are able to bring up the two servers fine.

Can you provide us with the /var/log/messages of the time when the servers fail to start?

If Auditing is working fine then have a look at SAP KBA: 2331564 - bo_config.mutex file corrupts after running out of disk space, not allowing Webi Processing and/or Connection servers to start

Regards,

Megh

Share
10 |10000 characters needed characters left characters exceeded
Andrea Pancotti Feb 21, 2017 at 03:40 PM
0

Hi Megh

Thank you very much for you answer.

- Change made on this host, prior the issue, was the disinstallation of the SAP Solution Manager Diagnostic, but, after the issue, we restored the file systems involved. Now SMD is working fine again, but BO’s issue is still present.

- There are no space issue on this host !

- We checked folder “<inst_dir>/sap_bobj/data/Auditing”: its size is 156 KB

In that folder there are 1454 files with extension *.txt; all of them are 0 KB size except one:

-rw-rw-r-- 1 boprod sapsys 3646 Jan 27 11:03 audit2e4e17215a03001ecd4f8489ba17c0d3361658878C6435E6.txt

However we tried to rename Auditing folder and restarted the SIA (all servers are not in automatic start)

but WIPS and CS still going in failed state.

In attach you’ll find “/var/log/messages” file recorded during SIA startup.

- We have just checked note 2331564: in “/tmp” folder we found this file:

-rw-r--r-- 1 root sapsys 0 Jul 10 2014 bo_config787.mutex

But, as you can see, it is dated “10 July 2014”… this node was working fine until January 27th!

Furthermore we see similar files on other hosts (working fine)... do you advice us to try to remove it and retry ?


Share
10 |10000 characters needed characters left characters exceeded
Megh Vora Feb 21, 2017 at 06:17 PM
0

Hi Andrea,

The /var/log/messages is not giving much insight. Can you apply High level traces on Connection and WebI Processing Server by changing the log level from unspecified to High and start both the servers. Once they fail collect both the glf files and provide it to us.

Lets see the log files first and then we can think of performing the steps mentioned in KBA: 2331564

Thanks,

Megh

Share
10 |10000 characters needed characters left characters exceeded
avatar image
Former Member Jul 19, 2017 at 08:52 AM
0

/var/log/messages would have some things captured if nothing in BO logs

try creating new WIPS & CS servers on the problematic node and see if they are working fine

not recommended but recreating a SIA would be a last option to be tried for the problematic node..

Share
10 |10000 characters needed characters left characters exceeded