Skip to Content
author's profile photo Former Member
Former Member

CMS crash with core files and multiple report output generation

Happy new year to everyone,

Our BOXIR3.1SP6FP2 env has recently started behaving weirdly by triggering multiple output to users inbox and email notification out of scheduled reports. Also we have noticed the CMS crash with core file (almost 4GB) generation at the time of multiple report output.

Most of the times, CMC crashes and recycles itself. At few times, CMS services alone went shut down.

OS details: RHEL 5.5, 32 GB RAM, 8 core processor on each of the clustered node, Oracle 10GR2.4 CMS DB server, 11GR2.4 oracle reporting DB server and oracle 11.1.0.6 client.

2015/01/21 23:54:37.946|>=| | |28123|1534131088|{|||||||||||||||DBQueue::Read

2015/01/21 23:54:37.946|==| | |28123|1496185744|
|||||||||||||||(OracleStatement.cpp:156) Prepare: SQL: SELECT ObjectID,
Version, LastModifyTime, CRC, Properties FROM CMS_InfoObjects6 WHERE ObjectID
IN (1004050) ORDER BY ObjectID

2015/01/21 23:54:37.946|==| | |28123|1496185744| ||||||||||||||(OracleStatement.cpp:183) Prepared statement Execute

2015/01/21 23:54:37.965|==| | |28123|1496451984| |||||||||||||||SResourceSource::LoadString 50293

2015/01/21 23:54:37.966|==| | |28123|1496451984| |||||||||||||||SResourceSource::LoadString Unknown exception in database thread

2015/01/21 23:54:37.967|==| | |28123|1496451984| |||||||||||||||SResourceSource::LoadString 33007

2015/01/21 23:54:37.967|==| | |28123|1496451984| |||||||||||||||SResourceSource::LoadString CMS is unstable and will shut down immediately. Reason: %1...

2015/01/21 23:54:38.506|==| | |28123|1496185744| |||||||||||||||(OracleStatement.cpp:156) Prepare: SQL: SELECT ObjectID,
Version, LastModifyTime, CRC, Properties FROM CMS_InfoObjects6 WHERE ObjectID IN (1009213) ORDER BY ObjectID

2015/01/21 23:54:38.506|==| | |28123|1496185744| |||||||||||||||(OracleStatement.cpp:183) Prepared statement Execute

2015/01/21 23:54:38.512|==| | |28123|1455592672| |||||||||||||||(sidaemon.cpp:549) SUNIXDaemon::run: server restart flag is 1..

2015/01/21 23:54:38.513|==| | |28123|1455592672| |||||||||||||||(sidaemon.cpp:552) SUNIXDaemon::run: in abort ...

2015/01/21 23:54:38.513|==| | |28123|1455592672| |||||||||||||||(sidaemon.cpp:555) SUNIXDaemon::run: doing the WithAbort case ...

2015/01/21 23:54:38.520|==| | |28123|1496185744| |||||||||||||||(dbq.cpp:1357) DBQ: Time required to read 1 objects: 20.000000 ms

Thank you,

Karthik

Add a comment
10|10000 characters needed characters exceeded

Related questions

3 Answers

  • Best Answer
    author's profile photo Former Member
    Former Member
    Posted on Sep 24, 2015 at 07:03 PM

    Hi All,

    The issue got resolved few months ago and thought of updating the resolution here.

    Cause: CMS crashed while reading a large or inflated string value for metadata property SI_WEBIDOCPROPERTIES saved with the physical report. As per SAP support engineer, this might have grown abnormally through the use of incompatible webi rich client version.

    Resolution:

    A. Appended the parameter "-maxobjectsincache 1000" at CMC-->Servers-->CMS command line section

    B. Reduced the "SizeOfTheLargestObjectAllowedInTheCacheInBytes"="10000" at .registry file present at ../../bobje/data/.bobj/registry/software/business objects/suite 12.0/cms/instances/CMS_name.cms/

    C. Also SAP provided a SDK LA fix code to clean or remove the large string value for the property SI_WEBIDOCPROPERTIES for all webi documents present in the system.

    Preventive measure: As per SAP's recommendation, better to use the compatible BO client tool same as the server version.

    Thanks,

    Karthik

    Add a comment
    10|10000 characters needed characters exceeded

  • Posted on Jan 25, 2015 at 03:00 PM

    Is it VM server? if it is VM server, make sure server has enough resources available all the time.

    Add a comment
    10|10000 characters needed characters exceeded

    • Former Member Denis Konovalov

      Thanks Denis & Seb. I'll work with internal change management before updating the ulimit (to unlimited) and oracle client to 10GR2.4. Considering the reporting database (11.G2.4), we even tried installing 11.G2.4 client as per SAP's recommendation however it didnt help.

      Below is the result of my memory analysis shell script which shows the CMS crash at 3.6g.

      Wed Jan 28 19:05:35 GMT 2015

      MemTotal: 32947392 kB

      MemFree: 13285252 kB

      Buffers: 920916 kB

      Cached: 13169796 kB

      SwapCached: 196 kB

      PID USER VIRT RES CPU% MEM% PROC_NAME

      8295 bobje 25 3820m 3.6g 51m R 100.4 11.3 7:51.34 boe_cmsd

      9492 bobje 18 376m 252m 57m S 0.0 0.8 2:34.72 WIReportServer

      9589 bobje 18 278m 153m 57m S 0.0 0.5 0:42.87 WIReportServer

      **************EOL******************

      Wed Jan 28 19:05:45 GMT 2015

      MemTotal: 32947392 kB

      MemFree: 16565332 kB

      Buffers: 921304 kB

      Cached: 13554876 kB

      SwapCached: 196 kB

      PID USER VIRT RES CPU% MEM% PROC_NAME

      11260 bobje 15 88380 28m 18m S 46.5 0.1 0:04.36 boe_cmsd

      9492 bobje 18 376m 252m 57m S 0.0 0.8 2:34.73 WIReportServer

      9589 bobje 18 278m 153m 57m S 0.0 0.5 0:42.87 WIReportServer

      I'll keep you posted on the result after ulimit change.

      Thanks,

      Karthik

  • Posted on Jan 27, 2015 at 11:18 AM

    Hi Kartik,

    Could you confirm if all 14 or 15 connections are established with the DB

    - You could get this as an info in the system logs, /var/logs/messages

    - Also one can check the active connections using

    netstat -tulpn | grep 6400 >> /home/<username>/out.txt

    or

    netstat -an | grep 6400 >> /home/<username>/out.txt

    Additionally check if there are any audit backlogs in the /auditing directory

    -Guru

    Add a comment
    10|10000 characters needed characters exceeded

Before answering

You should only submit an answer when you are proposing a solution to the poster's problem. If you want the poster to clarify the question or provide more information, please leave a comment instead, requesting additional details. When answering, please include specifics, such as step-by-step instructions, context for the solution, and links to useful resources. Also, please make sure that you answer complies with our Rules of Engagement.
You must be Logged in to submit an answer.

Up to 10 attachments (including images) can be used with a maximum of 1.0 MB each and 10.5 MB total.