on 06-11-2016 12:05 AM
Hello colleagues,
We have a Production environment in BI4.1 recently updated from SP2 Patch 2 to SP7 Patch 1, composed of 4 nodes, two Intelligence nodes with (RAM 75G each) with CMS server and other two for processing(RAM 65G each), and CMS and Audit DB in Oracle 11g. Both CMS nodes have Tomcat running.
I'm not completely sure if after the update came the issue we are currently experiencing.
We login to the CMC web page without any issues or spinning concerns, but once we are inside the CMC, we are noticing that when we move to "Servers" or "Users and Groups" tabs take too long to load them completely, Servers tab takes 50 seconds to display all Nodes information(I must mention once is fully loaded, we can move through nodes, metrics and properties options without delay or concerns, the spinning is only occurring every time we move to one of these tabs; for Users and groups is taking 1 minute to populate every time we move there, in our DEV and QA environments take less than 10 sec for each tab. Also we have tried using other Tomcat environment(QA) with the same result, All of the other tabs load correctly without delays.
After update we just rebuilt the Paltform Search Index due was not working before update, it was been going slow and due some high usage of the system we just stopped the Platform Search indexing, but the Servers/Users and Groups(Almost 3000 users only) tabs still have the same behavior.
Can you consider this is a normal behavior? Have you experienced this before? Where could be the malfunctioning here is case is something wrong?
I'll appreciate your responses and feedback.
Thanks and regards!
Roy Garcia
Hello all,
For the last months, we have been looking for the solution to resolve this issue and I found that the high amount of objects in the CMS DB were failed instances at the system. We found that there were over 7 million of failed instances, caused by the DST bug:
1448881 - Multiple instances spawned after daylight savings time change.
So, we have been cleaning the failed instances with the script provided by SAP in the note attached. And currently we have over 700,000 so the amount have been decreasing considerably, last weekend after we have reduced the high amount of failed instances that means reduced the amount of objects in CMS DB, we performed a table reorg for all CMS tables, that action decreased the size of the CMS schema from 35Gb to 4Gb, and increased the MaxObjects in CMS cache to 300,000 per CMS. These highly imrpoved the performance for the issue we were having.
Currently the "Servers" and "Users and groups" tabs take over 15 sec to load, we truely believe, once we completely removed the high amount of failed instances in the system, the slowness stills present will be gone.
Thanks all for your help and support!
Roy
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello all,
Also I have validated that it's performing slow when a query goes to the CMS DB looking for SI_ID=16(Servers), SI_ID=19(Users) and SI_ID=20(Groups), almost 1 minute for each when querying using the Query Builder: "select SI_ID from CI_SYSTEMOBJECTS where SI_ID = 16"
Regards!
Roy
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello all,
We have performed the system restart last weekend, and also rebuilt the Catalina folder to clean the cache for Tomcat WebApp, but still the same. Untill now we have noted we currently have over 5 million objects inside CMS sytem DB and almost 100,000 at CMS system cache objects, when in our QA landscape is less than 700,00 objects into the CMS system DB, not sure if this could be the cause of the slowness presenting in the system.
Also we are noticing that each CMS process(2 within the environment) at OS level are cosuming over 9G of RAM(we don't have enabled -Xmx parameters for these servers), again, not completely sure if it's normal.
Thanks for your comments and feedback!
Regards!
Roy Garcia
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hello Roy,
If the number of objects in CMS cache is less than number of objects in CMS DB then we can have performance issues. In this case you can add the switch -maxobjectsincache 500000. This might improve the performance. Adding this switch will allow the CMS cache to increase upto 500000 objects as the default value is 100000.
Also, check size of "Logging" and "Auditing" folder in the installation directory. If the size is very large you might need to clear these folders. And also the LDAP and SAP authentication group updates can be crucial in your case. Try setting the schedule to a frequency like every 8 hours or 10hours.
-Check CMC==>Applications==>platform search==>properties and if the crawling frequency is set to "continuous crawling" change it to "scheduled crawling"
🙂
Thanks,
Kapil
Hello Kapil,
Thanks for the suggestion to increase the -maxobjectsincache to improve the CMS DB performance. I haven't applied it since it's a PRD system and should go thorugh a whole proccess but I will confirm it with SAP and suggest it to the customer. Logging and Auditing folders have low size so no problems with those. We currently have disabled the indexing functionality. For the LDAP and SAP authetication updates I will propose it to the customer to schedule it less frequest.
Thank you!
Roy
Hello all,
In addition to this I'm noticing at each CMS server metrics, the "Machine Name" and "Server Name" is not showing correctly, I don't think this is normal, have you seen it before? Or what could be the cause?
Also we have noticed the Promotion Management it's performing very slow, while trying to add objects or exporting them takes too much time, almost 30 min for one single report, And I'm leading all of the slowness presenting in the system to the same cause, that is unknown right now.
Regards!
Roy Garcia
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Jawahar,
I can try that, but the main thing is that in the other environments(DEV, QA) is currently set to auto assign and its showing the Machine Name(host) and the "Name Server"(host:port) correctly. So I was thinking what could happened that caused these metrics don't show correctly anymore.
Regards!
Roy Garcia
Hello there,
You can try following things:
-Clear tomcat cache.
-In CMC Homepage==>Preferences, check the number of objects per page and set it to 50 if it is set to a higher value.
-Check value of "Auditing events in the queue" in CMS metrics. If it is too high, you might need to clear "Auditing" folder in the installation directory.
-In case you are using any third party authentication like Windows AD or LDAP etc. then you can check if the plugin schedule frequency is not too less like every hour. Set it to run after every 12 hours.
Regards,
Kapil
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Kapil,
This weekend we'll have a planned restart where I will clean Tomcat's cache though in that too, also the numer of objects per page we currently have set to 50, the auditing events in queue for CMS is less tan 5 objects. For the third party authentication, we have enabled the LDAP and SAP auth. and is scheduled to update every hour, I'll check with the customer if it's viable to change it at least for every 6 hours.
Thank you so much for your comments.
Regards!
Roy García
Hello Raunak,
Actually we currently have almost 100,000 objects in CMS system cache for one CMS server and 102,000 objects in the other one. So I don't think setting to 400,000 would help, also it's a Production environment and it should go through a whole process to change a parameter for CMS server, what does that parameter will do? that can imrpove the CMS functionality.
Regards!
Roy Garcia
Have you done APS Configuration as per SAP Note?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
84 | |
24 | |
11 | |
9 | |
7 | |
6 | |
5 | |
5 | |
5 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.