Performance issue on 601/Provisioning

[Former Member]

Hi, all

We are on IDM 7.2 SP7. So far, we are using the 601/Provisioning to assign roles to user in ECC 6.0. The task is working fine except that it is very very slow. Looking at the task, there are a lot of steps involved and they run one by one. There are lots of logic too built in. Maybe that explains why this task takes 4 minutes to create a ECC user and then assign a role.

I am tempted to create a custom task with the 601/Provisioning as base. Before I do that, would you be able to share your tips? Is there any way you know of to improve its performance without doing an overhanul? I haven't found any OSS note or SDN threat that talks about performance improvement on that task.

Thanks,

Jonathan.

[Former Member]

Hi, Ian

The CPU is hopping between 60% to 100% used while the job is running. I did not check the disk utilization though as it is coming from the SAN. The windows diskmon may not be telling all the stories. Each user was assigned the master priviledge as well as an R/3 priviledge in IDM for my test case.

By looking at one of the job log(DSE.log) file for the job step "3462/Apply pending on Group", I saw the following log:

07.05.2013 13:18:26 :I:sap_core_getPendingMsKeysInGroup

07.05.2013 13:18:26 :I:MX_PRIV_GROUPING_GUID: not found - grouping disabled

07.05.2013 13:18:26 :I:Relevant mskeys (in group): already-set

07.05.2013 13:18:26 :I:sap_core_applyPendingOnGroup: already-set - {mskey=already-set}

07.05.2013 13:18:26 :I:already applied pending operation

07.05.2013 13:18:35 :I:sap_core_getPendingMsKeysInGroup

07.05.2013 13:18:35 :I:MX_PRIV_GROUPING_GUID: not found - grouping disabled

07.05.2013 13:18:35 :I:Relevant mskeys (in group): already-set

07.05.2013 13:18:35 :I:sap_core_applyPendingOnGroup: already-set - {mskey=already-set}

07.05.2013 13:18:35 :I:already applied pending operation

07.05.2013 13:18:42 :I:sap_core_getPendingMsKeysInGroup

07.05.2013 13:18:42 :I:MX_PRIV_GROUPING_GUID: not found - grouping disabled

07.05.2013 13:18:42 :I:Relevant mskeys (in group): already-set

07.05.2013 13:18:42 :I:sap_core_applyPendingOnGroup: already-set - {mskey=already-set}

07.05.2013 13:18:42 :I:already applied pending operation

07.05.2013 13:18:52 :I:To Generic pass completed in 26.763 seconds.

07.05.2013 13:18:52 :I:Handled: 3 Added: 3

07.05.2013 13:18:52 :I:Job completed in 27.197 seconds.

07.05.2013 13:18:52 :I:Handled: 3 Added: 3

07.05.2013 13:18:52 :I:Total time used is 29.465 seconds.

The javascript sap_core_applyPendingOnGroup took only one second for each MSKEY(looks like there are 3 in total passed from PAR). But IDM has waited exactly 10 seconds before the javascript is rerun against the next MSKEY. I am not sure where the 10 seconds are coming from. I went through the javascripts sap_core_getPendingMsKeysInGroup and sap_core_getPendingMsKeysInGroup and did not see any statement to "wait for 10 seconds".

I have opened an OSS message but it is not going anywhere.

Thanks,

Jonathan.

[Former Member]

Hi, Ian

It's a simple role, not a composite. There are 5000+ roles in the r/3 system, but I don't think this is substantial. THe step according to the IDM log to assign the role took between 3-9 seconds to do. The longest step by far is the "3462/Apply pending on Group". There are also a lot of time not accounted for in the job log. The number just don't add up.

Anyhow, we also setup the good old LDAP synchronization on the same ABAP system to compare(with 40 users). It is a lot faster to just create the users.

Thanks,

Jonathan.

[Former Member]

Hi Jonathan,

So in Idm speak you are assigning a single privilege and not a role, just to keep things clear.

What is the CPU load on the IdM server where the dispatcher is during this? Also, how is the database? We found IdM very sensitive to I/O performance on the database.

Cheers,

Ian

[Former Member]

Hi all

I have setup a test job to create 40 new users and then provision one role to each of them. It took 80 minutes to do so. I can see IDM is doing re-ordering and create all users first and then assign roles later. But the performance is still not very good considering we will be having 15000+ new users to setup in the next phase of the ESS/MSS rollout.

That means it will take 10 days if the number is proportional. Hopefully in the production environment we will get better hardware compared to the development server we are on. Is there a way to create more dispatchers? Since each repository can only have one "add task" event, I don't know how we can make use of the extra dispatchers. Maybe we will create a few different repositories for the same ECC system? Has anyone tried?

Thanks,

Jonathan.

[Former Member]

Hi Jonathan,

That is too slow! What is the content of the role, ie how many privileges from how many systems?

Thanks,

Ian

[Former Member]

Hi, all

Thank you for all your feedback. I will setup a job to load a bigger batch of users with some roles. I will update you how it goes.

Having worked on CUA and OM myself, the 4 min waiting is definitely a turnoff in my opinion. The day to day security requests only deals with a couple of users with a few roles. The security team on my current client doesn't do anything else other than su01, su10 and pfcg(we don't even have OM). Other good thing IDM brings is the HCM integration. But then I am sure they will ask me to compare it to OM which I am not ready to start the debate yet.

Thanks,

Jonathan.

[Former Member]

In my opinion, 4 minutes to provision one user to an ECC system is very slow. I have just started my second IdM implementation which is v7.2 SP7. I will keep an eye out for this potential performance issue.

4 minutes may not be a big issue for 1 user. However, this may indicate underlying performance issues. Things may become unacceptable when you do a mass update to many users. I'd suggest you need to analyse the performance issue and perhaps also raise a SAP Message.

To give you an idea of my expectations, on my first implementation we had IdM connected to ~60 SAP systems/clients. If we provisioned a single account to all systems then this completed in ~3-5 minutes (total). This was on a v7.1 system. I understand there have been performance improvements in 7.2 so my expectation is provisioning will be faster. I'd be hoping for <30 seconds to provision to one system.

Richard.

[Former Member]

Hey Richard

It may be 4 minutes for the entire process, but its not per user.  IdM will batch them up and run multiples at the same time (depending on the dispatcher thresholds).  So 20 people will probably take the same 4 minutes...

Peter

[Former Member]

I completely agree with Peter. If you are trying to demo this, I would assign a business role that contains ABAP and java privileges at least, as that will be quicker than just CUA.

The benefits however go much beyond the speed of provisioning. When you introduce self service, the end to end time should drop from a few hours/ a few days depending on your existing processes down to some minutes, with no security effort involved.

That is where the real performance improvements come in.

Cheers,

Ian

[former_member2987]

Peter,

Good point.  Jonathan, you might want to try working with some subsets of users and do some benchmarking, 1 user, 20 users, 100 users.

Also what's being loaded in for each user, if there are a number of privileges (roles) to be added it could take some time.  Also look into potential latency issues.

I've also noticed that DEV and even QA environments are slower than PROD.  Could this be part of the mix?

Thanks,

Matt

[Former Member]

Hi Folks,

I'm at SAP at the moment and so thought I would take the opportunity to try this out on one of the training system here - which are as vanilla IdM 7.2 SP7 P0 as you can get.

These are however a bad case for performance testing as all the IdM components, including the database are on the same server, along with also the ABAP stack and the Java stack I am provisioning to.

The test was to upload a business role that contained 2 account privileges (PRIV:<REPNAME>:ONLY) from a spreadsheet, one for an abap stack and one for a Java (DB UME) stack.

The timing was taken from the first provisioning task running, to the last provisioning task running in the job log.

What I found was

• For 1 user in the upload, it took 1 min 44seconds
• For 99 users in the upload, it took 11mins 5s

What I should say though was after about 1 minute in, the server was running at 100% CPU usage for almost the entire time, so this is not an IdM constraint itself, more of the kit it was running on.

Anyone else care to share some times for a similar test?

Cheers,

Ian

[Murali_Shanmu]

Thanks for the update Ian. Yes, those training systems are not the best. I will see if I can test something once I  visit my customer site.  The number of dispatchers and the runtime engine settings are also an important factor in optimizing the performance. As Matt pointed out, there will be difference between DEV and PRD environments (High Availability).

[Former Member]

Hi, Ian

Would you mind to share your system configuration, like the CPU speed, how many CPU, db version, how many servers, etc?

I have done some DB tuning(on DB2 UDB) and set the dispatcher trace level from info to error. It seems to make things go a little faster. To assign a single user with one role, the 601/Provisioning took 3 min 10 seconds. It took 2 min and 10 seconds before the ECC role was actually assigned. However, this is still too slow for the security team to accept the solution. Our wintel server CPU is AMD Opteron 8356 Dual 2.3GHZ. Definitely not the fastest, but it meets SAP's requirement.

I also turned on SQL statement analysis and don't see any stored procedure that took substantial amount of time, all less than 6 seconds.

I will run more testing on batches tomorrow as I am stuck with another task.

Thanks,

Jonathan.

[Former Member]

Hi Jonathan,

I saw similar problems, where is seems to take 1 minute between each task that executes. I found that a restart of the dispatcher (the trusty switch it off and back on again) helped. It is something of a concern to us, but we saw it only intermittently.

Does a restart help in your case?

Cheers,

Ian

[Former Member]

Hi Jonathan

Unfortunately most of the stuff there is required unless you're cutting out huge swathes of the default system.  You might find that its not the tasks that are slow but the dispatchers.  None of those tasks are particularly time consuming (although the actual write to ABAP can take time but there's nothing you can do about that).

Is there a particular task that's actually taking more then a few seconds to run?

The check interval means that there's a 5 second delay between each task.  You can try reducing this, or changing the jobs so that a dispatcher is dedicated to the tasks you want run quickly.

So... yes.  You can change it but I'd make that my last resort rather than the first as the flow on effects may be large and unanticipated.  I'd also have to say that 4 minutes isn't actually a long delay all things considered - its not really a real time system.

Peter

[Former Member]

Hi, Peter

Thanks for confirming that the 4 min is a "normal" IDM behaviour as I was wondering if there is something I am missing in the IDM config.

When I did the demo the other day on IDM to the security team, the reception was poor. It takes 4 minutes to provision an abap role compared to 10 seconds in CUA and that was a major concern. The security team also wants something easy to use/learn ... well ... they will never like IDM on this regard. But I am hoping to make the provisioning not 4 minutes, maybe a minute?

So I was looking into the 601/provisioning. I notice the task has a lot of steps. The longest steps that I am thinking of removing or streamlining is 3246/Apply Pending on Group. It takes between 16-20 seconds to run. The other steps I am thinking of modifying are 3114/SYNC, 3145/Trigger Notification, 3182/Apply Pending on Group - w/notify. As you point out, we also have to wait 5 seconds for the dispatcher to wake up and run each step. So the less step the better.

Any comment on that plan?

Thanks,

Jonathan.

[Former Member]

The CUA speed is because its not doing anything much - just click - make user.  The IDM solution is doing a hell of a lot more work to cater for all the other options.

The applyPending is needed.  However, it doesn't stop the user being provisioned.  That happens earlier.  ApplyPending triggers all the privileges that were waiting for the account to be created.  This is why its slow (and also because the script isn't massively efficient...).

I'm not sure what the SYNC job is doing off the top of my head.

Trigger notification can be removed (disabled) if you don't need an email saying its done.

The issue you have is that each small component is required and very few can be removed.  You can probably try and combine them into something bigger but with less steps but you'll have other issues.

Perhaps the solution is to notify them that the account is created immediately after 1. Exec Plugin is finished.  This will tell the admin guys that the account is created quickly and then the privileges can catch up while they're reading the email

Peter

[Murali_Shanmu]

Jonathan,

I can also confirm that 3-4 minutes is normal. If you have GRC Access Control integration, this will be even longer . The CUA is all ABAP based and will be quick with RFC Calls, whereas in IdM there are lot of other tasks which are performed even before a JCo/BAPI is called to update the data in your ABAP system.

I tried to dig in what 3114/SYNC does. The case statement above it just checks if the repository has the constant REPOSITORY_SYNC and then executes 3114/SYNC (which does nothing - the script is also empty).

3182/Apply Pending on Group has got few SQL statements which are required. Yes, you can get rid of Notification triggers if you are not using them.

Let us know if you managed to reduce the time with cut-down version,

Cheers

Murali

[Murali_Shanmu]

Jonathan,

There were lot of performance improvements as part of SP7. I am also curious to know of a way in which these tasks can be executed faster.

You have to be cautious if you are going to put up your own provisioning task (taking the standard as a base). In the future, when SAP enhances these tasks with some improvements, you would have to manually do the adjustments. (unlike ABAP where the Enhancement Framework allows us to put our logic in the customer namespace within the Standard Code). Just my thoughts,

Cheers

Murali.