BI 4.2 WinAD SSO - unusual multiple domains

[mike-rs]

Hi,

We currently have a BI 4.2 SP4 cluster on Win2012 servers, fronted by a load balancer running in a single domain (OLD). Automatic WinAD sso is setup and fully working. We are about to rebuild the SAP BI servers on Win2016 in a new domain (NEW) as part of an infrastructure regeneration program. I haven't been able to find anything relating to this particular scenario and am wondering what to do for the best. To stress the point:

- the SAP BI servers are in domain NEW
- but, everything else is in domain OLD - clients, load balancer and users who need to log on to BI 4.2
- I only need to map in users from the OLD domain. There is no requirement to map in users from the NEW domain

So, I think I have the reverse of the usual scenario where servers, clients, load balancer and most users would be in domain NEW and there is a need to map in some additional users from domain OLD?

My question is then, do I run the SAP BI servers in the NEW domain under a service account from the OLD or NEW domain?

I think the service account from the OLD domain?
- the OLD service account has already been configured for Kerberos delegation and has SPNs defined against it (HTTP/LOADBALANCER in particular)
- the only users I need to map in to the CMC are from the OLD domain
- I believe this would allow me to build and test isolated nodes on Win2016, join them to the cluster when ready and retire the old nodes. The load balancer would also need updating to route to the nodes that were running a Tomcat instance but the whole process would be pretty much seamless

Can anyone see a flaw in this plan?

If I run the servers under a service account from the NEW domain I have to start from scratch configuring that user for Kerberos delegation and all the users I want to impersonate are now in a different domain to the service account which just sounds more prone to issues from the start! I will also need an SPN HTTP/LOADBALANCER against the NEW service account and I already have an SPN of this name against the OLD service account which again rings alarm bells.

Any input gratefully received.

Many thanks,

Mike

[BasicTek]

see what comes up in the web/app logs https://apps.support.sap.com/sap/support/knowledge/preview/en/2820819

-Tim

[mike-rs]

So, WinAD login from the client tools and manual login to the webapps sorted via the krb5.ini file. But silent WinAD SSO refuses to work. I think I might have 2 problems actually. These are the SPNs defined on my service account in the NEW domain.

HTTP/DNS_ALIAS 
HTTP/SERVER.NEW.COM
BICMS/SERVICE_ACCOUNT.NEW.COM

I don't have a load balancer in play at the moment.This is a single server environment with a dns alias defined for the server to give users s friendly URL to connect to. Tomcat is running on port 443 with SSL enabled.

If I go to https://server.new.com/BOE/BI I am just presented with the usual login screen. No message or error of any kind. In the Tomcat log I find this:-

[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: GSS: Acceptor supports: KRB5
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: Ticket service name is: HTTP/SERVER.NEW.COM@NEW.COM
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: GSS name is: service_account@NEW.COM
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: Using keytab entry for: service_account@NEW.COM
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: ** decrypting ticket .. **
with key
  Principal: service_account@NEW.COM
  Type: 1
  TimeStamp: Thu Oct 31 11:49:30 GMT 2019
  KVNO: -1
  Key: [23,  9f c0 cc f9 5b 81 92 5c d8 7c c4 f7 da 29 62 2a ]

[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos:   decrypted ticket:
Ticket:
  encryption type: 23 (DECRYPTED OK) 
  key version num: 4
  service principal: HTTP/SERVER.NEW.COM@NEW.COM
  TransitedEncoding:
  client: radicssm@NEW.COM
  session key: [23,  27 6e 9a 44 ca 81 62 f4 bc d2 94 cc 14 45 41 64 ]
  ticket flags: forwardable renewable ok-as-delegate preauthent 
  valid from: Thu Oct 31 12:05:32 GMT 2019
  valid till: Thu Oct 31 22:03:34 GMT 2019
  renew till: Thu Nov 07 12:03:34 GMT 2019
  valid for:
    all addresses
  auth data:
    [1,  30 82 4 82 30 82 4 7e a0 4 2 2 0 80 a1 82 4 74 4 82 4 70 5 0 0 0 0 0 0 0 1 0 0 0 58 3 0 0... 
    0 42 0 49 0 4e 0 54 0 4c 0 4 0 0 0 1 4 0 0 0 0 0 5 15 0 0 0 59 15 2a 7b d2 42 30 2a e6 69 44 6e ...
    [1,  30 5d 30 3f a0 4 2 2 0 8d a1 37 4 35 30 33 30 31 a0 3 2 1 0 a1 2a 4 28 1 0 0 0 0 20 0 0 8d 57...
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: Checking ticket is valid at: Thu Oct 31 12:05:32 GMT 2019 with acceptable clock skew: 300000
Ticket valid from: Thu Oct 31 12:05:32 GMT 2019 till: Thu Oct 31 22:03:34 GMT 2019
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: Setting context expiry to [1572559414000]
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: Current wall time is [1572523532400]
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: ** decrypting application request .. **
with key
[23,  27 6e 9a 44 ca 81 62 f4 bc d2 94 cc 14 45 41 64 ]
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos:   decrypted application request:
++++ KRB-AP-REQ Message ++++
encryption type: 23 (DECRYPTED OK)
ap options: mutual-required 
Ticket:
  encryption type: 23
  key version num: 4
  service principal: HTTP/SERVER.NEW.COM@NEW.COM
client: radicssm@USER_DOMAIN.COM
subkey: [23,  65 32 7e 49 4a c8 13 f9 3a 98 ff 89 10 ff b 75 ]
client time: Thu Oct 31 12:05:32 GMT 2019
cusec: 38629
sequence number: 534569009
++++++++++++++++++++++++++++
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: Reply cache:  purge bin 1572522570 (cutoff = 1572523231), # entries = 7
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: ** creating application response .. **
  with key
[23,  27 6e 9a 44 ca 81 62 f4 bc d2 94 cc 14 45 41 64 ]
[DEBUG] Thu Oct 31 12:05:32 GMT 2019 jcsi.kerberos: created application response:
++++ KRB-AP-REP Message ++++
encryption type: 23
sequence number: 198890459
sub session key: null
client time: Thu Oct 31 12:05:32 GMT 2019
cusec: 38629
++++++++++++++++++++++++++++
com.crystaldecisions.sdk.exception.SDKException$InvalidArg: The argument has an invalid value null (FWM 02024)
at com.crystaldecisions.sdk.occa.security.internal.SecuritySession.decodeSerializedSession(SecuritySession.java:938)
at com.crystaldecisions.sdk.occa.security.internal.SecuritySession.makeSessionHelper(SecuritySession.java:1014)
at com.crystaldecisions.sdk.occa.security.internal.SecuritySession.makeSession(SecuritySession.java:1006)
at com.crystaldecisions.sdk.occa.security.internal.SecurityFactory.makeSecuritySession(SecurityFactory.java:143)
at com.crystaldecisions.sdk.occa.security.internal.SecurityMgr.getSession(SecurityMgr.java:191)

which I think looks pretty good to start of with. It seems to know it's me, radicssm, that needs to be authenticated but then the dreaded The argument has an invalid value null (FWM 02024) I have seen this several times in the past in my previous life as a consultant and never truly got to the bottom of it. For the one SAP Note I can find on this error - 2386283- the solution is to trust the service account for delegation and that is definitely done already.

If I try https://dns_alias/BOE/BI I again just get the logon page but this time there is nothing in the Tomcat log to suggest that an authentication attempt is made at all. This is more worrying really as this is the case we really need to work. With nothing in the Tomcat log I really don't have much to go on. Not getting any clues from Fiddler. Any suggestions gratefully received.

Many thanks,

Mike

[BasicTek]

This error should start a new topic, in a nuitshell when you use client tools they interface directly with the OS and microsoft DNS. Since they work it means all the trust info is setup correctly. When you use the web/app it uses java to connect to AD which does not use local api's or DNS. Because of this you must spell everything out in the krb5.ini which may include capaths. see this KBA for "the rules" https://apps.support.sap.com/sap/support/knowledge/preview/en/1245178

-Tim

[mike-rs]

So,we are now live with SAP BI on Win2016 servers in Domain NEW. We went with the service account from Domain OLD and all in all it was a pretty smooth transition. (All users logging on to SAP BI are in Domain OLD)

But, internal policy and so on dictates that I need to switch to using a service account from Domain NEW and now the problems have started. Just considering manual Win AD authentication for now.

- My krb5.ini contains details of both domains. Configuration uses Domain NEW as the default domain

- I can log on to client tools such as Universe Designer with an account from Domain OLD and Domain NEW

- I can log on to BI Launchpad with an account from Domain NEW

- But, trying to log on to BI Launchpad wit an account from Domain OLD fails with 'The Active Directory Authentication plugin could not authenticate at this time...'

- In the BILaunchpad_trace.glf file I get 'GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)'

- I can find several hits on Fail to create credential. (63) - No service creds but nothing that is really helping me. Does the capaths entry in my krb5.ini file come in to play now? I have tried a couple of things here but it makes no difference to the error

Is there anything obvious that 'Fail to create credential. (63) - No service creds' usually points to that I am missing?

Many thanks,

Mike

[BasicTek]

All existing SPN's are "active" the SPN is essentially like an account alias used by kerberos applications. Our CMS uses an SPN that we define, which could follow our suggested format or you could use spaghetti/meatballs if you wanted to, as long as the SPN defined in our application matches the one on the account.

When performing SSO IE or your browser is the application, it uses the URL to generate the kerberos SPN request, so if you wanted to you can create 1000's of SPN's on any account, as long as they are unique, you could change the SPN in the CMC every day to use one of the thousands (although who would want to).

In regards to the browser it's not as simply, the browser will always use HTTP/FQDNURL to send the SPN request, you just have to ensure that the FQDNURL's being accessed have an SPN on the service account that is defined in the global.properties. As always this value cannot exist twice.

The microsoft setspn command can be used with the -x & -f switch to search for duplicate SPN's but it only checks 1 forest at a time. Duplicate SPN's across forests could be difficult to troubleshoot so I would just be careful not to create them

https://blogs.msdn.microsoft.com/saurabh_singh/2009/01/08/new-features-in-setspn-exe-on-windows-serv...

[mike-rs]

Hi Tim,
Many thanks for your continued input. I think the Kerberos mists are clearing for me, slowly!
I'm building my first server in the new domain today so will soon be at the stage where I can just try things out. I'm quite intrigued now and will hopefully have the luxury of enough time to try setting up SSO with both the old and a new service account.

Best regards,

Mike

[BasicTek]

When a client hits a URL that is setup for kerberos SSO it will query for an SPN

example is https://mylbfqdnurl.com:port/BOE/BI

the accompanying SPN request would be for HTTP/mylbfqdnurl.com

as you can see from the example, HTTP/HTTPS is irrelevant, the port is irrelevant, the only value used to derive the SPN is from the URL itself (in this case mylbfqdnurl.com). The HTTP (called the principal) is due to the fact that it's using HTTP protocol.

That value must be unique because if it exists on more than 1 account kerberos will not know how to route the request. So it must be unique and used in the configuration of the BI server (global.properties and CMC).

So in that case both your existing or a new service account can be used but I would find it safer to use a new one if the environment is clean because multi-forest issues can get very complicated to troubleshoot.

When users from the other forest hit the BI server with local SPN, the forest trust information will ensure that the SPN it is found, so with a 2 way trust that should be ok.

-Tim

[BasicTek]

If the BI system is new (meaning it doesn't already contain the old users) then I would create the configuration new following KBA 2629070. This ensures your environment is up to the latest security and compatible with 2016. Your old system was probably not setup this way and could present problems as you integrate current standards (such as AES, constrained delegation, etc).

https://apps.support.sap.com/sap/support/knowledge/preview/en/2629070.

At this point if the old domain is in the same AD forest as the 2016 then you can map everything in and it should work fine.

If the old domain is in a separate AD forest then you would want at least a single test user from the new domain just to get the SSO working, then ensure the proper trust is setup so you can map in users from the old domain (forest). Follow KBA 1323391 for that.

https://apps.support.sap.com/sap/support/knowledge/preview/en/1323391

Now to avoid conflicts with the old service account and the new one, you must ensure the SPN's on the old account are not the same as the new one. The HTTP SPN's should be created in the format of HTTP/webappFQDNURL or HTTP/LBFQDNURL. If you are re-using the same web/app and load balancer those SPN's must be removed from the old account and added to the new one.

If you follow your plan I see possible issues with the following...

1. trust relationship between 2016 and 2012 domain (2 way forest, same forest or other?)
2. current encryption in old domain (is it already AES)
3. constrained delegation (is current account using constrained delegation or the default of delegated to any service?)
4. web/apps (are you going to redirect the old tomcat's at the new BI server or use the new ones) new ones would need new HTTP SPN's. Always test the direct tomcat 1st then the LB

-Tim

[mike-rs]

Hi Tim,

Many thanks for the response.

- I can now confirm that the 2012 and 2016 domains are indeed in separate forests but there is a 2 way forest trust between them
- constrained delegation is already set up on the OLD service account
- RC4_HMAC_MD5 encryption is in use in the OLD domain (this would still be supported in the NEW domain yes? AES is preferred but not essential?)

My vision is that I install SAP BI on each Win2016 server, standalone against Sybase SQL Anywhere. It can then be configured and tested in isolation. When ready I just use the Move Node option in the Central Configuration Manager to move the node in to the current cluster. No down time.

Thinking about the webapps, if I was to go with a new service account in the NEW domain. (I will be using new Tomcat instances on the Win2016 servers in any eventuality). I have an SPN HTTP/LBFQDNURL on the OLD service account already. I think I would also be able to create an SPN HTTP/LBFQDNURL on the NEW service account. SPNs must be unique in a forest but OLD and NEW are separate forests so I am not violating this.

This where my Kerberos knowledge runs out. When users hit http://LBFQDNURL/BOE/BI which SPN is 'active'? If I delete the SPN on the OLD service account is the SPN on the new one automatically picked up? I really can't picture what happens with Kerberos tickets around this area.

Thanks,

Mike