HANA HXE on GCP - XSA apps not working - Not able to reach the XS Controller API endpoint

[debanshu_mukherjee_1983]

Hi Team,

XSA not reachable on HXE (GCP).
I am working on HXE on GCP and it was all working well till today when I started receiving the following error:

hxeadm@hxehost:/usr/sap/HXE/HDB90> xs a

FAILED: Could not connect to controller on API endpoint https://hxehost:39030 See logs in /usr/sap/HXE/home/.xs_logs/xs_0.log for details. hxeadm@hxehost:/usr/sap/HXE/HDB90>

When I looked at the logs, it just said connection refused.
Below are the log entries:

PS: I also found that 98% of disc space is taken up by some block special file : /dev/sda1. I have no idea what's that and why it's taking up almost all of the space. Can that be a reason?

---------------------------------------------------------------------------------------------------------------------------------------------------

XS Log entries:

# XS command line client version: v1.0.99 (built 2018-12-10 12:35:58, codeline REL, SAP SE) # [2019-02-17 23:06:19:493] xs a [2019-02-17 23:06:20:182]-[CLI]-[error]-[.]-[.]: FAILED: Could not connect to controller on API endpoint https://hxehost:39030 com.sap.xs2rt.client.exceptions.ControllerConnectException: Could not connect to controller on API endpoint https://hxehost:39030 at com.sap.xs2rt.client.impl.operations.TargetOperations.getControllerInfo(TargetOperations.java:112) at com.sap.xs2rt.client.impl.operations.TargetOperations.getControllerInfo(TargetOperations.java:65) at com.sap.xs2rt.client.impl.operations.UaaOperations.refreshToken(UaaOperations.java:286) at com.sap.xs2rt.client.impl.command.Command.checkTokenRefresh(Command.java:629) at com.sap.xs2rt.client.impl.command.Command.canInvoke(Command.java:607) at com.sap.xs2rt.client.impl.command.TargetCommand.canInvoke(TargetCommand.java:100) at com.sap.xs2rt.client.impl.Main.main(Main.java:353) Caused by: com.sap.xs2rt.core.rest.client.RestException: Error executing request GET https://hxehost:39030/v2/info: java.net.ConnectException: Connection refused (Connection refused) (local port 54122 to address 127.0.0.1 (localhost), remote port 39030 to address 127.0.0.2 (hxehost.localdomain)) at com.sap.xs2rt.core.rest.client.RestResource.method(RestResource.java:778) at com.sap.xs2rt.core.rest.client.RestResource.get(RestResource.java:302) at com.sap.xs2rt.core.rest.client.RestResource.get(RestResource.java:255) at com.sap.xs2rt.core.generated.rest.resource.client.InfoClient.getInfo(InfoClient.java:57) at com.sap.xs2rt.client.impl.operations.TargetOperations.getControllerInfo(TargetOperations.java:92) ... 6 more Caused by: javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused) (local port 54122 to address 127.0.0.1 (localhost), remote port 39030 to address 127.0.0.2 (hxehost.localdomain)) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:284) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:278) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:753) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:229) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:752) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:419) at com.sap.xs2rt.core.rest.client.RestResource.method(RestResource.java:769) ... 10 more Caused by: java.net.ConnectException: Connection refused (Connection refused) (local port 54122 to address 127.0.0.1 (localhost), remote port 39030 to address 127.0.0.2 (hxehost.localdomain)) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:857) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:474) at sun.net.www.http.HttpClient.openServer(HttpClient.java:569) at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347) at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:390) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:282) ... 19 more

---------------------------------------------------------------------------------------------------------------------------------------------------

[lsubatin]

Hi, Debanshu,

Can you please share what HDB info shows as an output? Also, use cdtrace as hxeadm to move to the directory with traces and check logs called xscontroller*

A full disk could be the issue but you are listing the root directory there. /hana is a good place to start looking for trouble..

Best,

Lucia

[debanshu_mukherjee_1983]

Hi Lucia,

Thanks for the response and pointers.
Here's how the HDB info and version looks like. I also took some mem stats of /hana...I'm also trying to get to the trace files for issues :

HDB Info

Mem Stats of /hana

[debanshu_mukherjee_1983]

Hi,

I finally found the xscontroller.out file and it said that secure store is locked by 'hxeadm'.
The console also gave an option to fix this problem by doing a 'rm /hana/shared.....SSFS.LCK ' file and I did that but after that now..I am receving teh following error:

SSFS-4054: Record at file position 88286 does not start with preamble; maybe file is no RSecSSFs file (RC = -6)

Can you please suggest?

Regards
Debanshu

[lsubatin]

Hi, Debanshu,

The following could go wrong, so I would suggest you make a copy of the VM on GCP before doing this.

To fix error SSFS-4054: Record at file position xxxx does not start with preamble maybe file is no RSecSSFs file

As hxeadm, go with (it will ask for the password for SYSTEM in the SYSTEMDB):

rm -rf /usr/sap/HXE/SYS/global/xsa/security/ssfs/ 
XSA recover-ssfs --u SYSTEM
HDB reconf
XSA restart

Give it a little while for XSA to restart and try xs-admin-login again.

Fingers crossed....

Lucia.

[debanshu_mukherjee_1983]

Thank you very much Lucia!

Sorry for delayed response.. Got a bit occupied with other stuff at my new job. I've deleted the VM and have instantiated a news one to work with.. but your suggestions really helped.. Thanks.

Regards

Debanshu