Forward IoT data to smart data streaming

[Former Member]

hello - i'm setting up a scenario as below.

1. sensor sends data to iot services on HCP

2. configured service to forward sensor data to smart data streaming on cal.sap.com

3. Registered service from cal.sap.com to HCP - as mentioned in http://help-legacy.sap.com/saphelp_hana_options_sds_adapt/helpdata/en/68/df4e6a1cbd43f1890ce58e012f4...

4. Created project/workspace/etc and they are working fine

Issue - the sensor data doesn't seem flow to smart data streaming service.. not much useful info in the log file. Any suggestions.

Secondly - where do i see the data forwarded from iot service in cal (i configured HANA studio, but there is no data. i'm using SYSYEM schema) - any suggestion on which table holds the data.

thanks for your help

pandari

[JWootton]

Here are some things to check:

1. Are you sure that the SDS Streaming Web Service is running on your SDS server? Open the HANA Cockpit, go to the Streaming Cluster Configuration. If SWS isn't running, start it. See this blog post:

https://blogs.sap.com/2016/04/13/using-the-streaming-web-service-sp11/

2. Can you ping the host? Make sure you have connectivity to the host.

3. Check the SWS config file (see tutorial above) to ensure that it's using 9093

[Former Member]

Hey Pandari,

do you still have this Error with your BasicAuthFilter Exception? It seems that your user session was terminated, perhaps an temporary issue with the landscape. Could you please describe the steps, how you ran into that situation?

Regards,

Ralf

[Former Member]

yes, i still have this issue. my CAL session is paused.

thx

pandari

[RobertWaywell]

I'm a bit confused. Was your CAL session paused when you last tried to connect? That would certainly explain the session timing out.

Can you provide more detailed steps of exactly what you tried in this particular test now that other issues have been resolved?

[Former Member]

Ralf/Robert/Jeff- Any thoughts on my last comment.

Thx

Pandari

[JWootton]

I would suggest first making sure that you can connect to your SDS server via the SDS streaming web service (SWS). You say that you have deployed this from CAL, so I'm assuming that what you have deployed is the HANA 1.0 SPS12 Developer Edition in either AWS or Azure. I would suggest using this tutorial to ensure that you are able to publish to an SDS project via SWS:

https://www.sap.com/developer/tutorials/sds-rest-publish.html

- Be sure that SWS is running on your SDS cluster

- Use Postman or some other tool to try to connect to it. If successful, then see if you can configure MMS to publish to it.

But now a word of caution: we have never tried publishing from MMS to an SDS instance that's running outside of SAP Cloud Platform. I don't know if you will run into e.g. firewall issues, etc.

[Former Member]

yes, i deployed SDS on HANA developer edition on azure. i'm able to compile the project, webservice is enabled, http://13.81.0.52:9093/1/version - failed. i hv attached screen shots and trace file. please let me know if you have any other thoughts.

azure.jpg

get-service.jpg

trc-file-from-hana-admin.txt

[RobertWaywell]

The error you are getting strongly suggests that the SWS is not running. Have you worked through the 1st tutorial that Jeff mentioned which includes steps on configuring the SWS?

https://blogs.sap.com/2016/04/13/using-the-streaming-web-service-sp11/

[Former Member]

Hi Robert/Jeff,

My SWS is running, i also validated by invoking the url - /1/version, and 1/authorization. The issue i'm seeing now is HCP is throwing the authorization error below. I'm not sure, why HCP trial is using anonymous login.

This one when i used wss

2017 06 07 04:52:32#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-25#na#p296796trial#iotmms#web#pXXXtrial#No websocket connection available.|

This error when i used https

2017 06 07 04:53:08#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceHttpsConnection##anonymous#IoT MMS pool thread-25#na#p296796trial#iotmms#web#pXXXtrial#SWS Token could not be retrieved. Response Code: 405|

Service registration - tried it with both wss and https. (first i tried with wss, then deleted the registration, then re-registered with https)

https://iotmmsXXXXXXtrial.hanatrial.ondemand.com/com.sap.iotservices.mms/v1/api/http/processing

{ "deviceType":"3041ca0ef3a86a9c736e", "messageType":"e1b0068faba1203f6661", "processingServices":[ { "name":"sds", "properties":{ "mms.processing.sds.protocol":"wss", "mms.processing.sds.host":"52.174.242.241", "mms.processing.sds.port":30026, "mms.processing.sds.user":"SYSTEM", "mms.processing.sds.pwd":"XXXX", "mms.processing.sds.workspace":"default", "mms.processing.sds.project":"zstreamproj", "mms.processing.sds.input_stream":"ziotstream", "mms.processing.sds.proxy_enabled":"false" } } ] }

I also checked the HCP trial java application of iotmms, message forward configuration, i have entered the userid/password to the CAL streaming project.

Any thoughts?

Thanks

Pandari

[Former Member]

Hey Pandari,

as far as you leave the "trial" landscape of SAP Cloud Platform with your forward request, it could be possible to enter proxy settings in the SDS processing service.

Could you please try to set the mms.processing.sds.proxy_enabled to "true" and use the following settings:

Host: proxy-trial

Port: 8080

Kind regards,

Ralf

[Former Member]

still not working...pl see below error from default trace file in hanatrial.ondemand.com for iotmms application. pretty close to giving up 😞

2017 06 07 21:45:39#+00#ERROR#com.sap.iotservices.mms.auth.BasicAuthFilter##anonymous#http-nio-8041-exec-9#na#p296796trial#iotmms#web#p29XXXtrial#Unable to loginjavax.security.auth.login.CredentialNotFoundException: No user identifier is provided. at com.sap.security.auth.module.basic.PasswordLoginModule.handleUserName(PasswordLoginModule.java:320) at com.sap.security.auth.module.basic.PasswordLoginModule.login(PasswordLoginModule.java:105) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at com.sap.core.jpaas.security.auth.service.internal.LoginContext.login(LoginContext.java:101) at com.sap.iotservices.mms.auth.BasicAuthFilter.login(BasicAuthFilter.java:102) at com.sap.iotservices.mms.auth.BasicAuthFilter.doFilter(BasicAuthFilter.java:57) at com.sap.iotservices.mms.auth.BasicOAuthFilter.doFilter(BasicOAuthFilter.java:84) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615) at com.sap.core.jpaas.security.auth.service.lib.AbstractAuthenticator.invoke(AbstractAuthenticator.java:170) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) at com.sap.core.tenant.valve.TenantValidationValve.invokeNextValve(TenantValidationValve.java:168) at com.sap.core.tenant.valve.TenantValidationValve.invoke(TenantValidationValve.java:94) at com.sap.js.statistics.tomcat.valve.RequestTracingValve.invoke(RequestTracingValve.java:45) at com.sap.core.js.monitoring.tomcat.valve.RequestTracingValve.invoke(RequestTracingValve.java:27) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1770) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1729) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:805)

[Former Member]

Hello Pandari,

it seems that your SDS is not configured properly. To ensure that, could you please send a POST request with an HTTP client (e.g. POSTMAN) to your SDS authorization service as described here in the SDS guide: https://apps.support.sap.com/sap/support/knowledge/preview/en/2344255

Method: POST

URL:

https://<system>sws<account>.<landscape-suffix>:443/1/authorization

Header:

- Content-Type: application/json

- Authorization: BASIC (with your username and password to access Smart Data Streaming)

Body:

[ { "privilege":"write", "resourceType":"stream", "resource":"<your workspace>/<your project>/<your stream>" } ]

as far as I can see this from your configuration picture, the resource string should be "default/zstreamproj/ziotstream"

If the response code is ...

- 200 with the response [{"sws-token":"<token>"}], your SDS service seems to be configured correctly and we should investigate the interaction between IoT Service MMS and SDS in detail

- 401, your credentials are not valid

- 405, your SDS is not configured correctly. Maybe you have not enable the Streaming Webservice (SWS) or Web Services Provider (WSP) functionality - Than please follow the configuration instructions in the link provided above.

Kind regards,

Ralf

[Former Member]

hello ralf - i'm using SAP cloud appliance (cal.sap.com) + azure, and i have the ip address of SDS instance. so i constructed my url as https://<ip-address>:443//1/authorization, content json, basic authentication - user id/pass, and raw payload - [ { "privilege":"write", "resourceType":"stream", "resource":"default/zstreamproj/ziotstream" } ]

I got - "The requested url can't be reached". I'm able to access this url from eclipse and using browser.

i changed the port from 443 to 30026, i got "install self signed certificate" error..

any thoughts on how the url should look like.

thx

pandari

[RobertWaywell]

If you are trying to publish to a CAL instance, then the host name and port will be different than if you are working in HCP. For example, HCP handles the routing from a generic port 443 to the actual SWS port. The default port for websocket connections on the SWS itself is 9093. When trying to connect to the SWS on an Azure VM the address for the uri should be "https://<external or public ip address of the Azure host>:9093/..."

[Former Member]

Pandari, were you able to get the authorization token from the SDS in your CAL instance with the link Robert provided?
You can also check the version with "https://<external or public ip address of the Azure host>:9093/1/version". Do you get a response there?

[Former Member]

i tried https://40.68.97.67:9093/1/authorization , https://40.68.97.67:9093/1/version. tried with http just in case - got error "This page can't be displayed". I know this ip address is valid, as i can access http://40.68.97.67:8000/sap/devs/edition/index.html.

Any thoughts?

Thx

Pandari

[Former Member]

hello Anton - I'm getting the error below. This is in the iotmms default trace file.

2017 05 22 21:23:47#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceHttpsConnection##anonymous#Delayed request processing for device '6115856b-7fec-4b94-b930-09704ecff342'#na#pXXXtrial#iotmms#web##SWS Token could not be retrieved. Response Code: 405|

Additional info:

FILE_TYPE:DAAA96DE-B0FB-4c6e-AF7B-A445F5BF9BE2 FILE_ID:1495477078755 ENCODING:[UTF8|NWCJS:ASCII] RECORD_SEPARATOR:124 COLUMN_SEPARATOR:35 ESC_CHARACTER:92

HEADER_END 2017 05 22 18:18:07#+00#ERROR#com.sap.iotservices.mms.processing.ProcessingServiceRegistry##anonymous#localhost-startStop-1#na#p296796trial#iotmms#web##Could not retrieve the service mappings from the data base.java.io.IOException: Failed to find a corresponding processing service named 'sds' for device type '3041ca0ef3a86a9c736e' and message type 'e1b0068faba1203f6661'.

21:23:47#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceHttpsConnection##anonymous#Delayed request processing for device '6115856b-7fec-4b94-b930-09704ecff342'#na#p296796trial#iotmms#web##SWS Token could not be retrieved. Response Code: 405|

Any thoughts on how to resolve this.

Thx

Pandari

[Former Member]

further update. i deleted all the previous registrations and registered the smart data streaming service, with "wss" protocol, then i got the message below in "iotmms"

2017 05 22 22:06:10#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-2#na#pXXXtrial#iotmms#web##No websocket connection available.|

[Former Member]

Registed to iotmms using: https://iotmmspXXXXtrial.hanatrial.ondemand.com/com.sap.iotservices.mms/v1/api/http/processing

{ "deviceType":"3041ca0ef3a86a9c736e", "messageType":"e1b0068faba1203f6661", "processingServices":[ { "name":"sds", "properties":{ "mms.processing.sds.protocol":"wss", "mms.processing.sds.host":"52.166.240.XXX", "mms.processing.sds.port":30026, "mms.processing.sds.user":"SYSTEM", "mms.processing.sds.pwd":"XXXX", "mms.processing.sds.workspace":"default", "mms.processing.sds.project":"zstreamproj", "mms.processing.sds.input_stream":"ziotstream", "mms.processing.sds.proxy_enabled":"false" } } ] }

[anton_levin]

Dear Pandari, I still see that your iotmms instance is not able to connect to the underlying DB. Please, as stated above in my comment from May, 20th - bring iotmms to the operating state first. If it is not feasible with MDC instance, then I would suggest to switch back to shared HANA schema. Without underlying DB it is not possible for iotmms to operate with its processing services, since all configurations are stored there. Regards, Anton

[Former Member]

i reconfigured to use shared instance, now i'm getting the error below, which is similar to what i received before. Any other thoughts.

FILE_TYPE:DAAA96DE-B0FB-4c6e-AF7B-A445F5BF9BE2 FILE_ID:1495670532241 ENCODING:[UTF8|NWCJS:ASCII] RECORD_SEPARATOR:124 COLUMN_SEPARATOR:35 ESC_CHARACTER:92 COLUMNS:Time|TZone|Severity|Logger|ACH|User|Thread|Bundle name|JPSpace|JPAppliance|JPComponent|Tenant Alias|Text| SEVERITY_MAP:FINEST|Information|FINER|Information|FINE|Information|CONFIG|Information|DEBUG|Information|PATH|Information|INFO|Information|WARNING|Warning|ERROR|Error|SEVERE|Error|FATAL|Error HEADER_END 2017 05 25 00:31:35#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceHttpsConnection##anonymous#Delayed request processing for device '6115856b-7fec-4b94-b930-09704ecff342'#na#pXXXXXtrial#iotmms#web##SWS Token could not be retrieved. Response Code: 405|

[Former Member]

i restarted the iot services, and deleted the previous registration, and registered the service as wss and https, still it failed.

+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-3#na#p296796trial#iotmms#web##No websocket connection available.| 2017 05 25 00:51:46#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-3#na#p296796trial#iotmms#web##No websocket connection available.| 2017 05 25 00:52:03#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-3#na#p296796trial#iotmms#web##No websocket connection available.| 2017 05 25 00:52:14#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceWssConnection##anonymous#IoT MMS pool thread-3#na#p296796trial#iotmms#web##No websocket connection available.|

[anton_levin]

Hello Pandari, now monitoring tool indicates that the iotmms operates normally, thanks. But in the logs I see the following: "Failed to find a corresponding processing service named 'sds' for device type '304***36e' and message type 'e1b***661'". May you please post here a screenshot of your "T_IOT_PROCESSING_SERVICE_CONFIG" table? IoT MMS Cockpit -> Application Data (tile, top-left corner) -> T_IOT_PROCESSING_SERVICE_CONFIG table. (Please, mask user sensitive info if required). Regards, Anton

[Former Member]

Please see attached. iot-processing-service.jpg

[anton_levin]

Hello Pandari, our monitoring tool and the logs show that the iotmms instance on your account is not operating normally. It lacks the connectivity to the underlying DB. Since "processing service" configurations are stored in the DB, this might be a potential issue why iotmms is unable to forward data to SDS. What type of the DB are you using? Is it a shared HANA or you have acquired the MDC instance? In case of MDC ensure that it is up and running (be aware that is stops automatically after some period of time). You might need to re-configure SDS processing service after you fix the DB binding issue. Regards, Anton

[Former Member]

Hello Anton - thanks. did you get a chance to review the log files attached. i'm using MDC instance, and i made sure it's up and running.

thx

pandari

[anton_levin]

Yes, i looked into them (also remotely) but according to the metrics, iotmms is not able to connect to the DB. If you say MDC is up and running, then check if data source binding to iotmms is there (maybe it needs to be re-binded because of the password change etc). I would also recommend to re-strart iotmms process and as suggested above, check the SDS configurations.

[Former Member]

Thanks Anton - I will send you the account details directly.

Thanks Ralf for your comments. Response to your comments:

1. have you used the SDS studio plugin for Eclipse to create the project/workspace/etc.? - Yes, and i can see the test data.i'm able to connect to the server, etc.

2. Whether the data is stored somewhere persistent in a table depends on your stream configuration project. - Thanks for the clarification. i thought by default the streaming data is stored in a table or somewhere.

3. Have you created an SDS Processing Service in your MMS instance? - yes. i created the structure including the g_ fields. All the field names in the iot service is reflected in the streaming structure

4. I will send you the log data.

Thanks

Pandari

[Former Member]

Attached are the trace files.

Thx

Pandari

[Former Member]

Hello Pandari,

have you used the SDS studio plugin for Eclipse to create the project/workspace/etc.? If you change to the "SAP HANA Streaming Run Test" perspective, you can connect to your SDS, select your running input stream and open the stream viewer window. There you can see all incoming data. The same perspective also provide the possibility to test your stream with sample data.

Whether the data is stored somewhere persistent in a table depends on your stream configuration project. If you use there a HANA output adapter for your input stream, you can specify the "landing table", where the data should be send to.

These topics above are related to SDS. If you have further questions there, please have a look in the official SAP SDS documentation

https://help.sap.com/viewer/product/HANA_STREAMING/2.0.01/en-US

Regarding your problems that data is not forwarded from SAP Cloud Platform Internet of Things Service to SDS, we need some more information. Have you created an SDS Processing Service in your MMS instance? The procedure doing that is described here: https://help.sap.com/viewer/7436c3125dd5491f939689f18954b1e9/Cloud/en-US/b81fd20c92574587ae745aa692f...

The SDS project structure must match to your used message type as described in the "SDS project setup" section.

Kind regards,

Ralf

[anton_levin]

Hello Pandari, in spite you say the logs do not show something useful, we would still need your SAP Cloud Platform account ID and landscape you are working on to check them. You may either share them by replying on this answer or via direct message in case you do not want to share those publicly (I will send you an initial message to start the conversation). Regards, Anton