Skip to Content
avatar image
Former Member

Forward IoT data to smart data streaming

hello - i'm setting up a scenario as below.

1. sensor sends data to iot services on HCP

2. configured service to forward sensor data to smart data streaming on cal.sap.com

3. Registered service from cal.sap.com to HCP - as mentioned in http://help-legacy.sap.com/saphelp_hana_options_sds_adapt/helpdata/en/68/df4e6a1cbd43f1890ce58e012f4f7d/content.htm

4. Created project/workspace/etc and they are working fine

Issue - the sensor data doesn't seem flow to smart data streaming service.. not much useful info in the log file. Any suggestions.

Secondly - where do i see the data forwarded from iot service in cal (i configured HANA studio, but there is no data. i'm using SYSYEM schema) - any suggestion on which table holds the data.

thanks for your help

pandari

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

10 Answers

  • Jun 05, 2017 at 01:29 PM

    Here are some things to check:

    1. Are you sure that the SDS Streaming Web Service is running on your SDS server? Open the HANA Cockpit, go to the Streaming Cluster Configuration. If SWS isn't running, start it. See this blog post:

    https://blogs.sap.com/2016/04/13/using-the-streaming-web-service-sp11/

    2. Can you ping the host? Make sure you have connectivity to the host.

    3. Check the SWS config file (see tutorial above) to ensure that it's using 9093

    Add comment
    10|10000 characters needed characters exceeded

  • May 19, 2017 at 05:18 AM

    Hello Pandari, in spite you say the logs do not show something useful, we would still need your SAP Cloud Platform account ID and landscape you are working on to check them. You may either share them by replying on this answer or via direct message in case you do not want to share those publicly (I will send you an initial message to start the conversation). Regards, Anton

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    May 19, 2017 at 11:42 AM

    Hello Pandari,

    have you used the SDS studio plugin for Eclipse to create the project/workspace/etc.? If you change to the "SAP HANA Streaming Run Test" perspective, you can connect to your SDS, select your running input stream and open the stream viewer window. There you can see all incoming data. The same perspective also provide the possibility to test your stream with sample data.

    Whether the data is stored somewhere persistent in a table depends on your stream configuration project. If you use there a HANA output adapter for your input stream, you can specify the "landing table", where the data should be send to.

    These topics above are related to SDS. If you have further questions there, please have a look in the official SAP SDS documentation

    https://help.sap.com/viewer/product/HANA_STREAMING/2.0.01/en-US

    Regarding your problems that data is not forwarded from SAP Cloud Platform Internet of Things Service to SDS, we need some more information. Have you created an SDS Processing Service in your MMS instance? The procedure doing that is described here: https://help.sap.com/viewer/7436c3125dd5491f939689f18954b1e9/Cloud/en-US/b81fd20c92574587ae745aa692f2976c.html

    The SDS project structure must match to your used message type as described in the "SDS project setup" section.

    Kind regards,

    Ralf

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    May 19, 2017 at 01:44 PM

    Thanks Anton - I will send you the account details directly.

    Thanks Ralf for your comments. Response to your comments:

    1. have you used the SDS studio plugin for Eclipse to create the project/workspace/etc.? - Yes, and i can see the test data.i'm able to connect to the server, etc.

    2. Whether the data is stored somewhere persistent in a table depends on your stream configuration project. - Thanks for the clarification. i thought by default the streaming data is stored in a table or somewhere.

    3. Have you created an SDS Processing Service in your MMS instance? - yes. i created the structure including the g_ fields. All the field names in the iot service is reflected in the streaming structure

    4. I will send you the log data.

    Thanks

    Pandari

    Add comment
    10|10000 characters needed characters exceeded

  • May 20, 2017 at 04:57 PM

    Hello Pandari, our monitoring tool and the logs show that the iotmms instance on your account is not operating normally. It lacks the connectivity to the underlying DB. Since "processing service" configurations are stored in the DB, this might be a potential issue why iotmms is unable to forward data to SDS. What type of the DB are you using? Is it a shared HANA or you have acquired the MDC instance? In case of MDC ensure that it is up and running (be aware that is stops automatically after some period of time). You might need to re-configure SDS processing service after you fix the DB binding issue. Regards, Anton

    Add comment
    10|10000 characters needed characters exceeded

    • Yes, i looked into them (also remotely) but according to the metrics, iotmms is not able to connect to the DB. If you say MDC is up and running, then check if data source binding to iotmms is there (maybe it needs to be re-binded because of the password change etc). I would also recommend to re-strart iotmms process and as suggested above, check the SDS configurations.

  • avatar image
    Former Member
    May 22, 2017 at 09:30 PM

    hello Anton - I'm getting the error below. This is in the iotmms default trace file.

    2017 05 22 21:23:47#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceHttpsConnection##anonymous#Delayed request processing for device '6115856b-7fec-4b94-b930-09704ecff342'#na#pXXXtrial#iotmms#web##SWS Token could not be retrieved. Response Code: 405|

    Additional info:

    FILE_TYPE:DAAA96DE-B0FB-4c6e-AF7B-A445F5BF9BE2 FILE_ID:1495477078755 ENCODING:[UTF8|NWCJS:ASCII] RECORD_SEPARATOR:124 COLUMN_SEPARATOR:35 ESC_CHARACTER:92

    HEADER_END 2017 05 22 18:18:07#+00#ERROR#com.sap.iotservices.mms.processing.ProcessingServiceRegistry##anonymous#localhost-startStop-1#na#p296796trial#iotmms#web##Could not retrieve the service mappings from the data base.java.io.IOException: Failed to find a corresponding processing service named 'sds' for device type '3041ca0ef3a86a9c736e' and message type 'e1b0068faba1203f6661'.

    21:23:47#+00#ERROR#com.sap.iotservices.mms.processing.sds.connection.StreamingWebServiceHttpsConnection##anonymous#Delayed request processing for device '6115856b-7fec-4b94-b930-09704ecff342'#na#p296796trial#iotmms#web##SWS Token could not be retrieved. Response Code: 405|

    Any thoughts on how to resolve this.

    Thx

    Pandari

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    May 29, 2017 at 08:57 AM

    Hello Pandari,

    it seems that your SDS is not configured properly. To ensure that, could you please send a POST request with an HTTP client (e.g. POSTMAN) to your SDS authorization service as described here in the SDS guide: https://apps.support.sap.com/sap/support/knowledge/preview/en/2344255

    Method: POST

    URL:

    https://<system>sws<account>.<landscape-suffix>:443/1/authorization

    Header:

    - Content-Type: application/json

    - Authorization: BASIC (with your username and password to access Smart Data Streaming)

    Body:

    [ { "privilege":"write", "resourceType":"stream", "resource":"<your workspace>/<your project>/<your stream>" } ]

    as far as I can see this from your configuration picture, the resource string should be "default/zstreamproj/ziotstream"

    If the response code is ...

    - 200 with the response [{"sws-token":"<token>"}], your SDS service seems to be configured correctly and we should investigate the interaction between IoT Service MMS and SDS in detail

    - 401, your credentials are not valid

    - 405, your SDS is not configured correctly. Maybe you have not enable the Streaming Webservice (SWS) or Web Services Provider (WSP) functionality - Than please follow the configuration instructions in the link provided above.

    Kind regards,

    Ralf

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      i tried https://40.68.97.67:9093/1/authorization , https://40.68.97.67:9093/1/version. tried with http just in case - got error "This page can't be displayed". I know this ip address is valid, as i can access http://40.68.97.67:8000/sap/devs/edition/index.html.

      Any thoughts?

      Thx

      Pandari

  • Jun 02, 2017 at 03:58 PM

    I would suggest first making sure that you can connect to your SDS server via the SDS streaming web service (SWS). You say that you have deployed this from CAL, so I'm assuming that what you have deployed is the HANA 1.0 SPS12 Developer Edition in either AWS or Azure. I would suggest using this tutorial to ensure that you are able to publish to an SDS project via SWS:

    https://www.sap.com/developer/tutorials/sds-rest-publish.html

    - Be sure that SWS is running on your SDS cluster

    - Use Postman or some other tool to try to connect to it. If successful, then see if you can configure MMS to publish to it.

    But now a word of caution: we have never tried publishing from MMS to an SDS instance that's running outside of SAP Cloud Platform. I don't know if you will run into e.g. firewall issues, etc.

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      still not working...pl see below error from default trace file in hanatrial.ondemand.com for iotmms application. pretty close to giving up :(

      2017 06 07 21:45:39#+00#ERROR#com.sap.iotservices.mms.auth.BasicAuthFilter##anonymous#http-nio-8041-exec-9#na#p296796trial#iotmms#web#p29XXXtrial#Unable to loginjavax.security.auth.login.CredentialNotFoundException: No user identifier is provided. at com.sap.security.auth.module.basic.PasswordLoginModule.handleUserName(PasswordLoginModule.java:320) at com.sap.security.auth.module.basic.PasswordLoginModule.login(PasswordLoginModule.java:105) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at com.sap.core.jpaas.security.auth.service.internal.LoginContext.login(LoginContext.java:101) at com.sap.iotservices.mms.auth.BasicAuthFilter.login(BasicAuthFilter.java:102) at com.sap.iotservices.mms.auth.BasicAuthFilter.doFilter(BasicAuthFilter.java:57) at com.sap.iotservices.mms.auth.BasicOAuthFilter.doFilter(BasicOAuthFilter.java:84) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615) at com.sap.core.jpaas.security.auth.service.lib.AbstractAuthenticator.invoke(AbstractAuthenticator.java:170) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) at com.sap.core.tenant.valve.TenantValidationValve.invokeNextValve(TenantValidationValve.java:168) at com.sap.core.tenant.valve.TenantValidationValve.invoke(TenantValidationValve.java:94) at com.sap.js.statistics.tomcat.valve.RequestTracingValve.invoke(RequestTracingValve.java:45) at com.sap.core.js.monitoring.tomcat.valve.RequestTracingValve.invoke(RequestTracingValve.java:27) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1770) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1729) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:805)

  • avatar image
    Former Member
    Jun 09, 2017 at 07:16 PM

    Ralf/Robert/Jeff- Any thoughts on my last comment.

    Thx

    Pandari

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Jun 13, 2017 at 08:08 AM

    Hey Pandari,

    do you still have this Error with your BasicAuthFilter Exception? It seems that your user session was terminated, perhaps an temporary issue with the landscape. Could you please describe the steps, how you ran into that situation?

    Regards,

    Ralf

    Add comment
    10|10000 characters needed characters exceeded

    • I'm a bit confused. Was your CAL session paused when you last tried to connect? That would certainly explain the session timing out.

      Can you provide more detailed steps of exactly what you tried in this particular test now that other issues have been resolved?