SAP IoT - File ingest

[gdillen]

Hi,

We have 2 different use cases for ingesting measurements/messages into SAP IoT:

- "Direct" connected devices publishing messages via MQTT
- "File based" ingest (=not direct connected devices): sensor devices buffer measurement readings send them on regular intervals to e.g. an SFTP/FTPS server. These measurements/files on the SFTP/FTPS server are on regular base picked up and should be ingested into the SAP IoT platform.

The first use case is perfectly clear to me, however for the second use case what's the recommended way to go?

Thanks.

Guy

[former_member701854]

Dear Jan Patrick,

Thanks.

The first option would definitely work for me. This was the option I already had in mind but didn't know it was a common practice for the use cases I described (file -> SAP IoT).

Best regards,

Guy

[former_member259476]

Dear Guy,

thanks for sharing more background info.

Based on my understanding of your requirements and current SAP IoT functionality, I think developing a small custom component that acts as a "File-to-SAP IoT protocol bridge" might be a working solution in your case. This custom component needs access to the file drop-off location, have an understanding of how to read the file contents and how to covert them into a supported message/payload format for SAP IoT Connectivity. The actual ingestion can then happen centrally via Cloud Gateways REST or MQTT, or alternatively via Edge Platform REST or MQTT. Usage of Router devices might be interesting in this case.

Based on my experience, it's not uncommon to use such custom "read-convert-send style" protocol bridges running in a central place in the backend in IoT projects in cases where standard ingestion interfaces cannot be connected directly.

An alternative might be to utilize the Edge Platform custom adapter SDK to develop such a component and run it based on SAP IoT Edge Platform.

Would either of those two options work for you?

Thanks,
Jan Patrick

[gdillen]

Dear Jan Patrick,

Thanks for your answer.

1. Do I understand correctly that there are two different categories of devices (category 1 which is able to send messages via MQTT and category 2 which can only send files to an FTP service)? In other words: There are no devices that are planned to use both ingestion options in parallel, correct?

That's indeed correct. It mostly depends on what sensor providers can offer as connectivity options.

2. In case there are those two categories of devices: What prevents category 2 devices to send measurements via MQTT instead of writing them to a file and upload them to an FTP server? Lack of direct connectivity to the SAP IoT Cloud Gateway MQTT? If yes, have you considered using SAP IoT Edge Platform as a local edge gateway?

Indeed mainly lack of direct connectivity options from the sensor providers. I still see a lot of sensor providers in industry offering sensors based on legacy connectivity options: SFTP, FTPS and even unsecure FTP. I have the impression for a lot of them connectivity (and even security) is an afterthought. Not to blame them but a lot of them are already for ages in business long before the IIoT/IoT "hype" started. Additionally they are mainly concerned with the precision their sensors can offer. SAP IoT Edge is also not possible in a lot of use cases where sensors are installed in the field.

3. In your question you describe a component that regularly picks up the files from the FTP server. Is that component in place already? What does it do with the files today? Have you considered extending this component such that it converts the file contents into MQTT messages and sends them on behalf of the individual devices?

That component is indeed already in place and indeed that's also an option I already considered and even did: FTP -> serverless function -> SAP IoT (over MQTT). However, my concern is what is the additional benefit of going via an MQTT broker when the data is already there to process up the analytics pipeline. MQTT is in fact a protocol to be used by constrained devices directly connected to it. When the devices are already publishing the data to an SFTP/FTPS server this (constrained) advantage is gone.

4. Are you able to share some numbers here with respect to the min./max./avg. size of the files, total number of devices that upload files to the FTP server and the frequency in which they upload the files (e.g. once per hour).

Data loggers read mostly every 10-15 minutes and send the measurements mostly every 3-4 hours (always depending on the use case/project). A quick calculation learns between 12-24 measurements per (sensor) file. The number of devices is also dependent on the use case/project: this can be 10 devices but also 1000 devices.

Hope this gives some clarification.

Best regards,

Guy

[former_member259476]

Dear Guy,
thanks a lot for your question.

Let me ask some things such that I can understand your second use case "File based ingest" a bit better:

1. Do I understand correctly that there are two different categories of devices (category 1 which is able to send messages via MQTT and category 2 which can only send files to an FTP service)? In other words: There are no devices that are planned to use both ingestion options in parallel, correct?

2. In case there are those two categories of devices: What prevents category 2 devices to send measurements via MQTT instead of writing them to a file and upload them to an FTP server? Lack of direct connectivity to the SAP IoT Cloud Gateway MQTT? If yes, have you considered using SAP IoT Edge Platform as a local edge gateway?

3. In your question you describe a component that regularly picks up the files from the FTP server. Is that component in place already? What does it do with the files today? Have you considered extending this component such that it converts the file contents into MQTT messages and sends them on behalf of the individual devices?

4. Are you able to share some numbers here with respect to the min./max./avg. size of the files, total number of devices that upload files to the FTP server and the frequency in which they upload the files (e.g. once per hour).

Thanks a lot and best regards,
Jan Patrick