Is it possible to HANA DB to read data from Azure Data Lake?

[Boyan_P]

Hello,

We have BW4HANA On-Premise Installation wich we want to feed with data stored at the Azure Data Lake.

We searched for suitable documentation but didn`t find such.

Is it possible to HANA DB to read data directly from Azure Data Lake? Or we have to use file export from Azure and Import at BW4?

If its possible, what connection we should use? SDI or SDA, or may be somethig other?

The BW4 is with the following components version:

Software Component <II> Release Support Package <==> Support Package Level <==> Description

• BW4CONT <II> 200 <II> SAPK-20003INBW4CONT <II> 3 <II> SAP BW4 HANA Content Addon
• BW4CONTB <II> 200 <II> SAPK-20003INBW4CONTB <II> 3 <II> SAP BW4 HANA Content Basis Addon
• DW4CORE <II> 200 <II> SAPK-20004INDW4CORE <II> 4 <II> DATA Warehouse
• HDB <II> 2.00.046.00.1581325702 <II> (fa/hana2sp04)

Thanks in advance for the support.

Boyan.

[RolandKramer · Accepted solution]

Hello Boyan

At this time, you will find no Solution to connect the ADL directly to SAP BW/4HANA.

This is the time, were SAP Data intelligence comes into the game. See the Blog - Unified Data Integration for SAP. Especially with SAP BW/4HANA SP04 onwards, the Integration between SAP Data Intelligence and SAP BW/HANA is even more deeper.

Best Regards Roland

[werner_daehn]

There are multiple options to do that:

1. Spark Connector: As the Azure Data Lake can be read with Spark, the Spark Connector of Hana SDA can be used.
2. SDI File Adapter: Supports reading from HDFS and hence from Azure Data Lake.
3. Parquet Adapter: I have built a Parquet adapter to read files. https://github.com/rtdi/HanaSDIParquetAdapter
4. Build your own SDI Adapter.

As you see, the most important question is what format you store the data in the Data Lake. The Azure Data Lake is like a File System.

In case you want a custom tailored adapter, please ping me. I can whip together one within a few days.

[Boyan_P]

Hello Werner,

Thank you for your response. You are suggesting very interesting possibilities. Also you are right that I haven't mentioned how the data is stored. It's SQL DB so option 2 - reading from HDFS should be considered as not applicable.

Option 4 or any kind of external development are not at the scope at this stage.

However this BW4 is not a productive but test system and we decided to check what standard opportunities we have in order to transfer data between Azure and HANA. Actually I've never imagined that implementation of such connection could be such a complex task.

I'll check how the things looks like in case of option 1 and 3.

Once again I want to thank you for your advice.

Regards,

Boyan.

[werner_daehn]

@b.punchev If the source is an Azure SQL DB, then I have a ready to run solution for you. For a large customer I have built such adapter so they can get data from Azure SQL DB into BW (Hana) as well.

If you like I can provide you with the adapter and the source code.

https://blogs.sap.com/2020/06/03/hana-sdi-realtime-adapter-for-azure-sql-db-and-sql-server/

[Boyan_P]

Thank you for you support @wdaehn . I really appreciated it.
Ill have in mind your proposal for the adapter but unfortunately at this stage I wont receive budget for any custom features, for sure.

PS.: Sorry for the late response.

Regards,
Boyan.

[former_member617154]

Hello Werner,

If I need to connect to unmanaged table on Azure Data lake, can we do that ?

If yes, which connector/adapter to use ?

Link to understand unmanaged table on Azure Data lake :

https://docs.microsoft.com/en-us/azure/databricks/data/tables

Thanking you meanwhile.

Regards,

Mihir Kiri

[werner_daehn]

sap_user99 Reading about it, it says the tables are registered in Hive. Hence I would assume the SDI provided Hive adapter will do the trick. Might be worth a try.

[Boyan_P]

Thank you Rolad.

Ill check and try to find the right way.

Boyan.