Integrate SAP Data Intelligence with Azure Databricks

[DeLo]

Dear SAP Data Intelligence community members,

One of our top customers is interested in orchestrating Azure Databricks workloads with SAP Data Intelligence. Data bricks is based on Spark and the request might be an easy one to fulfil since DI already supports Spark.

Has any of you or your customers ever tried that? Can you share some insights?

Thank you in advance for your help.

[former_member465983]

Hi Gianluca, it can be done by SAP HANA as well as SAP BW/4HANA.

SAP HANA Smart Data Access using SPARK SQL (DataBricks), afterward, a connection between SAP DI and SAP HANA/BW must be made.

https://help.sap.com/docs/SAP_BPC_VERSION_BW4HANA/dd104a87ab9249968e6279e61378ff66/60231af0b6254cd9b...

[chunhui]

You can use Aecorsoft's data integrator for SAP-Databricks integration.

[sarcangeli]

hi Gianluca, Data Intelligence uses Apache Livy to submit jobs to Spark clusters. As far I've seen, according to the official docs, on Azure that is supported by the HDInsight service, but not by the Databricks one.

For Databricks, you would rather have to use their proprietary API (https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/).