SAP Leonardo Machine Learning - Product Image Classification API

[gharbi79]

Update 26/10/2018 - Following the SAP Inside Track Barcelona 2018 which has taken place today and where I led a hands-on session to build this Fiori app, I'm pleased to share the source code and Hands-on instructions with the SAP Community (Github link). Thanks to Backoffice Associates for sponsoring this event.

 

SAP Leonardo is a holistic digital innovation system that seamlessly integrates future-facing technologies and capabilities into the SAP Cloud Platform.



 

SAP Leonardo represents a System of Intelligence which delivers capabilities to create, integrate and scale new intelligent applications making your business processes more capable and smart.

This Blog series is to describe and illustrate the technologies involved in SAP Leonardo:

• Machine Learning


• Internet of Things (IoT)


• Analytics


• Big Data


• Blockchain


• Data Intelligence


• Design Thinking

Technically, the Foundation of SAP Leonardo is a set of services and APIs which depending on their nature could be cloud-based on SAP Cloud Platform or on-premise based on SAP HANA Platform. These services could be consumed and integrated with your System of Record (S/4HANA, Hybris, SuccessFactors,  Concur, Fieldglass, Ariba …). With SAP Leonardo, all depends on your imagination to transform and innovate in your business and processes.

 

My first Blog is about SAP Leonardo Machine Learning and how Product Image Classification API could be leveraged to unlock the power of unclassified images. I will show you two scenarios where this API could be exploit:

• Scenario 1: Unclassified Images on a server


• Scenario 2: Unclassified Images on Internet

 

In the second scenario, I will show you how an intelligent Fiori Cloud-Based App is capable of combining Custom Google Search API and SAP Leonardo Product Classification API in order to get Instant Business-Value by classifying non-downloaded images retrieved from different sites.

 



 

Let’s start! ?

 

SAP Leonardo Product Image Classification API

The SAP API Business Hub is the central place where you can discover and consume digital content packages with APIs. These packages are classified by Supported Platforms, Vendors, Line of Business, Industries, Products and Countries.

For SAP Leonardo machine Learning you can find up to 14 artifacts to implement several functional services.



 

As you can see, Product Image Classification API classifies images into a fixed set of categories of products that are Common in eCommerce. There are more than 29 categories available such as rechargeable_batteries, smartphones, servers, mobile_device_chargers, networking_cables …

Note: The model of this service is based on Icecat Image Classifier. Icecat analyzes the performance of thousands of brands. Its statistics are based on billions of annual product data-sheets downloads by the tens of thousands connected ecommerce websites: online shops, ERP systems, comparison sites, purchase systems, rating portals, and other applications.

 

The SAP API Business Hub provides a good documentation and offers an API Endpoint to try out any of its services:



Tip: You need to have an SAP s-user and get an API Key in order to test the service.

 

For the Product Image classification API, the parameter files is required. You can upload an image and submit to SAP sandbox.



 

The response comes out with the probability scores for several possible categories, being the first one the most probable: in this case the image is 92,59% a printer or printers accessories.

 



Note: In real scenarios, you need to call these API’s from your applications. SAP provides Code Snippet in several languages. The code will depend also on the API type (REST, SOAP …).

 

Scenario 1 : Unclassified Images on a server

Your business generates hundreds of images every day. The pictures are sent to your server and need to be classified and exposed to your business partners.

• ABAP can read your images, perform a REST call to SAP Leonardo Product Image Classification API.

DATA: lo_http_client TYPE REF TO if_http_client.

DATA: response TYPE string.



"create HTTP client by url

"API endpoint for API sandbox 

CALL METHOD cl_http_client=>create_by_url

  EXPORTING

    url                = 'https://sandbox.api.sap.com/ml/prodimgclassifier/inference_sync'



  IMPORTING

    client             = lo_http_client

  EXCEPTIONS

    argument_not_found = 1

    plugin_not_active  = 2

    internal_error     = 3

    OTHERS             = 4.



IF sy-subrc <> 0.

  "error handling

ENDIF.



"setting request method

lo_http_client->request->set_method('POST').



"adding headers

lo_http_client->request->set_header_field( name = 'Content-Type' value = 'multipart/form-data' ).

lo_http_client->request->set_header_field( name = 'Accept' value = 'application/json' ).

"API Key for API Sandbox

lo_http_client->request->set_header_field( name = 'APIKey' value = 'XXXXXXXXXXXXXXXXXXXX' ).





"Optional parameters

"DATA: l_xstring TYPE xstring.

"DATA: binary_tab TYPE STANDARD TABLE OF x255.

"DATA: file_length type i.



"CALL FUNCTION 'GUI_UPLOAD'

  "EXPORTING

    "filename   = '<file_path>'

    "filetype   = 'BIN'

  "IMPORTING

    "filelength = file_length

  "TABLES

    "data_tab   = binary_tab.



"CALL FUNCTION 'SCMS_BINARY_TO_XSTRING'

  "EXPORTING

    "input_length = file_length

  "IMPORTING

    "buffer       = l_xstring

  "TABLES

    "binary_tab   = binary_tab

  "EXCEPTIONS

    "OTHERS       = 1.



"IF sy-subrc <> 0.

  "error handling

"ENDIF.



"lo_http_client->request->append_cdata(

  "`--undefined`

  "&& cl_abap_char_utilities=>cr_lf

  "&& `Content-Disposition: form-data; name="files"; filename="<file_name>";`

  "&& cl_abap_char_utilities=>cr_lf

  "&& cl_abap_char_utilities=>cr_lf ).

"lo_http_client->request->append_data( l_xstring ).

"lo_http_client->request->append_cdata(

  "cl_abap_char_utilities=>cr_lf 

  "&& `--undefined--` ).



CALL METHOD lo_http_client->send

  EXCEPTIONS

    http_communication_failure = 1

    http_invalid_state         = 2

    http_processing_failed     = 3

    http_invalid_timeout       = 4

    OTHERS                     = 5.



IF sy-subrc = 0.

  CALL METHOD lo_http_client->receive

    EXCEPTIONS

      http_communication_failure = 1

      http_invalid_state         = 2

      http_processing_failed     = 3

      OTHERS                     = 5.

ENDIF.



IF sy-subrc <> 0.

  "error handling

ENDIF.



response = lo_http_client->response->get_cdata( ).



WRITE: 'response: ', response.

Note: in the second scenario Javascript will be used to call the API.

• Store the results in your database, so the images are not required to be processed again


• Build an oData service in your SAP Gateway


• Then, you can consume this oData in your applications


• Or you can also expose the oData to your subscribers via SAP Cloud Platform API Management where you can control security policies and monetize your own APIs.

 

Scenario 2: Unclassified Images on Internet

Now, the images you want to classify are not physically in your server. They are on Internet!

You want to get images from different sites, potentially from web shops offering regular discounts, then classify them and get an overview of desired products with discounts.

To showcase this scenario, I have developed a Fiori App which retrieves images using a Google Custom Search Engine, posts them to SAP Leonardo Foundation and then classfies them in base of the Score and Label attributes.

The beauty of this app is that you don’t need to store any images, just display them in the browser and resend them to SAP Leonaro for classification!

 

Get Product Images URLs from the WEB

Google Custom Search enables you to create a search engine for your website, your blog, or a collection of websites. You can configure your engine to search both web pages and images.

Please read documentation and create your Custom Search engine.

 



Add your desired Sites to search according to your needs:



Note: Make sure you enable the Image Search option and take note of your CX key.

Tip: check advanced options to highly customize your search engine. On the right side you can test your Search engine and check the results.

Note: Fortunately Google Custom Search API supports jsonp datatype without worrying about cross-domain issues.

This code calls Google Custom Search API:

 

var sSearchString = this.getView().byId("searchString").getValue();

var sUrl = "https://www.googleapis.com/customsearch/v1?";

sUrl += "q=" + sSearchString;

sUrl += "&cx=XXXXXXXXXXXXXXXXX";

sUrl += "&key=YYYYYYYYYYYYYYYYYYYYYYY";

$.ajax({

	url: sUrl,

	jsonpCallback: 'processJSON',

	dataType: 'jsonp',

	success: function(result) {

		//Search Results from Google API

	},

	error: function(e) {

		//Handle errors

	}

});

 

Convert the images to base64

The Google API returns a JSON with the search results where you can get the image_link.



 

Then the following Javascript code converts the image_link into a base64  string and then builds the corresponding Blob.

//Helper function: Convert Image into base64

function convertImgIntoDataURL(googleURL, callback) {

	var oImg = new Image();

	oImg.crossOrigin = 'Anonymous';

	oImg.onload = function() {

		var oCanvas = document.createElement('CANVAS');

		var oCtx = canvas.getContext('2d');

		oCanvas.height = this.height;

		oCanvas.width = this.width;

		oCtx.drawImage(this, 0, 0);

		sDataURL = canvas.toDataURL();

		callback(dataURL);

	};

	oImg.src = googleURL;

}



//Convert Image into base64

var oBlob;

convertImgIntoDataURL(sGoogleURL, function(base64Img) {

	$("#img_id").attr("src", base64Img);

	oBlob = $("#img_id").imageBlob().blob();

});

Note: for commodity you can use this jQuery Plugin to convert into Blob.

 

Post base64 images to SAP Leonardo Product Image Classification API

The following Javascript code posts the previous Blob as a Form file using FormData interface to SAP Leonardo Product Image Classification API and gets back the Label and Score attributes.

var oForm = new FormData();

oForm.append("files", oBlob, "image.jpg");



var xhr = new XMLHttpRequest();

xhr.withCredentials = false;



//Request method and API endpoint for API sandbox 

xhr.open("POST", "https://sandbox.api.sap.com/ml/prodimgclassifier/inference_sync");



//Request headers

xhr.setRequestHeader("Accept", "application/json");



//API Key for API Sandbox

xhr.setRequestHeader("APIKey", "XXXXXXXXXXXXXX");



//Event listener 

xhr.addEventListener("readystatechange", function() {

			if (this.readyState === this.DONE) {

				var aLeonardo = jQuery.parseJSON(this.responseText);

			}

		});

Note: Unfortunately, SAP Leonardo APIs does not support jsonp datatype. You might need considering to create a destination in your SCP or enable CROSS-ORIGIN in your browser for tests purposes.

 

Classify images and put them into categories

 

The app performs a Google search:



 

Tip: You can add any of these parameters to make more accurate your search. Example: dateRestrict restricts results to URLs based on date.

 

When Leonardo Classification option is checked, it calls SAP Leonardo to classify the images:

 



 

And if Leonardo Categorization option is checked, it puts the images into dynamic Categories Tabs according to SAP Leonardo results:

 



 

I hope you have enjoyed this Blog! See you on the next one! ?