All Blogs in this Series -
AI Powered Invoice Management with SAP RAP and ABAP on Cloud
AI Powered Invoice Management with SAP RAP and ABAP on Cloud – Part 1
AI Powered Invoice Management with SAP RAP and ABAP on Cloud – Part 2
AI Powered Invoice Management with SAP RAP and ABAP on Cloud – Part 3
This is the continuation of the main blog. Please go through it for better understanding.
If you are coming from the reference from the initial blog. Congrats you might have already done the ABAP RAP developments. Lets jump to the point -
We need to create a Invoice Process class. Where we will modularize the functionality for API Calls and also use latest Class methods available to call external apis.
CLASS zsaba_inv_process DEFINITION
PUBLIC
FINAL
CREATE PUBLIC .
PUBLIC SECTION.
TYPES: inv_doc TYPE c LENGTH 10.
INTERFACES: if_oo_adt_classrun.
CLASS-DATA: out TYPE REF TO if_oo_adt_classrun_out.
TYPES: BEGIN OF ty_post_document_body,
fileName TYPE string,
format TYPE string,
rawData TYPE string,
END OF ty_post_document_body.
TYPES: BEGIN OF ty_line,
material TYPE string,
description TYPE string,
quantity TYPE string,
amount TYPE string,
unitprice TYPE string,
END OF ty_line.
TYPES: tt_items TYPE TABLE OF ty_line.
TYPES: BEGIN OF ty_extracted,
receivercontact TYPE string,
grossamount TYPE string,
comments TYPE string,
invoicedate TYPE string,
refpurchaseorder TYPE string,
END OF ty_extracted.
CLASS-METHODS: analyze_doc
IMPORTING iv_inv_doc TYPE inv_doc
EXPORTING et_items TYPE tt_items
es_header TYPE ty_extracted,
analyze_doc_dummy
IMPORTING iv_inv_doc TYPE inv_doc
EXPORTING et_items TYPE tt_items
es_header TYPE ty_extracted,
create_client
IMPORTING url TYPE string
RETURNING VALUE(result) TYPE REF TO if_web_http_client
RAISING cx_static_check.
PROTECTED SECTION.
PRIVATE SECTION.
CONSTANTS:
base_url TYPE string VALUE 'https://inv_<INV_RAP>
_rap.cfapps.XXXXXXXX.hana.ondemand.com/upload', "Prd-BASF Account
content_type TYPE string VALUE 'Content-type',
txt_content TYPE string VALUE 'plain/txt',
json_content TYPE string VALUE 'application/json; charset=UTF-8'.
ENDCLASS.
CLASS zsaba_inv_process IMPLEMENTATION.
METHOD if_oo_adt_classrun~main.
me->out = out.
analyze_doc( iv_inv_doc = '0002001321' ).
ENDMETHOD.
METHOD analyze_doc_dummy.
et_items = VALUE #( ( amount = '200' description = 'Test' material =
'1002' quantity = '10' unitprice = '10' )
( amount = '2000' description = '2nd Material'
material = '1003' quantity = '100' unitprice = '20' ) ).
es_header = VALUE #( comments = 'Test' grossamount = '2200' invoicedate
= '2023.04.27' receivercontact = '8013865207' ).
ENDMETHOD.
METHOD create_client.
DATA(dest) = cl_http_destination_provider=>create_by_url( url ).
result = cl_web_http_client_manager=>create_by_http_destination( dest ).
ENDMETHOD.
METHOD analyze_doc.
DATA: post_document_body TYPE ty_post_document_body.
" Create the body..
SELECT SINGLE attachment, mimetype, filename
FROM zinvtable_sab
WHERE invoice = @iv_inv_doc
INTO @DATA(ls_pdf_data).
TRY.
post_document_body = VALUE #( fileName = ls_pdf_data-filename
format = 'jpg'
rawData = ls_pdf_data-attachment ).
DATA(json_post) = xco_cp_json=>data->from_abap( post_document_body
)->apply(
VALUE #( ( xco_cp_json=>transformation->underscore_to_camel_case ) )
)->to_string( ).
DATA(url) = |{ base_url }|.
DATA(client) = create_client( url ).
DATA(req) = client->get_http_request( ).
req->set_text( json_post ).
req->set_header_field( i_name = content_type i_value = json_content
).
DATA(result) = client->execute( if_web_http_client=>post )-
>get_text( ).
DATA: ls_extracted_head TYPE ty_extracted.
DATA: lo_data TYPE REF TO data,
lo_head TYPE REF TO data,
lo_item TYPE REF TO data,
lo_items TYPE REF TO data.
FIELD-SYMBOLS: <lfs_data> TYPE any,
<lfs_header> TYPE any,
<lfs_items> TYPE any.
CALL METHOD /ui2/cl_json=>deserialize
EXPORTING
json = result
pretty_name = /ui2/cl_json=>pretty_mode-user
assoc_arrays = abap_true
CHANGING
data = lo_data.
ASSIGN lo_data->* TO <lfs_data>.
ASSIGN COMPONENT 'HEADER' OF STRUCTURE <lfs_data> TO <lfs_header>.
ASSIGN COMPONENT 'ITEMS' OF STRUCTURE <lfs_data> TO <lfs_items>.
IF <lfs_header> IS ASSIGNED.
lo_head = <lfs_header>.
ASSIGN lo_head->* TO FIELD-SYMBOL(<lfs_structure>).
IF <lfs_structure> IS ASSIGNED.
ASSIGN COMPONENT 'COMMENTS' OF STRUCTURE <lfs_structure> TO
FIELD-SYMBOL(<lfs_any>).
IF <lfs_any> IS ASSIGNED.
es_header-comments = <lfs_any>->*.
ENDIF.
ASSIGN COMPONENT 'GROSSAMOUNT' OF STRUCTURE <lfs_structure> TO
<lfs_any>.
IF <lfs_any> IS ASSIGNED.
es_header-grossamount = <lfs_any>->*.
ENDIF.
ASSIGN COMPONENT 'INVOICEDATE' OF STRUCTURE <lfs_structure> TO
<lfs_any>.
IF <lfs_any> IS ASSIGNED.
es_header-invoicedate = <lfs_any>->*.
ENDIF.
ASSIGN COMPONENT 'RECEIVERCONTACT' OF STRUCTURE <lfs_structure>
TO <lfs_any>.
IF <lfs_any> IS ASSIGNED.
es_header-receivercontact = <lfs_any>->*.
ENDIF.
ASSIGN COMPONENT 'REFPURCHASEORDER' OF STRUCTURE <lfs_structure>
TO <lfs_any>.
IF <lfs_any> IS ASSIGNED.
es_header-refpurchaseorder = <lfs_any>->*.
ENDIF.
ENDIF.
ENDIF.
DATA: ls_item TYPE ty_line.
IF <lfs_items> IS ASSIGNED.
lo_items = <lfs_items>.
ASSIGN lo_items->* TO FIELD-SYMBOL(<lfs_table>).
LOOP AT <lfs_table> ASSIGNING FIELD-SYMBOL(<lfs_line>).
ASSIGN COMPONENT 'AMOUNT' OF STRUCTURE <lfs_line>->* TO
<lfs_any>.
IF <lfs_any> IS ASSIGNED.
ls_item-amount = <lfs_any>->*.
ENDIF.
ASSIGN COMPONENT 'MATERIAL' OF STRUCTURE <lfs_line>->* TO
<lfs_any>.
IF <lfs_any> IS ASSIGNED.
ls_item-material = <lfs_any>->*.
ENDIF.
ASSIGN COMPONENT 'DESCRIPTION' OF STRUCTURE <lfs_line>->* TO
<lfs_any>.
IF <lfs_any> IS ASSIGNED.
ls_item-description = <lfs_any>->*.
ENDIF.
ASSIGN COMPONENT 'QUANTITY' OF STRUCTURE <lfs_line>->* TO
<lfs_any>.
IF <lfs_any> IS ASSIGNED.
ls_item-quantity = <lfs_any>->*.
ENDIF.
ASSIGN COMPONENT 'UNITPRICE' OF STRUCTURE <lfs_line>->* TO
<lfs_any>.
IF <lfs_any> IS ASSIGNED.
ls_item-unitprice = <lfs_any>->*.
ENDIF.
APPEND ls_item TO et_items.
ENDLOOP.
ENDIF.
CATCH cx_root INTO DATA(err).
ENDTRY.
ENDMETHOD.
ENDCLASS.
This Code is having reference to call external APIs and JSON Formatter codes in ABAP.
Using the same aspects, you can create your method for sent_for_payment.
SAP Document Information Extraction Service Configuration
Enable Document Information extraction services
Create service key for ‘Document Information Extraction Trial’.
SAP BTP NodeJS Express Server and CF Deployments( API for AI enabled Invoice Processes )
Create a basic npm server by using the below package file.
{
"name": "inv_image_scan",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"start": "node index.js",
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "Sabarna Chatterjee",
"license": "ISC",
"dependencies": {
"body-parser": "^1.20.2",
"btoa-atob": "^0.1.2",
"express": "^4.18.2",
"express-fileupload": "^1.4.0",
"image-to-pdf": "^2.0.0",
"nodemon": "^2.0.20",
"sap-cf-axios": "^0.4.8"
}
}Now create Index.js
var express = require('express');
var bodyParser = require('body-parser');
var app = express();
var PORT = 8080;
var lib = require('./lib');
// app.use(express.json());
app.use(bodyParser.json({limit: '500mb'}));
app.use(bodyParser.urlencoded({limit: '500mb', extended: true}));
app.post('/upload', async function (req, res) {
console.log('Upload Triggered')
await lib.upload(req, res);
});
app.get('/', async function (req, res) {
await lib.base(req, res);
});
app.listen(PORT, function (err) {
if (err) console.log(err);
console.log("Server listening on PORT", PORT);
});Now implement your code block in lib.js
const axios = require('axios')
var FormData = require('form-data');
var fs = require('fs');
const { resolve } = require('path');
var inv_ocr_json = require('./inv_ocr_config.json');
var inv_ocr_str = JSON.stringify( inv_ocr_json )
function loadSecret() {
return require('./.secret.doc_extract_inv.json') // The json file with the service key.
}
const _getDocAPIToken = async function(){
var secret_json = loadSecret();
var configToken = {
'headers': {
'Authorization': 'Basic ' + Buffer.from(secret_json.uaa.clientid + ':' +
secret_json.uaa.clientsecret).toString('base64')
}
}
var oauthURL = secret_json.uaa.url + '/oauth/token?grant_type=client_credentials';
const token = await _getToken(oauthURL, configToken);
return token;
}
const _document = async function (configPost) {
console.log('Posting Started');
return new Promise((resolve, reject) => {
axios(configPost)
.then(function (response) {
resolve(response.data);
console.log('I am here');
})
.catch(function (error) {
console.log('Error Happened');
console.log(error);
})
});
}
const _execDocPost = async function(filename) {
var secret_json = loadSecret();
var token = await _getDocAPIToken()
var data = new FormData();
data.append('file', fs.createReadStream(filename)); //'./invoice_2001321.pdf'));
data.append('options', inv_ocr_str );
var configPost = {
'method': 'post',
'url': secret_json.endpoints.backend.url + secret_json.swagger + 'document/jobs',
'headers': {
'Authorization': 'Bearer ' + token
},
'data': data
}
var postedData = await _document(configPost)
// console.log(postedData);
var response = {
"id": postedData.id,
"token": token
}
return response;
}
const _execDocGetLoop = async function(id,token) {
const delay = ms => new Promise(resolve => setTimeout(resolve, ms))
for(var i=0;i<30;i++){
console.log('Loop Iteration -> ', i)
var secret_json = loadSecret();
var configGet = {
'method': 'get',
'url': secret_json.endpoints.backend.url + secret_json.swagger + 'document/jobs/' + id,
'headers': {
'Authorization': 'Bearer ' + token
}
}
var result = await new Promise((resolve, reject) => { axios(configGet)
.then(function (response) {
resolve(response.data)
})
.catch(function (error) {
console.log(error);
}) });
if( result.status === "DONE"){
return result
}
else{
await delay(1000)
}
}
return result
}
const _getToken = async function (url, header) {
return new Promise((resolve, reject) => {
axios.get(url, header)
.then(response => {
resolve(response.data.access_token)
})
.catch(function (error) {
if (error.response) {
// Request made and server responded
console.log(error.response.data);
console.log(error.response.status);
console.log(error.response.headers);
} else if (error.request) {
// The request was made but no response was received
console.log(error.request);
} else {
// Something happened in setting up the request that triggered an Error
console.log('Error', error.message);
}
})
})
}
const _fileSave = async function fileSave(fileName,format,rawData){
console.log('my-filename---->', fileName);
require("fs").writeFile(fileName, rawData, 'hex', function(err) {
console.log(err);
return false;
});
return true;
}
module.exports = {
base: async function base(req, res) {
res.send(JSON.stringify({ 'uri': '/upload' }));
},
upload: async function base(req, res) {
var data = JSON.stringify(req.body);
var date = new Date();
function pad2(n) { return n < 10 ? '0' + n : n }
var ts = date.getFullYear().toString() + pad2(date.getMonth() + 1) + pad2( date.getDate()) + pad2(
date.getHours() ) + pad2( date.getMinutes() ) + pad2( date.getSeconds() )
var docData = {}
res.setHeader("Content-Type", "application/json");
var jsondata = JSON.parse(data);
console.log('my-filename---->', jsondata.filename);
try{
if( await _fileSave(jsondata.filename + ts + '.' +
jsondata.format,jsondata.format,jsondata.rawdata))
{
var response = await _execDocPost(jsondata.filename + ts + '.' + jsondata.format);
docData = await _execDocGetLoop(response.id, response.token);
console.log(docData.extraction.lineItems);
var lineitem = {
'material': '',
'description': '',
'quantity': '',
'amount': '',
'unitprice': '',
}
var docDataFinal = {
'header': { 'receivercontact': '', 'grossamount': '', 'invoicedate': '', 'comments': '' },
'items' : []
}
docData.extraction.headerFields.forEach(element => {
switch (element.name){
case 'receiverContact':
docDataFinal.header.receivercontact = element.rawValue;
break;
case 'grossAmount':
docDataFinal.header.grossamount = element.rawValue;
break;
case 'documentDate':
docDataFinal.header.invoicedate = element.rawValue;
break;
case 'paymentTerms':
docDataFinal.header.comments = element.rawValue;
break;
}
});
docData.extraction.lineItems.forEach( item =>{
lineitem = {};
item.forEach( column =>{
switch(column.name){
case 'description':
lineitem.description = column.rawValue;
break;
case 'netAmount':
lineitem.amount = column.rawValue;
break;
case 'quantity':
lineitem.quantity = column.rawValue;
break;
case 'unitPrice':
lineitem.unitprice = column.rawValue;
break;
case 'materialNumber':
lineitem.material = column.rawValue;
break;
}
})
if(!lineitem.material){
lineitem.material = ''
}
docDataFinal.items.push(lineitem);
});
// docData.receivercontact =
console.log(docDataFinal);
res.send(JSON.stringify(docDataFinal));
}
}
catch (error){
docData = {'process_status': "error"};
res.send(JSON.stringify({ 'message': docData }));
}
}
} Next is pushing your development to SAP Cloud Foundry. Use a manifest.yml and use cf8 push to deploy your code to SAP Cloud Foundry. Also you can deploy the code in Kyma Runtime environment.
---
applications:
- name: inv_<INV_RAP>_rap
random-route: false
path: ./
memory: 256M
buildpack: nodejs_buildpackIf you have followed the blog till now, you must have a better understanding now, how this program is skilled enough to process the Raw file and convert it into structured data using AI and OCR powered APIs.
- Configuration of Document Extraction Service in SAP BTP
- Creation of a NodeJS based wrapper interface and host it in SAP BTP
- Calling of the NodeJS API from the Custom Invoice Process Class - zsaba_inv_process->analyze_doc_dummy
Please visit Part 3.
- SAP Managed Tags:
