This blog post is part of a series of technical enablement sessions on SAP BTP for Industries. Check the full calendar here to watch the recordings of past sessions and register for the upcoming ones! The replay of the session related to this blog post and all the other sessions is available here.
Authors: cesare.calabria, yatsea.li and kristian.papai
Disclaimer: The source code of the sample solutions used in this blog post series are delivered as an SAP example under Apache 2.0 license with AS-IS manner. No SAP support available.
Pipelines are widely recognized as among the most pragmatic and economically efficient means of transporting hazardous and flammable substances in Oil & Gas industry. With a staggering network that spans over 2 million miles, pipelines have been established across 120 countries worldwide. One of the most significant challenges for Oil & Gas companies is pipeline corrosion caused by the corrosive and abrasive elements within the oil and gas they transport, which can lead to operational disruption, economic loss, severe environmental damage, safety issue etc. Shockingly, between 15% to 25% of significant pipeline incidents can be attributed to corrosion, which incurs annual damages of 1.4 billion USD.
Thereby, the depth of pipeline corrosion needs to be constantly assessed and monitored with high accuracy, cost effectiveness and timeliness in prevention of such severe loss and damages. More importantly, be capable of predicting the trend of pipeline corrosion depth and planning the maintenance of pipeline optimally to shift from reactive to predictive maintenance of pipelines, this results in pipeline operational efficiency substantial cost savings by reducing downtime, minimizing unnecessary maintenance, and extending the lifespan of pipeline.
Next, let's delve into some fundamentals of Pipeline Corrosion Assessment and explore various assessment methods.
In fact, there are also various other assessment methods and technologies for different situations, such as electromagnetic testing, radiographic testing, acoustic emission testing etc. In real-life, there is not one assessment method fit for all, assessment methods are often combined to achieve the goals of high accuracy, cost effectiveness, and multi-purpose detection.
With that ground knowledge about pipeline corrosion, now let have a look at our storyline of a fictional German oil company named S.A. Oil GmbH for their journey to embrace AI-powered Pipeline Corrosion Analysis:
After implementing the AI-powered Pipeline Corrosion Analysis, S.A. Oil gains several significant business benefits, such as:
Next, I would like to walk you through S.A. Oil’s journey of addressing their pipeline corrosion challenge.
Here you have a demo video of the AI-powered pipeline corrosion analysis solution:
https://youtu.be/DcCeof1M0eg
The solution utilized by S.A. Oil involves leveraging SAP S/4HANA Cloud for Oil & Gas as its Cloud ERP backend for critical operational control, alongside SAP Asset Performance Management, effectively optimizing asset management since January 1, 2022. This integration aids in refining maintenance strategies, assessing and enhancing asset performance, and aligning with their sustainability goals.
SAP Asset Performance Management is widely adopted within the oil and gas sector and is an integral part of the SAP Industry Cloud portfolio.
Aside from its core asset management functions, the solution offers numerous features facilitating predictive maintenance planning. S.A. Oil seeks to expand the capabilities of SAP Asset Performance Management with additional requirements, notably in developing proprietary machine learning algorithms to predict pipeline corrosion based on historical data. Moreover, the machine learning model requires adaptation over time, aligning with the latest data acquired through smart pigs and field technicians employing ultrasonic testing. Continual delivery of new AI models and predictive analyses for maintenance purposes is essential to S.A. Oil's operations.
Let’s have a close look into the functional aspects of the solution involving five key personas:
The reliability engineer oversees the management of pipeline master data and performs health checks within SAP Asset Management to ensure equipment functionality without failures. Additionally, they manage the connection of smart pigging sensors to the IoT gateway, gathering the captured measurements.
Field technicians utilize a custom-built CAP application to record physical visits, executed maintenance tasks, and on-site observations.
Data scientists employ collected data to train AI models and develop new machine learning models.
This role involves the deployment and management of newly created machine learning models.
The solution integrates an end-to-end monitoring system, consolidating all measures in a centralized platform. It aids in identifying and addressing anomalies, managed by the maintenance planner.
Relevant data is drawn from multiple systems, encompassing asset master data, maintenance planning, and cost data sourced from SAP S4/HANA Cloud. Additionally, IoT sensor data from Smart Pigs, pipeline assessment data recorded via Ultrasonic Testing by field technicians using the Pipeline Management App, and external sources like weather data are incorporated.
All this data eventually flows into the SAP HANA Cloud through the Data Lake. The acquisition of data for machine learning is efficiently managed via a CAP multi-tenant application, namely the Pipeline Management App.
Once acquired, this data forms the foundation for training a machine learning model within SAP AI Core and AI Launchpad. The model is developed to predict the pipelines corrosion depth. These predictions are stored in SAP HANA Cloud and further analyzed and visualized through an interactive geomap in SAP Analytics Cloud.
Field technicians utilize a mobile app to explore specific pipe Key Performance Indicators (KPIs) and add new corrosion measurements using Ultrasonic testing. The ML model's performance is continuously monitored against these new field measurements.
Over time, the introduction of new assessment data introduces variance in the model, leading to a decline in the performance of the pipeline corrosion ML model. The objective is to automate operational processes, ensuring continuous monitoring of the corrosion prediction model's performance. This involves auto-triggering model retraining with new datasets and deployment based on predefined performance metric rules. To accomplish this, they require the necessary infrastructure and tools to efficiently execute these automation tasks.
The AI Core is as an ideal solution for this scenario, providing a runtime environment capable of executing these automation tasks. Notably, it offers scheduling features and manual execution capabilities via AI Launchpad. Leveraging S.A. Oil's existing AI Core and AI Launchpad infrastructure appears highly sensible for running and managing its MLOps pipelines—a pivotal step towards implementing this approach in our Proof of Concept.
The proposal involves the use of three AI Core service instances. However, there is an opportunity to optimize landscape costs by deploying a single AI Core instance for three different purposes, utilizing the AI Core's resource groups.
Let's now explore various alternatives in constructing this architecture and understand the rationale behind choosing specific components and technologies.
Option 1 – SAP Data Intelligence Cloud:
Data Intelligence offers an alternative for running MLOps pipelines, providing a built-in scheduler and monitoring dashboard. However, its adoption would be justified only if S.A. Oil already integrates SAP Data Intelligence Cloud in their existing system landscape. Otherwise, investing solely for MLOps for AI Core might be challenging for S.A. Oil.
Option 2 – SAP Build Process Automation:
This tool is another alternative that offers a visual interface for running MLOps pipelines, including a scheduler and process automation capabilities. Similarly, its adoption would make sense only if S.A. Oil already incorporates SAP Build Process Automation in their current system landscape. Otherwise, justifying an investment exclusively for MLOps in AI Core might pose challenges.
Option 3 – Custom CAP Application on Cloud Foundry:
As an alternative approach, S.A. Oil's MLOps team could opt to develop a complete CAP (Cloud Application Programming Model) application on SAP BTP's Cloud Foundry runtime. This application could manage their MLOps activities specifically related to the Pipeline Corrosion Predictive Model within SAP AI Core. However, it's crucial to acknowledge that this option would require additional effort to maintain the CAP application and might incur added costs associated with SAP BTP's Cloud Foundry runtime.
Option 4 – Implementing the MLOps pipelines via Project Piper and Jenkins:
Jenkins, known for its proficiency in CI/CD, also presents a viable option for orchestrating and automating end-to-end MLOps processes in SAP AI Core, leveraging both Jenkins pipelines and the SAP AI Core SDK. Jenkins can help to streamline various workflows throughout ML lifecycle stages: from model training and validation to deployment and monitoring. Its seamless integration with version control systems facilitates continuous integration and continuous deployment (CI/CD) for ML models.
While SAP Continuous Integration and Delivery exists, it is not able to address the highly customizable tasks of MLOps for SAP AI Core. For easy setup of such highly customizable continuous delivery for SAP technologies, SAP introduces tooling for continuous delivery in project "Piper" integrated with Jenkins. This integration is applicable to projects on both the SAP Business Technology Platform (BTP) and SAP on-premise platforms, so is the MLOps of SAP AI Core.
Utilizing Project “Piper” in combination with Jenkins for MLOps ensures efficient workflows, automated model development, improved collaboration among team members, and consistent model deployment, ensuring the reliability and scalability crucial for ML applications. However, this necessitates a Jenkins infrastructure, which is justifiable if you seek to centrally manage all CI/CD workloads, be it software development or MLOps, within the same Jenkins environment.
In conclusion, the implementation of AI-powered Pipeline Corrosion Analysis emerges as a transformative solution for the oil and gas industry. By harnessing advanced technologies like machine learning models integrated with SAP platforms such as AI Core, businesses like S.A. Oil can proactively predict pipeline corrosion, mitigate risks, and optimize maintenance strategies. To ensure the reliability and performances for this AI-powered Pipeline Corrosion Analysis, Automated MLOps is required, we have also explored the options of MLOps infrastructure, such as the use of AI Core, AI Launchpad, and Jenkins together with Project “Pipe” etc. for orchestrating and managing MLOps pipelines efficiently. In next blog post, we will dive into the implementation detail of AI-powered Pipeline Corrosion Analysis.
Authors: cesare.calabria, yatsea.li and kristian.papai
Disclaimer: The source code of the sample solutions used in this blog post series are delivered as an SAP example under Apache 2.0 license with AS-IS manner. No SAP support available.
Introduction
Pipeline in Oil and Gas
Pipelines are widely recognized as among the most pragmatic and economically efficient means of transporting hazardous and flammable substances in Oil & Gas industry. With a staggering network that spans over 2 million miles, pipelines have been established across 120 countries worldwide. One of the most significant challenges for Oil & Gas companies is pipeline corrosion caused by the corrosive and abrasive elements within the oil and gas they transport, which can lead to operational disruption, economic loss, severe environmental damage, safety issue etc. Shockingly, between 15% to 25% of significant pipeline incidents can be attributed to corrosion, which incurs annual damages of 1.4 billion USD.
Thereby, the depth of pipeline corrosion needs to be constantly assessed and monitored with high accuracy, cost effectiveness and timeliness in prevention of such severe loss and damages. More importantly, be capable of predicting the trend of pipeline corrosion depth and planning the maintenance of pipeline optimally to shift from reactive to predictive maintenance of pipelines, this results in pipeline operational efficiency substantial cost savings by reducing downtime, minimizing unnecessary maintenance, and extending the lifespan of pipeline.
Pipeline Corrosion Assessment Methods
Next, let's delve into some fundamentals of Pipeline Corrosion Assessment and explore various assessment methods.
- Ultrasonic Testing (UT): Sending high-frequency sound waves into the pipeline material and measuring the time it takes for the waves to return. This data is used to calculate the thickness of the material and detect corrosion. This technology has been in use for decades. It is non-disruptive to pipeline operations, offer high accuracy, and are relatively cost-effective.
- In-line Inspection (ILI) Tools (e.g., smart pigs): Smart pigging technology is used to inspect pipeline interiors. These tools are called “pigs.” equipped with various sensors, inserted into and travel alone the pipeline. Pigs detect and measure pipeline conditions such as corrosion, pressure, temperature, geometry, and wall thickness etc.
In fact, there are also various other assessment methods and technologies for different situations, such as electromagnetic testing, radiographic testing, acoustic emission testing etc. In real-life, there is not one assessment method fit for all, assessment methods are often combined to achieve the goals of high accuracy, cost effectiveness, and multi-purpose detection.
Storyline of S.A. Oil GmbH
With that ground knowledge about pipeline corrosion, now let have a look at our storyline of a fictional German oil company named S.A. Oil GmbH for their journey to embrace AI-powered Pipeline Corrosion Analysis:
- Their pipeline network spans 800 KMs from the North East to the South West of Germany.
- They've been combining Smart Pigging and Ultrasonic Testing for periodical pipeline inspections since January 1, 2013. Smart pigging can identify general corrosion status and trends along the pipeline, while Ultrasonic Testing can assess critical areas in greater detail.
- Over the past decade, they’ve collected a large amount of historical data, including timestamp, temperature, pressure, pH value, corrosion depth, and more.
- Starting from January 1, 2022, they have implemented SAP Asset Performance Management for effective Asset Management of their pipelines.
- In addition, they've been harnessing the power of Machine Learning to predict pipeline corrosion with these historical data, paving the way for Predictive Maintenance of pipeline.
After implementing the AI-powered Pipeline Corrosion Analysis, S.A. Oil gains several significant business benefits, such as:
- Cost Savings: AI-driven corrosion analysis helps S.A. Oil optimize maintenance schedules. Shifting from reactive to predictive maintenance. This results in substantial cost savings by reducing downtime, minimizing unnecessary maintenance, and extending the lifespan of pipelines.
- Enhanced Safety: AI-powered Pipeline Corrosion Analysis significantly improves safety for the field technician in the S.A. Oil by enabling early detection and prediction of corrosion issues. This proactive approach helps prevent pipeline failures and potential leaks, reducing the risk of environmental damage, injuries, and costly shutdowns.
- Improved Operational Efficiency: The ML model not only can predict the pipeline corrosion depth accurately and timely, but also provide actionable insights to operators. This leads to more efficient maintenance schedule of pipelines, increased asset utilization, and improved overall performance.
Next, I would like to walk you through S.A. Oil’s journey of addressing their pipeline corrosion challenge.
Solution Demo
Here you have a demo video of the AI-powered pipeline corrosion analysis solution:
https://youtu.be/DcCeof1M0eg
Solution and Architecture
The solution utilized by S.A. Oil involves leveraging SAP S/4HANA Cloud for Oil & Gas as its Cloud ERP backend for critical operational control, alongside SAP Asset Performance Management, effectively optimizing asset management since January 1, 2022. This integration aids in refining maintenance strategies, assessing and enhancing asset performance, and aligning with their sustainability goals.
SAP Asset Performance Management is widely adopted within the oil and gas sector and is an integral part of the SAP Industry Cloud portfolio.
Aside from its core asset management functions, the solution offers numerous features facilitating predictive maintenance planning. S.A. Oil seeks to expand the capabilities of SAP Asset Performance Management with additional requirements, notably in developing proprietary machine learning algorithms to predict pipeline corrosion based on historical data. Moreover, the machine learning model requires adaptation over time, aligning with the latest data acquired through smart pigs and field technicians employing ultrasonic testing. Continual delivery of new AI models and predictive analyses for maintenance purposes is essential to S.A. Oil's operations.
Let’s have a close look into the functional aspects of the solution involving five key personas:
Functional Aspects & Personas
- Reliability Engineer
The reliability engineer oversees the management of pipeline master data and performs health checks within SAP Asset Management to ensure equipment functionality without failures. Additionally, they manage the connection of smart pigging sensors to the IoT gateway, gathering the captured measurements.
- Field Technician
Field technicians utilize a custom-built CAP application to record physical visits, executed maintenance tasks, and on-site observations.
- Data Scientist
Data scientists employ collected data to train AI models and develop new machine learning models.
- ML Operation Engineer
This role involves the deployment and management of newly created machine learning models.
- Maintenance Planner
The solution integrates an end-to-end monitoring system, consolidating all measures in a centralized platform. It aids in identifying and addressing anomalies, managed by the maintenance planner.
The Proposed Solution Architecture of AI-powered Pipeline Corrosion Analysis Solution:
Proposed Solution Architecture in full
Relevant data is drawn from multiple systems, encompassing asset master data, maintenance planning, and cost data sourced from SAP S4/HANA Cloud. Additionally, IoT sensor data from Smart Pigs, pipeline assessment data recorded via Ultrasonic Testing by field technicians using the Pipeline Management App, and external sources like weather data are incorporated.
All this data eventually flows into the SAP HANA Cloud through the Data Lake. The acquisition of data for machine learning is efficiently managed via a CAP multi-tenant application, namely the Pipeline Management App.
Once acquired, this data forms the foundation for training a machine learning model within SAP AI Core and AI Launchpad. The model is developed to predict the pipelines corrosion depth. These predictions are stored in SAP HANA Cloud and further analyzed and visualized through an interactive geomap in SAP Analytics Cloud.
Field technicians utilize a mobile app to explore specific pipe Key Performance Indicators (KPIs) and add new corrosion measurements using Ultrasonic testing. The ML model's performance is continuously monitored against these new field measurements.
The challenge about performance degradation of Pipeline Corrosion Prediction ML Model
Solution Architecture for MLOps
Over time, the introduction of new assessment data introduces variance in the model, leading to a decline in the performance of the pipeline corrosion ML model. The objective is to automate operational processes, ensuring continuous monitoring of the corrosion prediction model's performance. This involves auto-triggering model retraining with new datasets and deployment based on predefined performance metric rules. To accomplish this, they require the necessary infrastructure and tools to efficiently execute these automation tasks.
The AI Core is as an ideal solution for this scenario, providing a runtime environment capable of executing these automation tasks. Notably, it offers scheduling features and manual execution capabilities via AI Launchpad. Leveraging S.A. Oil's existing AI Core and AI Launchpad infrastructure appears highly sensible for running and managing its MLOps pipelines—a pivotal step towards implementing this approach in our Proof of Concept.
The proposal involves the use of three AI Core service instances. However, there is an opportunity to optimize landscape costs by deploying a single AI Core instance for three different purposes, utilizing the AI Core's resource groups.
Exploring the alternatives of Architecture
Let's now explore various alternatives in constructing this architecture and understand the rationale behind choosing specific components and technologies.
Option 1 – SAP Data Intelligence Cloud:
Data Intelligence offers an alternative for running MLOps pipelines, providing a built-in scheduler and monitoring dashboard. However, its adoption would be justified only if S.A. Oil already integrates SAP Data Intelligence Cloud in their existing system landscape. Otherwise, investing solely for MLOps for AI Core might be challenging for S.A. Oil.
Option 2 – SAP Build Process Automation:
This tool is another alternative that offers a visual interface for running MLOps pipelines, including a scheduler and process automation capabilities. Similarly, its adoption would make sense only if S.A. Oil already incorporates SAP Build Process Automation in their current system landscape. Otherwise, justifying an investment exclusively for MLOps in AI Core might pose challenges.
Option 3 – Custom CAP Application on Cloud Foundry:
As an alternative approach, S.A. Oil's MLOps team could opt to develop a complete CAP (Cloud Application Programming Model) application on SAP BTP's Cloud Foundry runtime. This application could manage their MLOps activities specifically related to the Pipeline Corrosion Predictive Model within SAP AI Core. However, it's crucial to acknowledge that this option would require additional effort to maintain the CAP application and might incur added costs associated with SAP BTP's Cloud Foundry runtime.
Option 4 – Implementing the MLOps pipelines via Project Piper and Jenkins:
Jenkins, known for its proficiency in CI/CD, also presents a viable option for orchestrating and automating end-to-end MLOps processes in SAP AI Core, leveraging both Jenkins pipelines and the SAP AI Core SDK. Jenkins can help to streamline various workflows throughout ML lifecycle stages: from model training and validation to deployment and monitoring. Its seamless integration with version control systems facilitates continuous integration and continuous deployment (CI/CD) for ML models.
While SAP Continuous Integration and Delivery exists, it is not able to address the highly customizable tasks of MLOps for SAP AI Core. For easy setup of such highly customizable continuous delivery for SAP technologies, SAP introduces tooling for continuous delivery in project "Piper" integrated with Jenkins. This integration is applicable to projects on both the SAP Business Technology Platform (BTP) and SAP on-premise platforms, so is the MLOps of SAP AI Core.
Utilizing Project “Piper” in combination with Jenkins for MLOps ensures efficient workflows, automated model development, improved collaboration among team members, and consistent model deployment, ensuring the reliability and scalability crucial for ML applications. However, this necessitates a Jenkins infrastructure, which is justifiable if you seek to centrally manage all CI/CD workloads, be it software development or MLOps, within the same Jenkins environment.
Further resources
- The source code of the sample AI-powered Pipeline Corrosion Analysis solution
- SAP Asset Performance Management
- SAP AI Core help page
- Project "Piper"
- A Data-Driven Machine Learning Approach for Corrosion Risk Assessment—A Comparative Study
- Visualizing Future Pipeline Projects Around the World
- Corrosion monitoring in piping systems
Wrap up:
In conclusion, the implementation of AI-powered Pipeline Corrosion Analysis emerges as a transformative solution for the oil and gas industry. By harnessing advanced technologies like machine learning models integrated with SAP platforms such as AI Core, businesses like S.A. Oil can proactively predict pipeline corrosion, mitigate risks, and optimize maintenance strategies. To ensure the reliability and performances for this AI-powered Pipeline Corrosion Analysis, Automated MLOps is required, we have also explored the options of MLOps infrastructure, such as the use of AI Core, AI Launchpad, and Jenkins together with Project “Pipe” etc. for orchestrating and managing MLOps pipelines efficiently. In next blog post, we will dive into the implementation detail of AI-powered Pipeline Corrosion Analysis.
- SAP Managed Tags:
2 Comments
