End-to-End Visual Inspection Process in SAP Digital Manufacturing

[jianghan-chang]

Introduction

Visual inspection enables early identification of nonconformances in manufacturing processes and ensures high-quality products. The available apps and plugins in SAP Digital Manufacturing support both manual and machine learning-based visual inspections.

The visual inspection process enables the transition from a labor-intensive, manual quality inspection approach to a more automated, streamlined quality control process. Without a machine learning model at hand, you can begin by performing manual inspections. Then, you can use the results of manual inspections to train a machine learning model and use it to perform AI-assisted inspections, which in turn can be used to further improve your model.

In this blog, I’ll guide you step by step through this process. As an example, we’ll use a toy bird as our subject of inspection which can be missing one or two wings:

Prerequisites

To perform visual inspections, you need to create noncoformance (NC) groups and nonconformance (NC) codes in SAP Digital Manufacturing. You then assign the NC codes to NC groups. For this scenario, I've created the NC group TOY_WINGS and assigned it the NC codes RIGHT_WING, LEFT_WING, and BOTH_WINGS. Of course, you also need an appropriate POD, material, work center, operation activity, routing, and so on.

Step 1 – Create Manual Visual Inspection View

Visual inspection views are the basic building blocks of visual inspections. You use the Manage Visual Inspection Views app to create visual inspection views, define their types and modes, assign NC codes to them, and, if applicable, assign machine learning models to them.

We'll create the visual inspection view TOY_VIEW_MANUAL:

1. Open the Manage Visual Inspection Views app.
2. Choose Create. The Inspection View screen appears.
3. Fill in the fields under General Information.
   1. As we haven't got a machine learning model yet, we’ll choose the Manual mode so that we can manually inspect SFCs, determine whether SFCs are conformant or nonconformant, and log appropriate NC codes.
   2. Because our inspection result can be either conformant or have one type of nonconformance, we’ll choose the Multiclass Classification type.
4. Assign our NC Codes.
   1. Under Codes And Classes, choose Add Nonconformance Codes. The Add Nonconformance Codes dialog appears.
   2. Use the value help to find and choose our NC group TOY_WINGS. The system displays the NC codes assigned to the group.
   3. Select the NC codes we need (RIGHT_WING, LEFT_WING, and BOTH_WINGS) and choose Add. (Note: In manual mode, you don’t need to define classes. We’ll discuss how to use classes when we start using a machine learning model.) 
5. Choose Create.

Step 2 – Create Visual Inspection

Next, well create the visual inspection by assigning the visual inspection view to it. We then assign the inspection to a business object in this case our toy (aka material).

We'll create the visual inspection TOY_INSPECTION:

1. Open the Manage Visual Inspections app.
2. Choose Create. The Visual Inspection screen appears.
3. Fill in the fields under General Information.
4. Assign the visual inspection view.
   1. Under Visual Inspection Views, choose Add. The Add Inspection View dialog appears.
   2. Use the search bar to find our visual inspection view.
   3. Select it and choose Add.
5. Assign the visual inspection to business objects. Here, we want to make this visual inspection available for the toy at any work center, resource, or operation activity in any routing, so we’ll assign it to only to the material TOY.
6. Choose Create.

Step 3 – Configure the Visual Inspection Bridge Plugin in your POD

Once you’ve set up your visual inspection, you can use it to inspect SFCs in the Perform Visual Inspections app. You can only open this app from a POD using the Visual Inspection Bridge plugin. This plugin transfers the data of the SFC you've selected to the Perform Visual Inspections app.

1. Open the POD Designer app.
2. Create a new Work Center or Operation Activity POD or open an existing one.
3. Add an action button to the toolbar.
4. In the Configuration panel of the action button, choose Assign Actions.
5. Assign the Visual Inspection Bridge plugin.
6. In the Configure Action Button dialog, choose Configuration and define whether you want the Perform Visual Inspections app to open in the current tab used by the POD, a new tab, or a new browser window.
7. Choose Save.

Step 4 – Perform Visual Inspection

Once you’ve added the Visual Inspection Bridge plugin in your POD, you can use it to open the Perform Visual Inspections app and inspect an SFC. You can select a visual inspection view, capture an image, review the image, and record inspection results. Your way of recording the results varies based on the selected visual inspection view and its corresponding mode and type.

Watch this video for a demonstration: Perform Visual Inspection in Manual Mode

1. Open you POD.
2. Select an SFC and choose the visual inspection action button you created.
3. The Perform Visual Inspections app opens.
4. Choose Visual Inspection and select our visual inspection.
5. Choose Visual Inspection Views and select our visual inspection view.
6. Choose Capture and Inspect to capture an image.
7. Review the image to identify defects.
   1. If there are no defects, choose Conformant.
   2. If you see a defect, identify the appropriate NC code from the Inspection Results table and log it by choosing Log. Then, choose Nonconformant.

Step 5 – Review and Export Visual Inspection Results

Upon finishing the visual inspection, the next step unfolds in the Manage Visual Inspection Results app. In this app, you can evaluate inspection results and choose to either approve or reject them.

1. Open the Manage Visual Inspection Results app.
2. Select the visual inspection result that you want to review. The system opens the detail view for the result.
3. You can see the logged nonconformances under Results.
4. To view the image saved for the result in detail, choose either the thumbnail image or Review. The Perform Visual Inspections app opens in a new tab where you can view the saved image.
5. Verify or reject the result by choosing either Approve or Reject.

You can export images of verified inspection and use them to train your machine learning model.

1. Open the Manage Visual Inspection Results app.
2. Use the filters to narrow down the relevant visual inspection results.
3. Select the results you want to export.
4. Choose Export -> Export Selected. Alternatively, choose Export All to export all filtered results.
5. Your browser starts downloading a .zip file which contains the images and one .json file that holds the metadata.

Step 6 – Create Machine Learning Model

We can use the exported images to train a machine learning model. There are many tools to do this. In this example, we’ll use the Teachable Machine by Google.

1. Go to Teachable Machine to start creating a standard image model.
2. Create the classes based on the types of our images. We have images that show the toy bird with both wings, missing right wing, missing left wing, and missing both wings. Therefore, we’ll create four classes: NO_DEFECT, RIGHT_MISSING, LEFT_MISSING, and BOTH_MISSING.
3. Choose Upload to upload the corresponding images to each class.
4. Choose Train Model and wait for training to complete.
5. Once the training is completed, you can test the accuracy of your model with a sample image.
6. Choose Export Model.
7. In the Tensorflow.js tab, select the Download option and choose Download My Model.
8. Your browser starts downloading a .zip file which contains a model file, a weights file, and a metadata file.

We can deploy our new machine learning model for visual inspection using the Manage Machine Learning Models app. 

We'll create the machine learning model TOY_MODEL:

1. Open the Manage Machine Learning Models app.
2. Choose Create.
3. Fill in the fields under General Information. Because we want to use our model to detect one type of nonconformance, we’ll choose the Multiclass Classification type.
4. Under Model Artefacts, choose Upload and select model.json and weights.bin we downloaded from Teachable Machine.
5. Add classes.
   1. Under Model Outputs, choose Add to add a class.
   2. Because our model has four outputs, we’ll create four classes named after the outputs: NO_DEFECT, RIGHT_MISSING, LEFT_MISSING, and BOTH_MISSING. The order of the model’s outputs must align with the order of the classes. You can use the up and down arrows to arrange their order.
   3. For the class NO_DEFECT, we choose Conformance as its type. For more information on threshold limits, refer to Model Outputs.
6. Choose Save.

Step 7 – Create Assisted Visual Inspection View

We'll now create a visual inspection view TOY_VIEW_ASSISTED using our machine learning model:

1. Open the Manage Visual Inspection Views app.
2. Choose Create. The Inspection View screen appears.
3. Fill in the fields under General Information.
   1. As we're going to use a machine learning model, we’ll choose the Assisted mode so that the machine learning model identifies potential nonconformances which the production operator can choose to either approve or reject manually.
   2. Because our model is of Multiclass Classification type, we’ll choose Multiclass Classification type.
   3. In the Machine Learning Model field, use the value help to search and add our machine learning model.
4. Assign our NC Codes.
   1. Under Codes And Classes, choose Add Nonconformance Codes. The Add Nonconformance Codes dialog appears.
   2. Use the value help to find and choose our NC group TOY_WINGS. The system displays the NC codes assigned to the group.
   3. Select the NC codes we need (RIGHT_WING, LEFT_WING, and BOTH_WINGS) and choose Add.
   4. Map our NC codes to their corresponding classes. The order of our NC codes must align with the order of the classes. You can use the up and down arrows to arrange their order.
5. Choose Create.

Step 8 – Add the Assisted Visual Inspection View to Visual Inspection

1. Open the Manage Visual Inspections app.
2. Search and open our visual inspection TOY_INSPECTION.
3. Choose Edit.
4. Add our assisted visual inspection view TOY_VIEW_ASSISTED.
5. Choose Save.

Step 9 - Perform Visual Inspection

Watch this video for a demonstration: Perform Visual Inspection in Assisted Mode

1. Open your POD.
2. Select an SFC and choose the visual inspection action button you created.
3. The Perform Visual Inspections app opens.
4. Under Visual Inspection and select the appropriate visual inspection.
5. Under Visual Inspection Views and select the assisted visual inspection view.
6. Choose Capture and Inspect to capture the image.
7. The machine learning model calculates the probability of each NC code. The results are shown in the Inspection Results table.
8. Review the image. You can either agree with the model’s prediction and log the NC code with the highest probability or disagree and log an NC code you think is appropriate.
9. Choose either Conformant or Nonconformant to log a result for the visual inspection. 

In the Manage Visual Inspection Results app, you can view the hit rate for assisted or semiautomated visual inspections. The hit rate indicates the ratio of correct predictions versus the total number of predictions. This helps you evaluate your machine learning model and further improve it.

Conclusion

Visual inspection in SAP Digital Manufacturing can simplify the identification of nonconformances by leveraging machine learning models. You now know each step required to transition from manual to assisted visual inspections, from setting up manual inspections, to performing inspections, to exporting inspection results, and finally, to training and implementing a machine learning model for automated inspections.

For more information, refer to the user assistance for SAP Digital Manufacturing on the SAP Help Portal.

Do you like this post? Feel free to tell us what you think in the comments. We appreciate your feedback!

Or, if you have any questions, please check SAP Community Q&A Area, or comment down below.