Low-Code Machine Learning with Ludwig and SAP AI Core #2-Defect Detection & Sound Anomaly Detection

[YatseaLi]

In my previous blog post, we have discussed the business values, overview and technical details about end-to-end Low-Code Machine Learning with Ludwig and SAP AI Core. In this blog post, I'll continue sharing my experience of replicating another two custom AI use cases from previous work steam with ludwig in low-code approach, comparing with their original implementations in pro-code approach.

Use Case#2: Defect Detection with Image Segmentation

In this use case, we use computer vision(image segmentation) for defect detection in production line of Light Guide Plate. Please refer to our blog post about Build Your Own Model with TensorFlow in SAP AI Core for defect detection and source code for more details. The original implementation is a U-Net model architecture with Tensorflow by my colleague cesarecalabria.

The original training code is available here with 483 lines of python code. Here is just a screenshot of some code snippet. In order to implement this training code, it not only require you to have a good understanding the unet model architecture, but also the skill of ML development with python and tensorflow.

Okay, no worry. Let's implement the same model based on the same dataset with Ludwig in a much easier way.

Dataset and Data Pre-Process

The original dataset is structured as below. The original images of products are stored in Images folder, their corresponding segmented image are store in Masks folder. Sub folders NG(Defected Product) and OK(Passed) are labels of the product image, which is not in used here.

Every product image in Images folder has its own masked pair image in Masks folder. For example

data/Images/NG/00000_NG_Image.bmp	data/Masks/NG/00000_NG_Mask.bmp
Prdocut Image	Mask of Defect

The original images in the dataset has a bmp format, which isn't supported in ludwig image feature, hence, we have to transform the original images from bmp format into jpg. Here is the sample code below.

import os
from PIL import Image

def convert_bmp_to_jpg(directory):
    # Walk through all files in the directory and its subdirectories
    for root, dirs, files in os.walk(directory):
        for file in files:
            if file.endswith(".bmp"):
                # Create the full file path
                bmp_path = os.path.join(root, file)

                # Open the BMP image
                with Image.open(bmp_path) as img:
                    # Convert the image to RGB mode
                    img = img.convert("RGB")

                    # Create the new file name with .jpg extension
                    jpg_path = os.path.splitext(bmp_path)[0] + ".jpg"

                    # Save the image as a JPG
                    img.save(jpg_path, "JPEG")

                # Optionally, delete the original BMP file after conversion
                os.remove(bmp_path)

                print(f"Converted {bmp_path} to {jpg_path}")

# Replace './data' with the actual path to your directory
convert_bmp_to_jpg("./data")

 In addition, we'll also need to create a csv file as dataset.csv based on the image dataset like

image_path,mask_path,label
data/Images/NG/00000_NG_Image.jpg,data/Masks/NG/00000_NG_Mask.jpg,NG
data/Images/NG/00001_NG_Image.jpg,data/Masks/NG/00001_NG_Mask.jpg,NG

Here is the python script to generate dataset.csv based on the folder structure of dataset

import os
import csv

# Define paths
image_dir = "data/Images"
mask_dir = "data/Masks"
output_csv = "dataset.csv"

# Collect all image and mask file paths
image_files = {"NG": [], "OK": []}
mask_files = {"NG": [], "OK": []}

for label in ["NG", "OK"]:
    image_files[label] = sorted([os.path.join(image_dir, label, f) for f in os.listdir(os.path.join(image_dir, label))])
    mask_files[label] = sorted([os.path.join(mask_dir, label, f) for f in os.listdir(os.path.join(mask_dir, label))])

# Create and write to the CSV file
with open(output_csv, mode="w", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["image", "mask", "label"])

    for label in ["NG", "OK"]:
        for img, mask in zip(image_files[label], mask_files[label]):
            writer.writerow([img, mask, label])

print(f"Dataset CSV file has been created: {output_csv}")

A declarative config.yaml

Based on the dataset above, all the code we need to train a defect detection model of LPG product with image segmentation is a declarative config.yaml as below. Please refer to this for more details about image feature in ludwig. Literally, we have simplified the training from 480 lines of python code into 40 lines of declarative yaml code. It is way too easy and straight-forward, even application developer and business users with limited machine learning knowledge can write such a declarative config.yam. Having said that, you still need the skill of data pre-processing, and the knowledge of selecting the input and output features based on business objectives and dataset.

input_features:
  - name: image_path
    type: image
    preprocessing:
      num_processes: 6
      height: 224
      width: 224
      infer_image_max_height: 1024
      infer_image_max_width: 1024
    encoder: unet

output_features:
  - name: mask_path
    type: image
    preprocessing:
      num_processes: 6
      height: 224
      width: 224
      infer_image_max_height: 1024
      infer_image_max_width: 1024
      infer_image_num_classes: true
      num_channels: 1
      num_classes: 2
    decoder:
      type: unet
      num_fc_layers: 0
    loss:
      type: softmax_cross_entropy
    metric:
      type: iou

combiner:
  type: concat
  num_fc_layers: 0

trainer:
  epochs: 100
  early_stop: -1
  batch_size: 1
  max_batch_size: 1

For the model training and serving, you can just use the ludwig command lines, like ludwig experiment/serve etc. which we have gone through in last blog post in detail. So are running the same ludwig workloads within SAP AI Core.

Pro-Code VS Low-Code

Next, let's have a look another use case about audio features.

Use Case#3: Predictive Maintenance based on Machinery Sound Anomaly Detection

In the last use case, we'll detect the sound anomaly of machinery for predictive maintenance of production line. Please refer to our blog post about Sound-based predictive maintenance with SAP AI Core and SAP AI Launchpad and source code for more details. The original implementation is a custom CNN model with Tensorflow by my colleague amagnani.

The original training code is available here with 345 lines of python code. Here is just a screenshot of some code snippet. In order to implement this training code, it not only require you to have a good understanding of CNN, but also the skill of ML development with python and tensorflow.

Okay, no worry. Let's implement the same model based on the same dataset with Ludwig in a much easier way.

Dataset and Data Pre-Process

The original dataset is structured as below. The sub folders anomaly1, anomaly2 and ok are the labels of the audio samples.

We'll also need to create a csv file as dataset.csv based on the image dataset like 

audio_path,class
data/ok/clip0568.wav,ok
data/anomaly1/clip0583.wav,anomaly1
data/anomaly2/clip0597.wav,anomaly2

Here is the python script to generate dataset.csv based on the folder structure of dataset 

import os
import csv

# Define the root directory and the output CSV file
root_dir = "data"
output_csv = "dataset.csv"

# Prepare the data list
data = []

# Traverse the directory structure
for subdir, dirs, files in os.walk(root_dir):
    for file in files:
        if file.endswith(".wav"):  # Filter for .wav files
            # Construct the file path and class label
            file_path = os.path.join(subdir, file)
            class_label = os.path.basename(subdir)
            data.append([file_path, class_label])

# Write to the CSV file
with open(output_csv, "w", newline="") as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(["audio_path", "class"])  # Write header
    writer.writerows(data)

print(f"CSV file '{output_csv}' generated successfully.")

A declarative config.yaml

Based on the dataset above, all the code we need to train a sound anomaly detection with CNN is a declarative config.yaml as below. Please refer to this for more details about image feature in ludwig. Literally, we have simplified the training from 345 lines of python code into 19 lines of declarative yaml code. It is much easier and more straight-forward, even application developers and business users with limited machine learning knowledge can write such a declarative config.yam. Having said that, you still need the skill of data pre-processing, and the knowledge of selecting the input and output features based on business objectives and dataset.

input_features:
  - name: audio_path
    type: audio
    preprocessing:
      audio_file_length_limit_in_s: 7.5
      audio_feature:
        type: stft
        window_length_in_s: 0.04
        window_shift_in_s: 0.02
    encoder:
      type: parallel_cnn

output_features:
  - name: class
    type: category

trainer:
  epochs: 100
  early_stop: -1

Pro-Code VS Low-Code

Conclusion

In these two use cases, we have witnessed the ease and efficiency of the low-code machine learning with Ludwig and SAP AI Core over the pro-code approach. Ludwig simplifies the machine learning process with a low-code approach. Not only opening the door of machine learning for citizen ML developers like application developer, even business user etc. but also increasing the productivity of the professional machine learning engineers and data scientists to quickly experiment, build, train, and deploy models. When paired with SAP AI Core, these machine learning operations can be streamlined and managed at scale, fostering faster AI-driven innovation across the enterprise with enterprise-grade of security, scalability and compliance.