Unleash the Power of Anomaly Localization in Industrial Automation
The industrial automation landscape is evolving rapidly, driven by digital 2.0 transformation and advanced AI/ML solutions. In industrial automation, detecting anomalies and pinpointing their exact locations are vital to maintaining operational efficiency and reducing downtime. Traditional anomaly detection models can identify irregularities but often fail to localize anomalies accurately, especially in complex settings. Anomaly detection and localization play a critical role in maintaining machinery uptime and ensuring operational efficiency. To improve these processes, companies are leveraging state-of-the-art AI pipelines combining object detection and anomaly localization models.
This blog explores a solution powered by SiMa.ai Machine Learning System-on-Chip (MLSoC) that integrates CenterNet for object detection and STFPM (Student-Teacher Feature Pyramid Matching) for precise anomaly localization. This approach enhances accuracy, reduces latency, and provides actionable insights to prevent costly downtimes by leveraging SiMa.ai MLSoC edge capabilities.
Driving Precision with Advanced Anomaly Detection
Why Is Accurate Anomaly Localization Critical?
In industrial settings, identifying anomalies alone is insufficient — precise localization within the frame is crucial for rapid resolution. Conventional detection techniques can signal an issue but frequently fall short in pinpointing its exact location, resulting in prolonged troubleshooting and inefficiencies. AI/ML pipeline powered by SiMa.ai MLSoC edge device addresses this challenge through a two-stage process:
1\ CenterNet for Object Detection: Identifies machinery components or regions of interest (ROIs) in real-time.
2\ STFPM for Anomaly Localization: Compares feature maps between a “teacher” model (pre-trained) and a “student” model (current observation) to detect and localize anomalies within detected objects.
Business Impact
Localization of anomalies not only saves time in the identification of areas of rework, it can provide valuable insights with respect to targeting root-cause areas like product design, process and/or discrete workflow design, and equipment (i.e. production machinery) design, improving yields through reduced reworks and rejects.
By enhancing accuracy in defect detection, manufacturers can achieve key industrial KPIs, including a reduction in rework rates by up to 30%, a decrease in defect-related scrap by 20-40%, and an increase in overall equipment effectiveness (OEE) by 10-15%. These improvements lead to higher yields, lower operational costs, and more efficient production cycles.
Building a Pipeline for Anomaly Localization
Combining object detection with anomaly detection can enable more precise localization, helping operators quickly address issues. This blog explores an AI/ML pipeline using CenterNet for object detection, followed by STFPM for anomaly localization in industrial automation.
Architecture Overview
Industrial automation architecture shown in Fig.1 highlights three key components working together to ensure efficient operations and proactive decision-making.
1\ Object Detection serves as the foundation, leveraging advanced machine learning models to accurately identify and classify objects in real-time, enabling seamless automation and quality control. CenterNet, a real-time object detection model, detects key objects or machinery parts within the scene.
2\ Anomaly Localization builds upon this by detecting deviations or defects within the production line, pinpointing irregularities with precision to minimize disruptions and reduce downtime. The Student-Teacher Feature Pyramid Matching (STFPM) model leverages the feature representations from the CenterNet model to locate anomalies by comparing normal and anomalous feature representations.
3\ Visualization and Monitoring integrates data from these processes into an intuitive dashboard, providing operators with actionable insights, real-time alerts, and historical analytics to optimize performance and maintain operational efficiency.
Pipeline Workflow
A well-structured pipeline workflow is essential for seamless industrial automation, ensuring efficiency, accuracy, and real-time decision-making. Let’s dive deeper into the various stages involved in the industrial automation workflow.
1\ Data Ingestion: High-frequency raw sensor and visual image data are collected from industrial processes (typically this will be GigE cameras). These inputs are critical for real-time processing.
2\ Object Detection with CenterNet: CenterNet is used to identify the spatial location of key objects (e.g., machine parts) in images. Each detected object serves as a region of interest (ROI) for anomaly detection. In a typical industrial environment, this model detects the target object on the conveyor belt and creates a bounding box around the object. The original frame then gets cropped and sent to the Anomaly Detector. The Centernet model for object detection is benchmarked at ~366 FPS on SiMa MLSOC.
3\ Anomaly Localization with STFPM: STFPM is trained with a student-teacher architecture. The teacher model learns a distribution of “normal” features, and the student model tries to match these. Discrepancies between the two are treated as anomalies. We used a STFPM framework using ResNet18 models for anomaly localization. This model takes its input from the object detector (referred to in step 2 above ) and creates an anomaly heatmap. Cropped results are resized to 256x256x3 or 512x512x3, and the model outputs a corresponding heat-map for the anomalies, which can be visualized as desired The standalone model is benchmarked at ~1000 FPS on SiMa MLSOC.
4\ Real-Time Monitoring: Once anomalies are detected and localized, alerts are sent, and the data is logged in a time-series database with a visualization dashboard to monitor in real-time as shown in Fig.2 below. Data is further aggregated in the cloud for advanced analytics.
Code Snippets
Refer to the following sections with sample code snippets (based on python) that illustrate key implementation steps, ensuring a practical and hands-on approach to build and deploy anomaly localization for industrial automation. For access to the complete source code and GStreamer pipeline implementation details, visit SiMa.ai developer guide.
1. CenterNet Object Detection
The CenterNet model detects objects by predicting their centers.
The CenterNet model detects objects by predicting their centers.
def run_infer(self, img: np.ndarray, filename:str, dump=False,xDilate=0,yDilate=0): """ Executes model inference on a given image.
Parameters
----------
img : np.ndarray
The image on which to perform inference. Should be in the format expected by the model.
basename : str
Base name for saving the inference results.
Returns
-------
None
"""
import time
start = time.time()
preprocessed_image = self.preprocess(img, Constants.CNET_MEAN, Constants.CNET_STD_DV)
output = self.run_mla(preprocessed_image)
detections = self.postprocess(output)
if len(detections[0]) > 0:
roi = cnet_helper.get_crop_roi(detections, Constants.FRAME_WIDTH, Constants.FRAME_HEIGHT,xDilate=xDilate,yDilate=yDilate)
dets = [item for sublist in detections for item in sublist]
for det in dets: # Dilation
det['x1'] = max(0,min(Constants.FRAME_WIDTH-1,int((1-xDilate)*det['x1'])))
det['y1'] = max(0,min(Constants.FRAME_HEIGHT-1,int((1-yDilate)*det['y1'])))
det['x2'] = max(0,min(Constants.FRAME_WIDTH-1,int((1+xDilate)*det['x2'])))
det['y2'] = max(0,min(Constants.FRAME_HEIGHT-1,int((1+yDilate)*det['y2'])))
if(det['x1'] - det['x2'] == 0 or det['y1'] - det['y2'] == 0):
dets.remove(det)
return dets, roi
else:
return ([], [])
2. STFPM Anomaly Localization
The STFPM model leverages a teacher-student setup, where discrepancies between teacher and student embeddings are used to locate anomalies.
def run_infer(self, img: np.ndarray, basename="", dump=False, returnOverlay=True):
"""
Executes model inference on a given image.
Parameters
----------
img : np.ndarray
The image on which to perform inference. Should be in the format expected by the model.
basename : str
Base name for saving the inference results.
Returns
-------
None
"""
preprocessed_image = self.preprocess(img, Constants.STFPM_MEAN, Constants.STFPM_STD_DV)
nn_frame = self.run_mla(preprocessed_image)
frame_out = self.postprocess(in_frame=img, out_frame=nn_frame,returnOverlay=returnOverlay)
return frame_out
3. Visualizing Anomalies in Real-Time
We use a simple visualization tool to overlay anomaly scores on detected ROI’s, highlighting probable anomalies for operators.
# Prepare the image file for HTTP POST
files = {'original_image': og_jpeg.tobytes(),'adnet':run_adnet,'servervis':serverVis}
if(run_adnet): # Only pack heatmaps if AD was Run.
if(serverVis):
# Transform the Heatmap into Bytes
heatmaps = np.stack(heatmaps,axis=-1)
files.update({'centernet_shape':io.StringIO(json.dumps(img.shape)),'heatmaps_shape':io.StringIO(json.dumps(heatmaps.shape)),'heatmaps':heatmaps.tobytes(),'xywhs':io.StringIO(json.dumps(xywhs)),'crop_shapes':io.StringIO(json.dumps(crop_shapes))})
# Already Computed Heatmap on Board
else:
_, heat_jpeg = cv2.imencode('.jpg',heatmap_img)
files.update({'heatmap_image':heat_jpeg.tobytes()})
# Send the image to the server via HTTP POST
# Replace the url with whatever you receive from the server code
try:
response = requests.post(http_url, files=files)
except requests.exceptions.RequestException as e:
print(f"Debug: Error sending image to server: {e}")
Live Demo
We have actively participated in leading industry events and conferences, showcasing live demonstrations of anomaly localization powered by SiMa.ai’s Machine Learning System-on-Chip (MLSoC). These demos highlight the transformative impact of advanced AI-ML edge systems in industrial automation, and provided a firsthand look at how the end-to-end solution accurately identified and localized anomalies within the production workflow. Watch the video shown in Fig.3 to see anomaly localization in action and experience the future of intelligent automation.
Conclusion
Combining state-of-the-art AI/ML pipeline with CenterNet and STFPM models for anomaly localization offers a powerful, scalable solution for industrial automations. This hybrid approach enables operators to detect and localize anomalies with high accuracy, reducing downtime and maintenance costs. By combining advanced detection techniques, this system offers a robust and efficient solution for real-time anomaly detection and localization, ensuring optimal performance and reliability. As a next step, further improvements can include integrating reinforcement learning to adaptively adjust thresholds based on anomaly frequency and severity, providing even more precise localization.
Ready To Get Started?
Discover how SiMa.ai can tailor an AI solution to meet your unique needs and drive transformative outcomes. Join our Edgematic Early Access Program to experience our No Code/Low Code, Drag & Drop Programming Environment, empowering you to create, build, and deploy edge AI solutions seamlessly on SiMa.ai MLSoC silicon. Contact our solutions experts today.