Drones were once an idea of science fiction and now are becoming almost ubiquitous in daily life. One of the largest segments of growth in the drone market is the commercial and industrial use of drones. Consulting firm BCG estimates the industrial drone fleet in Europe and the US will be $50 billion by 2050 and more than 1 million units, with most of the value linked to drone services and data collection.
Many companies are already using drones to monitor and inspect large structures, capturing images and data taking advantage to quickly and cheaply gather localized visual information. A client came to SFL Scientific utilizing drones to capture thermal video with the goal of using the video to extract temperature readings.
With this request, SFL Scientific set out to build a state-of-the-art (SotA) machine learning (ML) pipeline to automate the temperature extraction process for cylinder vents in drone imagery. In this 7 week engagement, SFL Scientific developed a deep learning (DL) computer vision model for vent segmentation using the drone captured RGB videos. The trained model was evaluated to provide a vent detection performance score of 93.1% averaged over test videos.
The developed Python-based algorithm supported the functionalities of model training, model testing, and the extraction of thermal values based on the associated thermal videos. All the findings in the proof of concept (POC) showed the potential that modern machine learning (ML) techniques can offer for vent detection and temperature extraction using drone imagery data.
The Data and Model
SFL Scientific received 14 pairs of RGB and thermal videos for the vent detection model development. The videos had a variety of durations and frame rates, adding up to 864 seconds for both RGB and thermal videos. Computer vision modeling requires ground truth labeling of the target objects on the images, i.e. vents on the RGB images. Since the raw video data does not contain any annotation of vents on cylinders, SFL Scientific manually annotated the vents on all images.
The pipeline is developed to first detect vent locations on RGB images and then transfer these locations to the corresponding thermal images. SFL Scientific employed the Mask- and Region-Based Convolutional Neural Network model (Mask_RCNN) for the vent segmentation. A state of the art (SOTA) DL algorithm used for object detection for high inference speed and high model detection performance.
The vent detection model has an exceptional detection performance score of 93.1% even using a relatively limited training data size. All of the model performance analyses strongly indicate that SotA ML tools can be employed to construct reliable and robust temperature extraction algorithms on drone imagery.