Drones were once an idea of science fiction and now are becoming almost ubiquitous in daily life. One of the largest segments of growth in the drone market is the commercial and industrial use of drones. Consulting firm BCG estimates the industrial drone fleet in Europe and the US will be $50 billion by 2050 and more than 1 million units, with most of the value linked to drone services and data collection.
Many companies are already using drones to monitor and inspect large structures, capturing images and data taking advantage to quickly and cheaply gather localized visual information. A client came to SFL Scientific utilizing drones to capture thermal video wanting to utilize the video to extract temperature readings.
SFL Scientific set out to build a state-of-the-art (SotA) machine learning (ML) pipeline to automate the temperature extraction process for cylinder vents in drone imagery. In this 7 week engagement, SFL Scientific developed a deep learning (DL) computer vision model for vent segmentation using the drone captured RGB videos. The trained model was evaluated to provide a vent detection performance score of 93.1% averaged over test videos.
The developed Python-based algorithm supported the functionalities of model training, model testing, and the extraction of thermal values based on the associated thermal videos. All the findings in this POC show the potential that modern ML techniques can offer for vent detection and temperature extraction using the drone imagery data.
The Data and Model
SFL Scientific received 14 pairs of RGB and thermal videos for the vent detection model development. The videos have a variety of durations and frame rates, adding up to 864 seconds for both RGB and thermal videos. Computer vision modeling requires ground truth labeling of the target objects on the images, i.e. vents on the RGB images. Since the raw video data does not contain any annotation of vents on cylinders, SFL Scientific manually annotated the vents on all images.
The pipeline is developed to first detect vent locations on RGB images and then transfer these locations to the corresponding thermal images. SFL Scientific employed the Mask- and Region-Based Convolutional Neural Network model Mask_RCNN [ for the vent segmentation. It is a SotA DL algorithm used for object detection for high inference speed and high model detection performance.
The vent detection model has an exceptional detection performance score of 93.1% especially considering the relatively limited training data size. All of the model performance analyses strongly indicate that SotA ML tools can be employed to construct reliable and robust temperature extraction algorithms on drone imagery.