Articles 5 MIN

OptoAI UxV Payload: An AI-Based Data Fusion Platform

Ben Letson
By Benjamin Letson, PhD,

 

Challenge

Rapid identification of chemical and visual threats is crucial in a variety of CONOPS and mission environments. Next-generation UxV devices must be integrated with the capability to autonomously identify, locate, and help prioritize decisions in the detection of threats, anomalous behaviors, and other key indicators to support the safety and effectiveness of mission units.

To meet these challenges, SFL Scientific continues to strategically advance state-of-the-art defense capabilities by integrating hardware, sensor, and AI-based algorithms to develop a unified platform capable of solving the capture and real-time analysis of multimodal data at the tactical edge. UxV devices equipped with visible, infrared,  and chemical sensor suites coupled with custom AI solutions enable exponentially faster detection and threat response, providing operators real-time decision support and environmental analysis tool.

OptoAI Detect

OptoAI Detect is a rotary-wing UAV equipped with fused visual, thermal, and chemical sensors. Onboard at the tactical edge, it provides a hardware and software platform for custom classification, segmentation, and tracking of visual targets coupled with commercial sensors to allow for chemical, RF, or other spectrum analysis. At its core is our OptoAI UxV module, a low-SWaP payload with embedded deep learning capabilities on NVIDIA hardware,  built as a rapidly customizable, vehicle-agnostic solution for onboard real-time AI/ML analysis.

OptoAI Detect is orders of magnitude more sensitive to chemical threats than current alternatives. For visual and chemical threat detection, the foundation of this capability is the ability to combine a molecular chemosensor AI, leveraging a graph neural network (GNN) of 156,000 threat compounds, and custom and mission-specific computer vision algorithms. For visual detection or inspection, the solution allows for automated text and HAZMAT label interpretation, which further minimizes time to action, and has been purposed for physical target and structural identification as well.

In addition to chemical sensing, this target ID is powered by visual and thermal sensors coupled with custom models utilizing state-of-the-art deep learning methods. This multimodal data integration makes the chemical, visual, and thermal analyses greater than the sum of their parts. Armed with advanced operational intelligence, the OptoAI Detect platform deployed for semi-autonomous mission execution enables a single minimally-trained operator to serve as both a pilot and threat inspector augmented with cutting-edge decision systems.

Differentiating Technology

The OptoAI platform is based on an open, modular, and configurable design. Mission-specific modules can be created, leveraged, and tuned for specific environments, conditions, and objectives.

Multimodal UxV Deep Learning (DL)

Visual, thermal, chemical sensor, environmental, and other data are fused in real-time using spatial co-registration. Each potential threat in the environment is detected by the visual or chemical threat identification model; if a visual threat is detected, additional chemical data will be collected and vice versa. Once a complete visual and chemical inspection has been performed, all outputs are collected into a threat bundle sent to the deep learning threat assessment model. SOTA algorithms are fine-tuned to identify hundreds of potential threats.

Data flow diagram for multimodal deep learning threat assessment model.

Unparalleled Chemoinformatic AI

SFL Scientific’s unique solution to overcome chemosensory limitations was inspired by recent advances in machine representations of chemical compounds. The OptoAI model was trained on a database of 156,000 hazardous organic chemicals with their corresponding Globally Harmonized System of Classification and Labeling of Chemicals (GHS) hazard designations. By predicting where each encountered compound or mixture falls, our model is capable of handling any known or unknown chemical threat that the sensors detect. Further, OptoAI can be equipped with a block of chemical sensors, plus humidity, temperature, and pressure, for detection at the ppb/ppm levels. These high sensitivity measurements, along with time and UxV location data, can establish chemical gradients and optimize detection of tactically relevant chemical categories. This allows the system to consider environmental factors as a whole, instead of both false-negative and lower resolution, independent units.

Text/Label/Diamond Interpretation

Correctly associating NFPA hazard diamonds and other HAZMAT labels with appropriate safety protocols is often a time-consuming step in rapid response. The embedded visual detection model includes automatic identification and interpretation of all pictograms included in the Hazard Communications Standard and any relevant text. OptoAI Detect automatically looks up the appropriate response to hazardous chemicals, alerts the pilot which precautions to take, and delivers relevant chemical information to the user.

Semi-Autonomous Mission Execution

As an advanced military-ready rotary-wing UAV equipped with fused visual, thermal, and chemical sensors, the combined hardware, software, and data processing culminates in semi-autonomous mission execution. In this mode, the system autonomously seeks out threats, and the pilot makes AI-assisted executive decisions as they are identified (e.g., investigate, avoid, search for more). Visual and chemical data are fused to create a heads-up display designed to give a human operator comprehensive situational awareness. Once a potential threat is detected, onboard compute resources are used to generate OptoAI Smartpath options. These flight paths are computed to ensure the safety of the drone, operator, and surrounding environment while optimizing the efficiency of threat localization.

To access the full white paper on OptoAI UxV Payload: An AI-Based Data Fusion Platform

Work with Us

Start a Project

We'd like to help creating your next solution, whether modernizing legacy platforms or developing new AI solutions. Technology moves fast, let's build sustainable solutions.
Get Started