Natural Language Processing for Petabyte Scale Social Monitoring

SFL Scientific designed and implemented a state-of-the-art (SOTA) petabyte-scale monitoring platform for intelligence gathering on 1B+ documents, passages, and communication records with mixed-language capabilities for Asia-Pacific languages. Our system provides an advanced search and answers query capability, utilizing key extraction pipelines, anomaly detection, content summarization, image and text search ranking, and geotagging capabilities for event tracing and filtering. Our work is part of a larger platform for the U.S. Army as an active RDT&E effort focused on cutting-edge NLP solutions leveraging transformers (e.g., BERT) and custom language models for threat detection and intelligence.

This system provides the DoD with a new capability to gather intelligence, respond to free-form analyst queries for better situational awareness, through the examination of potentially billions of records in real-time across multi-modal interactions, transactions, and social/media communications for mission-specific activity. Further, our solution capability allows for new paradigms in data processing by supporting graph neural networks and clustering analysis for relevant operations and topic or target analysis. 

Detection Ability:  The system processes these large corpora of text and works as a key extraction pipeline, anomaly detection system, and search tool to identify important context and multi-channel features in social and digital communications, including mixed-language communications. Extending the application to topic modeling, summarization, geo-inferencing is supported. Our solution has been deployed on AWS, contains encryption at rest safeguards, and continuously examines model performance and relevant retraining as functions from understanding ingested information.

Work with Us

Start a Project

We'd like to help creating your next solution, whether modernizing legacy platforms or developing new AI solutions. Technology moves fast, let's build sustainable solutions.
Get Started