Vibration analysis is often used to detect potential faults in rotating machinery and structures (e.g., bridges, turbine blades) by evaluating velocity or acceleration data measured from inherent vibrations in the machine or structure of interest.
In one nondestructive testing engagement, SFL Scientific utilized 100,000 vibration-testing samples of velocity data that were synthetically generated to build a model for predicting risk scores for shaft misalignment (SM) in machinery. Besides SM fault, varying amounts of other irrelevant faults were also present in the synthetic dataset. The inclusion of these additional irrelevant faults enabled further investigation into the model performance in the presence of more than one fault. The end goal was to have a model that could extrapolate well to real-world data despite having insufficient real-world data for model building.
Each data sample was a velocity-time series (Exhibit 1A). Feature engineering on each time series was performed to create input variables for the model. Features included variables from the time series distribution, the frequency-transformed distribution (Exhibit 1B), and information related to the rotating machinery component. Exhibit 3. (A) Example velocity data (B) Frequency spectrum of the example velocity data
After feature engineering, each model was built using a portion of the synthetic dataset allocated for training (Exhibit 3). Different model iterations were explored by training different selections of possible features. Importantly, the goal is to achieve robust performance even when the model is used on real-world data. Specifically, while the synthetic data may be based on some known aspects of the fault characteristics, this is not exhaustive and can sometimes inadvertently include nuances different from real data which are erroneously taken into account by the model during the training. After some in-depth comparisons of the different models, it was determined that generalizability to real data is improved by adding extra feature normalization steps and excluding features related to partial autocorrelation and peak measures. The latter features are identified as highly vulnerable to dynamic aspects of real-world data that often vary depending on data collection choices, such as the sampling rate.
The model outputs risk scores indicating the probability between 0 and 1 that a SM fault is present. Using a threshold (default threshold value= 0.5), binary classification labels can be derived, in which samples with predicted risk scores above the threshold are labeled as having SM fault present. These predicted binary labels can be compared against the true labels to quantify performance. Model performance was evaluated using ROC AUC, which estimates the amount of tradeoff between the true and false positive rates over all possible classification thresholds, and ranges from 0 indicating poor tradeoff to 1 indicating best possible tradeoff. The ROC AUC of the model on the synthetic data was 0.93, indicating an optimal tradeoff between true and false positives. When the subset of the testing dataset where only SM fault is present was used, the model resulted in a ROC AUC value of 0.97, indicating even better performance when no other fault types were present (Exhibit 2).
20 samples of real data were also provided to estimate model performance in the real world. Unsurprisingly, model performance using the real data was lower but still well above the baseline resulting in a ROC AUC score of 0.71 (Exhibit 2).
Finally, feature rankings can be derived from each model, allowing examination of how different features might contribute differently to prediction (Exhibit 3).
|Model||Test dataset||ROC AUC|
|Final Model||Synthetic data (N=15,000)||0.93|
|Final Model||Synthetic data subset with only SM fault (N = 1000)||0.97|
|Final Model||Synthetic data subset with at least one other fault type (N = 14,000)||0.84|
|Final Model||Real data (N = 20)||0.71|
Exhibit 2. Overall model results.
This use case example provides a real life scenario of leveraging machine learning for vibration analyses in nondestructive testing. Importantly, it not only saves time, effort, and money, it also relinquishes the dependency on explicit knowledge of how faults are identified as needed in manual analyses. This is especially important given the many ways different faults can manifest in the vibration data.