Anomaly detection in time series data, which is pivotal for practical applications like monitoring industrial systems and detecting fraudulent activities, has been facing challenges in terms of its metrics. Existing measures such as Precision and Recall, designed for independent and identically distributed (iid) data, fail to entirely capture anomalies, potentially leading to flawed evaluations in key areas like financial fraud detection and medical diagnostics.
To mitigate this, a new metric—Proximity-Aware Time series anomaly Evaluation (PATE)—has been introduced. Unlike prior metrics, PATE incorporates proximity-based weighting and temporal correlations for a more precise and comprehensive evaluation.
Time series data, due to its sequential nature, have been previously assessed using metrics such as R-based, TS-Aware, and the PA-F1 among others. However, these measures often necessitate subjective threshold settings and do not consider onset response timings, early and late detections. Though alternatives like AUC-ROC and VUS offer threshold-free evaluations, they neglect the temporal dynamics and correlations within the time series data.
PATE attempts to fill these gaps by offering a weighted version of the Precision and Recall curve. This novel approach incorporates coverage level, early and late detections, and onset response timing to categorize prediction events into true detections, delayed detections, early detections, and false positives or negatives. The respective categories are then assigned weights as per their significance for an early warning or timely coverage of anomalies.
PATE, by integrating buffer zones and temporal proximities, enables a thorough and accurate evaluation of anomaly detection models. As it provides proximity-based weights to false and true positives and negatives, it ensures precise and insightful assessment of model performance. PATE’s adaptability to different buffer sizes without compromising on fairness or consistency further indicates its potential applicability in real-world cases.
A comparison of PATE with other anomaly detection methods revealed notable discrepancies in performance assessments. While point-adjusted metrics tend to overestimate model performance, alternatives like ROC-AUC and VUS-ROC may disregard subtle detection errors, lacking discernability between models. This new metric, therefore, leads to a shift in the understanding of the performance of existing models.
In summary, the unique contribution of PATE lies in giving a more accurate and nuanced assessment of anomaly detection models, by employing temporal proximity and buffer zones. This could have significant implications for future research, industry adoption, and the development of practical applications in critical sectors like healthcare and finance.
