Cross-Domain Generalization of Anomaly Detection on Network-Intrusion Data

A PyTorch reimplementation of my drift-aware, uncertainty-weighted online ensemble (ESWA 2020), evaluated under real distribution shift — trained on one attack distribution, tested on another.

Project Background

An intrusion detector that scores beautifully on its own test set can collapse the moment the traffic shifts — new attack types, a different network, a different population. The number that matters is therefore not in-domain accuracy but the generalization gap: how much performance drops under distribution shift. This is the same failure mode a clinical-AI model faces moving between patient populations — here posed on network-security data, where it can be measured on a real, canonical benchmark.

This project reimplements the drift-aware method from my Expert Systems with Applications (2020) paper in PyTorch and evaluates it the honest way: external validation across attack distributions.

PyTorch Online / Streaming ML Domain Shift / Generalization Intrusion Detection

Code: github.com/Emad-Mahmodi/adaptive-ids-generalization

Method — Uncertainty-Weighted Online Ensemble

Several online linear learners (Perceptron, Passive-Aggressive I/II, a Confidence-Weighted learner) process the traffic stream one sample at a time. Their predictions are fused with weights inversely proportional to each learner’s prediction-error variance — a learner making stable, low-variance errors is trusted; an erratic one is automatically down-weighted. This is what makes the method drift-aware without any explicit drift detector, and it is the PyTorch port of the paper’s original fusion contribution.

Results — Generalization Under Distribution Shift

NSL-KDD ships two official splits, KDDTrain+ and KDDTest+, that form a built-in distribution shift: the test set deliberately contains attack types under-represented or absent in training. Training on one and testing on the other is genuine external validation, not a random split.

Figure 1: Every metric drops from in-domain (held-out KDDTrain+) to cross-domain (shifted KDDTest+). F1 falls from 0.97 to 0.76 — a 21-point generalization gap. An in-domain number alone would badly overstate real-world performance.

The headline finding: a detector that reads as near-perfect on its own distribution (F1 0.97) drops to F1 0.76 under a realistic attack-distribution shift. Reporting only the in-domain score would be misleading — which is exactly why external validation across distributions is the honest way to evaluate a detector. The uncertainty-weighted ensemble also edges out every individual base learner on the shifted set.

Why This Framing Matters

The methodology — measuring and reporting the in-domain → cross-domain gap — is domain-transferable. The same question (“does a model trained on one population still work on another?”) underlies digital pathology and medical imaging, where models must hold up across hospitals and patient groups. Demonstrating it end-to-end on security data I know well is a concrete, reproducible bridge into that space.

Framework, datasets, and full results: github.com/Emad-Mahmodi/adaptive-ids-generalization

Based On

A drift-aware adaptive method based on minimum uncertainty for anomaly detection in social networking
E. Mahmodi, H. Sadoghi Yazdi, A. Ghaemi Bafghi — Expert Systems with Applications, 2020
doi:10.1016/j.eswa.2020.113881 · original MATLAB code: Emad-Mahmodi/AdaptiveLarning