Drift-Aware Fraud Detection — ML Lifecycle & Model Governance
Built an end-to-end ML system that detects data drift, evaluates model degradation, and governs retraining decisions.
Key Insight
“Link distributional drift to model degradation and automatically recover performance via retraining.”
System Capabilities
- Real-time drift detection (KL divergence + PSI) with timeline tracking
- Automated retraining triggered by drift with cooldown constraints
- Live observability dashboard with shadow vs. production comparison
Built an end-to-end ML system that detects data drift, evaluates model degradation, and governs retraining decisions.
The system continuously monitors feature distribution shifts and model performance, triggering retraining when degradation is detected. Candidate models are compared against production before deployment.
Includes: • Drift detection (statistical distribution monitoring) • Performance tracking (AUC-based evaluation) • Complete ML system loop: Frozen Dataset → Failure-mode metrics (AUC degradation, PSI drift) → Δ Analysis → Policy Gate (Business Loss < 0) → CI Promotion • Explicit decision loop (retrain / no_action) • CLI observability for drift history and decisions
Result: Prevents revenue loss from misclassification trade-offs. Candidate models are rejected if accuracy gains (+0.002 AUC) result in higher business loss (+7% higher false positive cost), ensuring threshold optimization always serves business objectives.
Key Highlights
- Complete ML system loop: Frozen Dataset → Failure-mode metrics (AUC degradation, PSI drift) → Δ Analysis → Policy Gate (Business Loss < 0) → CI Promotion
- Maintains 0.975 AUC-ROC baseline with business-loss-aware promotion gates using 10:1 FN/FP cost weighting
- Promotion gates block candidates with unstable thresholds or higher estimated loss — even if accuracy improves (+0.002 AUC)
- Real-time drift detection via KL divergence and PSI with persistent drift score timeline tracking distributional evolution over time
- Built on 284K transactions dataset with feature shift explanation surfacing top shifted feature driving drift
- Retraining cooldown constraint prevents unstable loops under noisy drift signals — demonstrates real-world system constraints
- Shadow deployment architecture — every prediction scored by both production and candidate models simultaneously
- Versioned model registry with full provenance: trigger reason, drift score, top shifted feature, training timestamp, deployment status
- Live Next.js observability dashboard: drift timeline, confidence distribution histogram, prediction trends, fraud rate, system health
Tech Stack
Backend
ML
Monitoring
Frontend
Challenges
- Extreme class imbalance (0.17% fraud) — required careful evaluation metrics (AUC-PR over accuracy) and SMOTE-based rebalancing
- Designing a shadow deployment loop that doesn't double latency — both models score every request but shadow results are non-blocking
- Drift threshold calibration — PSI > 0.2 and KL > 0.1 trigger retraining without causing false alarms on normal distribution shift
- Cooldown constraint design — preventing retraining instability under noisy drift without masking real distributional change
Key Learnings
- The model is the easy part — drift detection, cooldown constraints, failure handling, and registry governance are where production ML gets hard
- Feature shift explanation transforms a number into an actionable insight — interpretable drift is far more valuable than raw metrics
- Cooldown mechanisms are essential in any automated trigger system — without them, noisy signals cause runaway retraining loops
- Explicit promotion gates (shadow → production) prevent silent model degradation that auto-promotion would miss