Overview
Production-grade ML pipeline for Banking77 intent classification, covering the full lifecycle from data validation and model training to serving, monitoring, and explainability. Built to demonstrate real ML engineering beyond API integration, trained on Kaggle's free T4 GPU.
Key Features
- ModernBERT Fine-Tuning: Encoder-only transformer with FlashAttention, fine-tuned on 13,083 banking queries across 77 intent classes
- ONNX INT8 Quantization: 4.52x inference speedup (10.11ms per prediction on CPU) with model size reduced from 575MB to 146MB
- Full Observability: Prometheus metrics, Grafana dashboards, AlertManager with configurable thresholds (confidence, latency, error rate)
- Drift Detection: Evidently monitors for data and prediction drift in production
- Explainability: SHAP feature importance and attention heatmaps showing which tokens drive predictions
Technical Achievements
Model Performance
- 91.3% accuracy (91.4% F1 macro) on Banking77 dataset
- Outperforms classical baselines: SVM (87.9%), Logistic Regression (84.5%), Random Forest (83.9%)
- Class-weighted cross-entropy loss handles 77-class imbalance effectively
Training Optimizations
- Adafactor optimizer for memory efficiency over AdamW
- Gradient checkpointing + FP16 mixed precision to fit on free-tier T4 GPU (16GB VRAM)
- Batch size 2 with gradient accumulation to effective batch size 32
- Early stopping with F1 macro patience of 3 epochs
Production Infrastructure
- FastAPI serving with Redis caching, rate limiting, and low-confidence flagging for human review
- Dagster orchestration for pipeline management
- Docker Compose deployment with 5 services (API, Redis, Dashboard, Prometheus, Grafana)
- 86 passing tests with comprehensive coverage
- Feedback loop: Users submit corrections tied to specific predictions via UUID tracking
What I Learned
This project filled a real gap in my experience. I'd integrated plenty of AI APIs before, but training, fine-tuning, and deploying my own model was a completely different challenge. The biggest lesson was how much production ML is about everything around the model: data validation with Pandera, experiment tracking with MLflow, quantization trade-offs, monitoring for drift, and building feedback loops.
Training on a free T4 GPU forced creative optimization. Gradient checkpointing, mixed precision, and Adafactor instead of AdamW were all necessary to make ModernBERT fit in 16GB VRAM. Those constraints taught me more about efficient training than having unlimited compute would have.
Tech Stack
Model: ModernBERT-base, PyTorch, HuggingFace Transformers Inference: ONNX Runtime (INT8 dynamic quantization), FastAPI Data: Pandera validation, DVC versioning, Banking77 dataset Tracking: MLflow experiments, Dagster orchestration Monitoring: Prometheus, Grafana, AlertManager, Evidently drift detection Explainability: SHAP, attention heatmaps Dashboard: Streamlit with Plotly Infrastructure: Docker Compose, Redis, uv