NLP Classification / Selective Routing
Financial Complaint Auto-Routing with NLP
Leakage-safe eight-class CFPB complaint-routing study comparing a locked TF-IDF + Linear SVM benchmark with a frozen DistilBERT challenger, using development-selected routing policies and retrospective 2025 evaluation.
Executive Summary
Decision support with an intentional review path
Problem
CFPB complaint narratives are unstructured and must be assigned consistently to one of eight product categories before entering the appropriate workflow.
Approach
Two frozen models were evaluated with separately selected policies that recommend either an eligible automatic route or Human Review.
Scope
Offline decision-support study with selective routing and Human Review; not a production deployment.
Business Problem
Route clear cases without forcing uncertain ones
Routing mistakes can send a complaint into the wrong operational workflow. The project therefore evaluates selective routing rather than treating every model output as safe to automate.
- Public CFPB complaint narratives contain free-form, unstructured text.
- Each narrative is classified into one of eight product categories.
- Weak, ambiguous, tied, invalid, or otherwise unusable model signals remain eligible for Human Review.
Leakage-Safe Data Design
Correct the split before comparing models
The strongest methodological correction was removing duplicate-text leakage and rebuilding the 2024 evaluation boundary around normalized-text groups.
Audit
Duplicate normalized complaint text exposed leakage in the original row-level split.
Remove leakage
Conflicting-label text groups and repeated same-label extras were removed.
Split safely
Group-aware 2024 development and final-test partitions preserved text-group boundaries.
Verify
Normalized-text overlap between the 2024 development and final-test partitions was zero.
Hold separate
The 2025 cohort remained separate for the precommitted retrospective analysis.
Benchmark vs. Challenger
Parallel models, separately frozen policies
V1 and V2 are compared on matched evidence; they are not presented as a sequential replacement pipeline.
Temporal Benchmark
V1: TF-IDF + Linear SVM
Classical NLP benchmark selected through group-aware development evaluation, locked before final scoring, and retained as the temporally validated reference model.
Frozen Challenger
V2: DistilBERT
Transformer challenger trained using the locked 2024 development data and evaluated on the shared benchmark. It improved classification metrics but was not promoted to a final production champion.
Key Results
Shared 2024 benchmark
Version 2 improved classification and coverage, while Version 1 retained slightly better routed accuracy and misroute rate. The trade-off did not establish a clear production winner.
Macro F1
- V1
- 0.7671
- V2
- 0.7949
V2 +0.0278
Accuracy
- V1
- 87.12%
- V2
- 88.82%
V2 +1.70 pts
Coverage
- V1
- 77.05%
- V2
- 81.77%
V2 +4.72 pts
Routed Accuracy
- V1
- 95.03%
- V2
- 94.76%
V1 +0.27 pts
Misroute Rate
- V1
- 4.97%
- V2
- 5.24%
V1 lower by 0.27 pts
Temporal Evidence
2025 Retrospective Comparison
A precommitted comparison on a leakage-resistant primary cohort, without changing either frozen model or routing policy.
Primary cohort
30,156
leakage-resistant rows
Both frozen models weakened on the stricter 2025 primary cohort. Because this was a retrospective comparison rather than a new untouched validation for V2, V1 remains the temporal benchmark and V2 remains the frozen challenger. Neither model nor routing policy was changed before the comparison.
Selective Routing
Auto-Route recommendation or Human Review
Each model uses its own development-selected routing policy. Eligible cases receive an Auto-Route recommendation, while uncertain or unusable signals remain in Human Review.
Model scores are not treated as one shared probability scale.
Auto-Route
When a model-specific policy passes both required signal thresholds, the case is eligible for an automatic-route recommendation to the predicted product workflow.
Human Review
Uncertain, tied, invalid, non-finite, or otherwise unusable signals remain with a reviewer. This is an intentional oversight outcome, not a model failure state.
Final Decision
Retain V1 as the benchmark
Retain Version 1 as the temporally validated benchmark. Version 2 improved shared-benchmark classification performance and coverage, but it did not establish a clear routing-risk or operational advantage. Version 2 remains a frozen challenger pending evaluation on a new untouched period.
Evaluation Boundaries
Limitations and Next Validation
A concise boundary between offline evidence and future validation work.
- The 2025 V2 comparison is retrospective; promotion evidence requires evaluation on a new untouched period.
- This is an offline decision-support study with no production API, operational queue, monitoring system, or deployment.
- Linear SVM and DistilBERT outputs are model-specific signals, not calibrated values on a shared probability scale.
- Routing thresholds are development-selected assumptions; no stakeholder-approved limits, realized savings, workload reduction, or operational-impact claims are made.
Technology and Methods
Implementation stack
Evidence and Next Actions
Review the full project
Use the source repository and report for detailed methodology, category-level results, routing assumptions, and reproducibility notes.