NLP Classification / Selective Routing
Financial Complaint Auto-Routing with NLP
Leakage-safe eight-class CFPB complaint-routing study comparing a locked TF-IDF + Linear SVM benchmark with a frozen DistilBERT challenger, model-specific selective routing, and retrospective 2025 comparison.
Executive Summary
Decision support with an intentional review path
Problem
CFPB complaint narratives are unstructured and must be assigned consistently to one of eight product categories before entering the appropriate workflow.
Approach
Two frozen models were evaluated with separately selected policies that recommend either an eligible Auto-Route recommendation or Human Review.
Scope
Offline decision-support study with selective routing and Human Review; not a production deployment.
Business Problem
Route clear cases without forcing uncertain ones
Routing mistakes can send a complaint into the wrong operational workflow. The project therefore evaluates selective routing rather than treating every model output as safe to automate.
- Public CFPB complaint narratives contain free-form, unstructured text.
- Each narrative is classified into one of eight product categories.
- Weak, ambiguous, tied, invalid, or otherwise unusable model signals remain eligible for Human Review.
Leakage-Safe Data Design
Correct the split before comparing models
The strongest methodological correction was removing duplicate-text leakage and rebuilding the 2024 evaluation boundary around normalized-text groups.
39.39%
3,876 of 9,840 original test rows shared normalized text with training
0 overlap
between development and final-test normalized-text groups after redesign
Audit
Duplicate normalized complaint text exposed leakage in the original row-level split.
Remove leakage
Conflicting-label text groups and repeated same-label extras were removed.
Split safely
Group-aware 2024 development and final-test partitions preserved text-group boundaries.
Verify
Normalized-text overlap between the 2024 development and final-test partitions was zero.
Hold separate
Kept outside 2024 development; after V1 temporal evaluation, V2 used it only for the precommitted retrospective comparison.
Benchmark vs. Challenger
Parallel models, separately frozen policies
V1 and V2 are compared on matched evidence; they are not presented as a sequential replacement pipeline.
Temporal Benchmark
V1: TF-IDF + Linear SVM
Classical NLP benchmark selected through group-aware development evaluation, locked before final scoring, and retained as the temporally validated reference model.
Frozen Challenger
V2: DistilBERT
Transformer challenger trained using the locked 2024 development data and evaluated on the shared benchmark. It improved classification metrics but was not promoted beyond frozen challenger status.
Key Results
Shared 2024 benchmark
Version 2 improved classification and coverage, while Version 1 retained slightly better routed accuracy and misroute rate. The trade-off did not establish a clear promotion case.
Macro F1
- V1
- 0.7671
- V2
- 0.7949
V2 +0.0278
Accuracy
- V1
- 87.12%
- V2
- 88.82%
V2 +1.70 pts
Coverage
- V1
- 77.05%
- V2
- 81.77%
V2 +4.72 pts
Routed Accuracy
- V1
- 95.03%
- V2
- 94.76%
V1 +0.27 pts
Misroute Rate
- V1
- 4.97%
- V2
- 5.24%
V1 lower by 0.27 pts
Temporal Evidence
2025 Retrospective Comparison
A precommitted retrospective comparison on the leakage-resistant primary cohort. This was not a new untouched Version 2 holdout.
Primary cohort
30,156
leakage-resistant rows
Macro F1
- V1
- 0.7527
- V2
- 0.7620
Coverage
- V1
- 72.51%
- V2
- 75.71%
Routed Accuracy
- V1
- 92.03%
- V2
- 91.74%
Misroute Rate
- V1
- 7.97%
- V2
- 8.26%
Both models and policies remained frozen. V2 again improved aggregate classification and coverage, but did not improve routed risk on the headline cohort. Both models also weakened relative to their shared 2024 benchmark.
Failure Analysis
Where better classification did not mean safer routing
Category-level evidence explains why aggregate gains were not enough to promote the challenger.
Debt Collection
23.56% → 29.84%
Primary-cohort misroute rate increased from V1 to V2.
Student Loan
16.67% → 32.08%
V2 nearly doubled the primary misroute rate, and its primary F1 was 0.0257 below V1.
Risk–Coverage Trade-off
More routes, uneven risk
V2 routed more complaints, but broader coverage did not consistently reduce routed error across categories.
Model Trade-offs
Better benchmark scores came with higher complexity
Model selection considered classification, selective-routing risk, and practical complexity together.
V1 — Current Benchmark
TF-IDF + Linear SVM
3.23 MiBSmaller CPU-only stack, substantially faster on the measured local notebook benchmark, and slightly lower routed risk under the locked 2024 policy.
V2 — Frozen Challenger
DistilBERT
256.35 MiBAbout 79× larger, with higher shared-benchmark Macro F1 and coverage but a slower measured local inference path and greater dependency, monitoring, privacy, and governance complexity.
Selective Routing
Auto-Route recommendation or Human Review
Each model uses its own development-selected routing policy. Eligible cases receive an Auto-Route recommendation, while uncertain or unusable signals remain in Human Review.
Model scores are not treated as one shared probability scale.
Auto-Route
When a model-specific policy passes both score and margin criteria, the case is eligible for an Auto-Route recommendation to the predicted product workflow.
Human Review
Uncertain, tied, invalid, non-finite, or otherwise unusable signals remain with a reviewer. This is an intentional oversight outcome, not a model failure state.
Final Decision
Retain V1 as the benchmark
Retain Version 1 as the temporally validated benchmark. Version 2 improved shared-benchmark classification performance and coverage, but routed-risk improvement was not established, complexity increased, and retrospective 2025 evidence remained mixed. Version 2 remains a frozen challenger pending evaluation on a new untouched period.
Evaluation Boundaries
Limitations and Next Validation
A concise boundary between offline evidence and future validation work.
- The 2025 V2 comparison is retrospective; promotion evidence requires evaluation on a new untouched period.
- This is an offline decision-support study with no production API, operational queue, monitoring system, or deployment.
- Linear SVM and DistilBERT outputs are model-specific signals, not calibrated values on a shared probability scale.
- Routing thresholds are development-selected assumptions; no stakeholder-approved limits, realized savings, workload reduction, or operational-impact claims are made.
What This Demonstrates
Applied Data Science & NLP signals
The study connects careful evaluation design with transparent model and decision-policy trade-offs.
Leakage-safe evaluation
Group-aware validation
Classical ML vs. Transformer comparison
Temporal validation
Human-in-the-loop decision policies
Error and category-risk analysis
Reproducible model evaluation
Performance and complexity trade-offs
Technology and Methods
Implementation stack
Evidence and Next Actions
Inspect the project evidence
Review the source, methodology, matched benchmark, and retrospective evaluation directly.