PROJECT · SPEC SHEET · PYTHON · SCIKIT-LEARN · LIGHTGBM
Mortgage Default Risk Model
Three models tie at 0.832 AUC on 2017 loans; what separates them is whether their scores can be read as probabilities
Which Fannie Mae mortgages go 180+ days delinquent within two years, judged only from what was known at origination? Trained on 2015–16 loans, tested on 2017.
Loans modeled
602,838
10% sample, 2015–17
Default rate
0.30%
2017 test · ≈ 1 in 330
Test ROC AUC
0.832
all three models tie
Catch at 100:1
39%
flagging 6.9% of loans
Where do you draw the line?
Drag the cost ratio
- Flagged
- 6.9%
- Caught
- 39%
- vs flag nothing
- −17%
At 100:1 the model earns its keep: flagging 6.9% of loans cuts total cost by 17%. Flagging starts to pay at about 44:1.
Same ranking, different honesty
Do the scores mean what they say?
BRIER · LR 0.207 · RF 0.047 · LGBM 0.003
1 bin with no defaults not shown on the log scale
| Mean predicted | Observed rate | Loans |
|---|---|---|
| 0.01% | 0.00% | 18,121 |
| 0.02% | 0.04% | 18,120 |
| 0.03% | 0.04% | 18,121 |
| 0.04% | 0.02% | 18,120 |
| 0.06% | 0.10% | 18,121 |
| 0.09% | 0.17% | 18,120 |
| 0.15% | 0.24% | 18,120 |
| 0.26% | 0.31% | 18,121 |
| 0.50% | 0.61% | 18,120 |
| 1.62% | 1.51% | 18,121 |
All three rank loans equally well. Only LightGBM's scores can be read as probabilities, so it sets the threshold.
Where defaults cluster
Default rate by vintage and region
Hover or focus a point
| Vintage | Region | Default rate | Loans |
|---|---|---|---|
| 2015Q1 | FL | 0.17% | 2,325 |
| 2015Q1 | TX | 0.12% | 3,459 |
| 2015Q1 | rest of US | 0.15% | 39,552 |
| 2015Q2 | FL | 0.30% | 2,968 |
| 2015Q2 | TX | 0.23% | 4,324 |
| 2015Q2 | rest of US | 0.15% | 45,170 |
| 2015Q3 | FL | 0.25% | 2,843 |
| 2015Q3 | TX | 0.33% | 4,259 |
| 2015Q3 | rest of US | 0.17% | 40,604 |
| 2015Q4 | FL | 0.12% | 2,570 |
| 2015Q4 | TX | 0.12% | 3,398 |
| 2015Q4 | rest of US | 0.17% | 35,577 |
| 2016Q1 | FL | 0.69% | 2,615 |
| 2016Q1 | TX | 0.53% | 3,571 |
| 2016Q1 | rest of US | 0.18% | 35,745 |
| 2016Q2 | FL | 1.22% | 3,453 |
| 2016Q2 | TX | 0.68% | 4,587 |
| 2016Q2 | rest of US | 0.15% | 48,920 |
| 2016Q3 | FL | 1.35% | 3,991 |
| 2016Q3 | TX | 0.70% | 5,313 |
| 2016Q3 | rest of US | 0.15% | 62,444 |
| 2016Q4 | FL | 1.31% | 3,660 |
| 2016Q4 | TX | 0.58% | 4,693 |
| 2016Q4 | rest of US | 0.14% | 55,592 |
| 2017Q1 | FL | 1.43% | 3,086 |
| 2017Q1 | TX | 0.76% | 3,575 |
| 2017Q1 | rest of US | 0.26% | 35,849 |
| 2017Q2 | FL | 1.02% | 4,022 |
| 2017Q2 | TX | 0.72% | 4,415 |
| 2017Q2 | rest of US | 0.21% | 45,176 |
| 2017Q3 | FL | 0.75% | 3,483 |
| 2017Q3 | TX | 0.41% | 4,148 |
| 2017Q3 | rest of US | 0.22% | 45,136 |
| 2017Q4 | FL | 0.33% | 2,102 |
| 2017Q4 | TX | 0.20% | 2,484 |
| 2017Q4 | rest of US | 0.25% | 27,729 |
The Gulf spike is hurricane forbearance after Harvey and Irma. Moving the label from 90 to 180 days removed most of it, and the remainder is why every model is also scored without FL+TX.
Where risk compounds
Default rate by credit score and loan-to-value
Hover or focus a cell
| Credit score | Loan-to-value | Default rate | Loans |
|---|---|---|---|
| <620 | <=80 | fewer than 30 loans | 5 |
| <620 | 80-90 | fewer than 30 loans | 1 |
| <620 | 90-95 | fewer than 30 loans | 7 |
| 620-679 | <=80 | 0.88% | 38,807 |
| 620-679 | 80-90 | 1.51% | 4,710 |
| 620-679 | 90-95 | 1.85% | 5,962 |
| 620-679 | >95 | 2.28% | 1,711 |
| 680-739 | <=80 | 0.28% | 102,128 |
| 680-739 | 80-90 | 0.42% | 19,425 |
| 680-739 | 90-95 | 0.62% | 25,802 |
| 680-739 | >95 | 0.67% | 8,397 |
| 740+ | <=80 | 0.05% | 291,218 |
| 740+ | 80-90 | 0.10% | 44,129 |
| 740+ | 90-95 | 0.15% | 49,258 |
| 740+ | >95 | 0.25% | 10,989 |
Low credit score and high loan-to-value compound.
What drives the score
Top 8 features
- Credit score0.113
- State0.032
- Debt-to-income0.013
- Number of borrowers0.013
- Interest rate0.010
- Loan-to-value0.007
- Co-borrower credit score0.006
- Loan amount0.006
Drop in test ROC AUC when shuffled
Data, label and split
Origination fields only · out-of-time test (2017) · right-censored loans dropped · missing FICO/DTI imputed with indicators · synthetic-data tests for every module
Results table
| Model | ROC AUC | ROC AUC excl. FL+TX | PR AUC | Brier |
|---|---|---|---|---|
| Logistic | 0.832 | 0.828 | 0.017 | 0.207 |
| Forest | 0.832 | 0.824 | 0.016 | 0.047 |
| LightGBM | 0.832 | 0.824 | 0.016 | 0.003 |
PR AUC is judged against its no-skill baseline, the base rate of 0.003.
Limits
- A 10% loan sample of three years of originations; results are not tuned or cross-validated beyond early stopping.
- Origination data only: no payment history, home-price changes or local unemployment, which is why the models plateau.
- The 2017 hurricanes still affect the label for Florida and Texas; the robustness columns bound their influence rather than removing it.
- The cost ratios are illustrative, not Fannie Mae's actual loss and servicing costs.