Matthew Kim
← All projects

PROJECT · SPEC SHEET · PYTHON · SCIKIT-LEARN · LIGHTGBM

Mortgage Default Risk Model

Three models tie at 0.832 AUC on 2017 loans; what separates them is whether their scores can be read as probabilities

Which Fannie Mae mortgages go 180+ days delinquent within two years, judged only from what was known at origination? Trained on 2015–16 loans, tested on 2017.

Loans modeled

602,838

10% sample, 2015–17

Default rate

0.30%

2017 test · ≈ 1 in 330

Test ROC AUC

0.832

all three models tie

Catch at 100:1

39%

flagging 6.9% of loans

LIVE

Where do you draw the line?

Drag the cost ratio

100:1
Flagged
6.9%
Caught
39%
vs flag nothing
−17%

At 100:1 the model earns its keep: flagging 6.9% of loans cuts total cost by 17%. Flagging starts to pay at about 44:1.

TOGGLE

Same ranking, different honesty

Do the scores mean what they say?

BRIER · LR 0.207 · RF 0.047 · LGBM 0.003

Calibration curve, LightGBMReliability curve on log-log axes: mean predicted default probability against observed default rate, per bin. The dashed diagonal is perfect calibration. Highlighted: LightGBM.0.01%0.01%0.1%0.1%1%1%10%10%100%100%predicted probabilityobserved rate

1 bin with no defaults not shown on the log scale

Calibration bins, LightGBM
Mean predictedObserved rateLoans
0.01%0.00%18,121
0.02%0.04%18,120
0.03%0.04%18,121
0.04%0.02%18,120
0.06%0.10%18,121
0.09%0.17%18,120
0.15%0.24%18,120
0.26%0.31%18,121
0.50%0.61%18,120
1.62%1.51%18,121

All three rank loans equally well. Only LightGBM's scores can be read as probabilities, so it sets the threshold.

HOVER

Where defaults cluster

Default rate by vintage and region

FLTXRest of US
Default rate by vintage and regionLines for FL, TX and the rest of the US across origination vintages 2015Q1 to 2017Q4. A gap in a line means fewer than 30 loans. Each point can be focused to read its rate and loan count.0.0%0.4%0.8%1.1%1.5%2015Q12015Q32016Q12016Q32017Q12017Q3

Hover or focus a point

Default rate by vintage and region
VintageRegionDefault rateLoans
2015Q1FL0.17%2,325
2015Q1TX0.12%3,459
2015Q1rest of US0.15%39,552
2015Q2FL0.30%2,968
2015Q2TX0.23%4,324
2015Q2rest of US0.15%45,170
2015Q3FL0.25%2,843
2015Q3TX0.33%4,259
2015Q3rest of US0.17%40,604
2015Q4FL0.12%2,570
2015Q4TX0.12%3,398
2015Q4rest of US0.17%35,577
2016Q1FL0.69%2,615
2016Q1TX0.53%3,571
2016Q1rest of US0.18%35,745
2016Q2FL1.22%3,453
2016Q2TX0.68%4,587
2016Q2rest of US0.15%48,920
2016Q3FL1.35%3,991
2016Q3TX0.70%5,313
2016Q3rest of US0.15%62,444
2016Q4FL1.31%3,660
2016Q4TX0.58%4,693
2016Q4rest of US0.14%55,592
2017Q1FL1.43%3,086
2017Q1TX0.76%3,575
2017Q1rest of US0.26%35,849
2017Q2FL1.02%4,022
2017Q2TX0.72%4,415
2017Q2rest of US0.21%45,176
2017Q3FL0.75%3,483
2017Q3TX0.41%4,148
2017Q3rest of US0.22%45,136
2017Q4FL0.33%2,102
2017Q4TX0.20%2,484
2017Q4rest of US0.25%27,729

The Gulf spike is hurricane forbearance after Harvey and Irma. Moving the label from 90 to 180 days removed most of it, and the remainder is why every model is also scored without FL+TX.

HOVER

Where risk compounds

Default rate by credit score and loan-to-value

Default rate by credit score and loan-to-valueHeatmap of default rate: credit score band across, loan-to-value band down. Darker cells have higher rates; hatched cells have fewer than 30 loans. Each cell can be focused to read its rate.<620620-679680-739740+<=8080-9090-95>95credit scoreloan-to-valuefewer than 30 loans0.88%0.28%0.05%fewer than 30 loans1.51%0.42%0.10%fewer than 30 loans1.85%0.62%0.15%no loans2.28%0.67%0.25%

Hover or focus a cell

Default rate by credit score and loan-to-value
Credit scoreLoan-to-valueDefault rateLoans
<620<=80fewer than 30 loans5
<62080-90fewer than 30 loans1
<62090-95fewer than 30 loans7
620-679<=800.88%38,807
620-67980-901.51%4,710
620-67990-951.85%5,962
620-679>952.28%1,711
680-739<=800.28%102,128
680-73980-900.42%19,425
680-73990-950.62%25,802
680-739>950.67%8,397
740+<=800.05%291,218
740+80-900.10%44,129
740+90-950.15%49,258
740+>950.25%10,989

Low credit score and high loan-to-value compound.

What drives the score

Top 8 features

  1. Credit score0.113
  2. State0.032
  3. Debt-to-income0.013
  4. Number of borrowers0.013
  5. Interest rate0.010
  6. Loan-to-value0.007
  7. Co-borrower credit score0.006
  8. Loan amount0.006

Drop in test ROC AUC when shuffled

✂ HOW IT WAS BUILT
Data, label and split

Origination fields only · out-of-time test (2017) · right-censored loans dropped · missing FICO/DTI imputed with indicators · synthetic-data tests for every module

Results table
ModelROC AUCROC AUC excl. FL+TXPR AUCBrier
Logistic0.8320.8280.0170.207
Forest0.8320.8240.0160.047
LightGBM0.8320.8240.0160.003

PR AUC is judged against its no-skill baseline, the base rate of 0.003.

Limits
  • A 10% loan sample of three years of originations; results are not tuned or cross-validated beyond early stopping.
  • Origination data only: no payment history, home-price changes or local unemployment, which is why the models plateau.
  • The 2017 hurricanes still affect the label for Florida and Texas; the robustness columns bound their influence rather than removing it.
  • The cost ratios are illustrative, not Fannie Mae's actual loss and servicing costs.

Code on GitHub ↗