The Art and Engineering of Credit Risk Models
Why historic data might not be like future data (and what to do about it)
In credit risk modelling, it’s tempting to think that data alone holds all the answers and that more data automatically means better models. Feed enough data into an algorithm and it will “discover” the right relationships - or so the story goes. But experienced modellers know that data only tells part of the truth. It reflects past lending decisions, past customer behaviours, and past economic conditions, none of which perfectly represent the future.
That’s why good models aren’t just built; they’re engineered.
When Modellers Override the Data On Purpose
Engineering comes into play when a modeller deliberately adjusts or constrains the model to make it more logical, stable or interpretable.
Sometimes this is performed at the characteristic/feature level when understanding univariate predictive relationships. For example:
Forcing monotonic Weight of Evidence (WoE) patterns, so that higher income or stronger credit histories always correspond to lower risk, even if random data fluctuations suggest otherwise.
Example:
A variable like “Age of Oldest Account” may show a non-monotonic WoE pattern because a small, atypical group of very young customers had good outcomes (and maybe haven’t had time to ‘go bad’). The modeller enforces a monotonic pattern so that older credit histories imply lower risk, thus aligning with business reality.
Neutralising small or volatile groups, setting them to a reference level rather than trusting sparse data.
Example:
For “Employment Type”, if “Student” has only 40 cases, rather than using an unreliable WoE, the modeller assigns it a neutral score, preventing overreaction to noisy data.
Smoothing unstable patterns to prevent extreme jumps caused by temporary events or random chance.
Example:
A variable like “Number of Delinquent Accounts” might show a slight improvement in risk for customers with 3 delinquencies compared to 2, simply due to small sample noise. The modeller flattens or smooths that anomaly so the relationship remains intuitive and defendable, so that more delinquencies equate to higher risk.
Other engineering capabilities are sometimes required to control the overall model structure, such as:
Staged Models: Building in Layers
Sometimes, a modeller deliberately separates the problem into stages.
For example, Stage 1 might build a strong model on the core predictors - say, customer demographics and credit bureau history. Once that model is stable, the weights are frozen, and Stage 2 brings in additional data - such as behavioural or transactional variables.
This staged approach can be necessary when:
Different data sources have different refresh cycles and/or costs (e.g., bureau vs internal data).
The modeller wants to preserve interpretability of a foundational model while layering on incremental predictive power.
There is a risk that the more volatile predictors in later stages could distort or overfit the stable relationships found in the first stage.
In essence, staging allows the modeller to respect both stability and performance, rather than compromising one for the other.
For example, a lender may want to reserve the use of credit bureau data for only the borderline applications - for cost, timing or data access reasons. In this case, the modeller might design a two-stage approach:
Stage 1 builds a scorecard using only internal and application data, maximising its ability to confidently identify “super accepts” and “super declines.”
Only those applicants falling into the intermediate band, where the risk is less clear, are then passed to Stage 2, which brings in credit bureau data to refine the decision.
This setup ensures that the lender uses bureau data efficiently and strategically, focusing it where it adds the most value, while keeping the overall model interpretable and cost-effective.
Iterative Characteristic Addition: The Art of Balance
A common practice is to let an automated process select characteristics purely on statistical power - the top variables “win” the right to be in the model. But experienced modellers know that a great model isn’t just about power, it’s also about balance.
By adding characteristics iteratively, the modeller can control the blend of predictors:
Some that explain default risk directly.
Others that provide complementary or stabilising information.
And sometimes weaker predictors that improve long-term robustness or interpretability.
This engineering discipline guards against multicollinearity, overfitting and narrow models that perform brilliantly on one dataset but fail in production. In short, the goal isn’t to build the most powerful model it’s to build the most reliable and explainable one.
These are not shortcuts - they are deliberate design choices that make the model palatable, interpretable and robust for future conditions.
A cautionary tale: When data deceives
I once saw a model variable called “Previous Account Closure Reason” that looked incredibly predictive of default. One particular code had the highest default rate by far - until we found it was used internally to mark accounts that had already gone bankrupt. The model was unknowingly predicting the future using the future. It made the model look brilliant in development - but completely useless in production.
Without a deep understanding of the data lineage - how, when, and why each field is populated - even the best algorithm can be misled.
Engineering Control in the Age of Machine Learning
As machine learning methods such as Random Forests and XGBoost become more popular in credit risk, a new question emerges. How do we maintain the same level of engineering control and interpretability that traditional models provide?
In a logistic regression or scorecard, the modeller can see and shape every relationship - enforcing monotonicity, neutralising sparse groups, or staging predictors to manage stability. Each adjustment is transparent and explainable.
Machine learning models, by contrast, abstract that control away. They automatically capture complex nonlinearities and interactions, but they do so opaquely. The modeller can no longer directly adjust or constrain individual effects; instead the algorithm dictates the relationships it finds.
This creates challenges:
Controlling predictive patterns is harder. Ensuring logical relationships (for example, higher income implying lower risk) is not straightforward without extensive preprocessing or post-model calibration.
Interpretability decreases. Traditional scorecards naturally tell a story - variable by variable, group by group. Machine learning models, while powerful, often lose that simplicity.
Engineering shifts elsewhere. Instead of shaping the model directly, modellers must engineer the inputs and framework - through careful feature construction, constraints, and explainability tools like SHAP or partial dependence plots.
In other words, machine learning doesn’t remove the need for engineering - it changes where it happens. The modeller’s craftsmanship moves from adjusting coefficients to designing robust data transformations, validation methods and interpretability layers around the model. There is a clear trade-off. Machine learning offers extraordinary predictive power, but traditional modelling offers clarity, control, and regulatory defensibility. The best approach may be to use both - leveraging machine learning to discover relationships and inform the design of engineered, transparent models that can stand up to regulatory and operational scrutiny.
Conclusion: The Modeller as Engineer
As credit risk modellers, we’re building forecasting tools, not mirrors. The goal isn’t to perfectly describe the past, but to design something that stands up to the future.
The engineering mindset helps modellers question data, reinforce logic, and design models that stand the test of time. That’s what separates a quick analysis from a dependable model.
About Paragon Business Solutions
Paragon develops software tools that help credit risk teams build, validate, and manage models with transparency, control, and efficiency. Our Modeller tool embodies this engineering philosophy - giving users both flexibility and rigour in model development.
If you’d like to learn more about how Modeller supports end-to-end credit model development, we’d love to talk. Please get in touch.