XGBoost Summer Research Roundup

Since its publication in 2014, Tianqi Chen’s XGBoost has become ubiquitous across predictive applications, touching every domain. And while neural networks have come to dominate sequence-learning tasks in recent years, XGBoost is still a first resort for tabular modeling. Its predominance has driven a decade of research on its properties and extensions, continuing to this day. Here, I’ll summarize some highlights that I found this summer. Hopefully this is a useful reference for data scientists, but also gives security practitioners an idea of where the field of exploit prediction has room to innovate.

A Primer for Security Folks

In the context of EPSS, a machine learning model is a function that maps features of a vulnerability, such as its age or CVSS vector, to the probability that it gets exploited. If we considered all possible functions, there’d be too many and we could never choose one efficiently, so we use algorithms like XGBoost to narrow down the set of possible functions in a way that makes it efficient to pick one. That chosen function is EPSS - its outputs are the scores that you see on our API. These research papers help us address remaining challenges in choosing the EPSS function, from deciding which vulnerability attributes actually matter, to ensuring that the model generalizes across all vulns, instead of only specializing in a narrow slice.

qShap: Fast Shapley Decomposition of R2for Gradient-Boosted Trees

Zhongli Jiang, Min Zhang, Dabao Zhang    8/25/2026

The SHAP value for a feature in a model is its exact net contribution to the model’s output, that is how much the model’s output would be expected to change if we didn’t know that feature.  The paper begins by developing a quantity that can be assigned to each feature, obeys additivity, and sums over all features to the model’s R2. They call this quantity RSQ-SHAP. It’s a bit trickier to compute than SHAP because it contains quadratic terms, but the authors demonstrate that like SHAP, one can employ a dynamic programming trick to calculate it for all features in quadratic time with respect to the number of leaves per tree or tree depth. 

A model’s R2 can be thought of as the fraction of the variance in the true outcomes that the model can use the given features to explain. In the context of EPSS, it’s a performance metric: a better model can use the features of vulnerabilities that we know to do a better job predicting whether the vulnerability gets exploited. Feature selection is a particularly useful application of this paper. Previously, vanilla SHAP played a role in how we choose whether a new feature is useful enough for the model. But this is a flawed approach because sometimes, large swings in some samples don’t have a major impact on model accuracy. In fact, an alternative interpretation of RSQ-SHAP relates to the improvement to quadratic loss due to a feature. In my view, this is the most practically useful recent XGBoost paper, and we plan to employ it here at Empirical. 

A Functional Central Limit Theorem for Kernel Gradient Flow and Infinitesimal Gradient Boosting

Clement Dombry, Jean-Jil Duchamps    6/25/2026

This is a long, highly theoretical, and very cool paper on boosting with trees, but with a practical takeaway in the end. Most of the theoretical canon on boosting focuses on whether the model will eventually be successful in approximating ground truth, but tells us little about the path to arrive at that success. This is relevant because we never run XGBoost with an infinite number of samples, or for an infinite number of rounds - the finite universe restricts us. So it’s important to understand not just that we’ll approach ground truth, but that we’ll do so with a reasonable amount of data. 

The authors approximate boosting over a family similar to XGBoost’s decision trees as a differential equation by using a low learning rate, high number of boosting rounds limit. Employing a series of mathematical and statistical tools, they demonstrate that general boosting achieves a n rate, meaning it will approximate its performance on infinite data in reasonable time, on par with most statistical inference methods. 

Like most theoretical work, the authors’ assumptions don’t apply directly to real-world use cases like EPSS. But it’s reassuring to get some evidence that we don’t need millions of independent vulnerabilities to make a good predictor of whether they get exploited, overall a positive signal on EPSS’s performance. The drawback of only needing a small sample size is that a large sample size only helps so much. We’ve observed middling returns to scale using XGBoost for vulnerability forecasting, and this paper provides some evidence that it’s worth testing non-boosting approaches for such a data-rich regime like ours. 

Convolution Smoothed Quantile Regression for XGBoost

Mandy Yao, Meredith Franklin    8/15/2026

In standard regression tasks, we aim to estimate the expected value of an outcome, given the data. But what if instead of understanding the 50th percentile outcome, we wanted to know the 90th percentile outcome, so that we could be prepared for that risk? This is the quantile regression task, and it’s usually accomplished by training a standard regression model on a “pinball loss” that penalizes misses away from the direction of your quantile more than misses in its direction. But pinball loss is piecewise linear, so it doesn’t provide any of the curvature information that XGBoost uses to converge. The authors provide a method for smoothing pinball loss in a way that’s rather general, but maintains the loss function’s convexity. 

This paper isn’t especially useful for EPSS, where we predict 0/1 outcomes over whether vulnerabilities get exploited. But if we wanted to take our data and predict something continuous instead, like the total impact of a breach of the vuln, it could come in handy.

Reducing Learner Redundancy in Boosting via Residual Orthogonalization

Ye Su, Jipeng Guo, Xin Xu, Gangchun Zhang, Jinxin Chen, Di Wu, Longlong Zhao

This paper points out that XGBoost is prone to spending multiple rounds building trees, only to have them all update our model in the same direction. Specifically, this occurs when the residuals that we build trees on in each round are correlated with each other. If all our trees are similar, we’ll overfit into that one direction. To resolve this problem, the authors propose projecting the residuals before each boosting round into a subspace that’s orthogonal to previous residuals. They run experiments demonstrating large positive gains, as long as the learning rate is high enough to make sure we’re not neglecting important discovered directions. 

In the context of EPSS, a tree that only updates in correlated directions over and over again is roughly one that keeps missing a few types of vulns by a lot, so it spends time getting really good at those and neglects the others. This is especially bad if, for example, those are older vulnerabilities that don’t reflect the modern ecosystem. If their method works as stated, the authors’ suggestion would help ensure that EPSS is trained to generalize across diverse types of vulnerabilities, without major weaknesses. 

Most exploit prediction still stops at a single number. The research above shows there's a lot more headroom: better attribution, less overfitting to narrow vuln types, richer outputs than a binary score. We're building that into Empirical right now. Want to see what a model that catches the vulnerabilities other tools miss looks like on your own backlog? https://www.empiricalsecurity.com/contact