Looking for the AI vulnerability flood
Why a firehose worth of AI vulns hasn’t fully manifested itself in the CVE ecosystem
Key Points
Reports are that AI is finding a lot more vulnerabilities
Cataloguing a vulnerability as a CVE is a human process
24 of the top 100 CNAs have accelerated their rate of vulnerability publication in the last 18 months. Others have held steady or even declined.
Vulnerabilities were accelerating before AI, prioritization is going to be even more important in the post AI era.
Frontier AI models seem to be living up to the hype with respect to their ability to find new vulnerabilities. Whether it’s Mozilla reporting that the now banned Mythos model was able to find 271 vulnerabilities in a recent release or open source projects indicating that the flood of AI slop is now more of a flood of legitimate AI vulnerabilities (even if they aren’t Mythos powered).
Figure 1 — Good AI vulns are coming.
But we at Empirical generally don’t take anec-data1 at face value. Given that my title is “Head of Modeling” it’s worth it to do some modeling and see if the data support the breathlessly discussed increase. First let’s start by plotting the monthly published CVE count since 2017 across all CNAs.
Figure 2 — Monthly published CVE counts across all CNAs. Orange line is a segmented Poisson Count model.
What’s interesting about Figure 2 is that we have seen a constant increase in the number of CVEs published every month, with an increase in that rate2 increasing in October of 2022, certainly before the AI era.
But it’s important to remember that CVE publication is neither a natural or centralized process. CVE publication is federated across hundreds of CVE Numbering Authorities (CNAs), each with their own mission and quirks. It’s a deeply human process with arguments about whether a vulnerability report actually constitutes a CVE, whether CNA has the bandwidth to address everything that gets reported, and which vuln reports get prioritized. GitHub recently wrote about how they approach things as they navigate the vulnerability waters. Seems like a good transition point to ask “are any CNAs seeing an AI driven uptick?
Figure 3 — Some of these top 16 CNAs are publishing at a steady rate, while some seem to have seen a substantial recent shift.4
What’s clear from Figure 3 is that among the top 16 we’ve seen a relatively smooth increase in the cumulative vulns at various rates for most CNAs, but we’ve seen some rocket up recently, namely Patchstack, VulnDB, Wordfence, VulnCheck, and notably Github which actually looks like there is a hitch around the end of 2025.
We can be a bit more rigorous about this and ask exactly when these shifts occur in the top 100 most prolific CNAs.
Figure 4 — Cumulative vulns published by the top 100 CNAs, with inferred AI accelerated periods present.
What figure 4 shows is that only 24 of the top 100 CNAs have experienced a recent, statistically measurable acceleration in the vulnerability publication rate. Others like VulnCheck and VulnDB, have just been constantly accelerating since they started publishing, indicating that they are not reacting to a flood of AI vulnerability reports, but rather have been accelerating their capabilities the entire time.
So what are we to do? In a word, “prioritize”. For CNAs there will be a need to address the vulnerabilities that are reported to them, not as a queue, but with careful consideration to which ones present the highest risk, and get that information out as quickly and completely as possible. For vulnerability managers, calls to better prioritize are now (at least) 13 years old; The state of vulnerability publication has been overwhelming for years and the problem of “what do I address?” is not going to get better in the AI era.
Footnotes
1Portmanteau of “anecdote” and “data”, meaning just because some folks lived experience says one thing, doesn’t mean it’s evidence.
2How did we find this breakpoint? My colleague Jay Jacobs has insisted there is too much statistical modeling in my security writing so I am going to stick to the actual security results and implications in the main text rather than discursive discussions of the correct modeling technique. This is in spite of the fact that our company is called “Empirical Security” and we are building our reputation on doing security data modeling right. Also apparently he is allowed to write long discursive philosophy and semantic discussions without getting to the point about “what is a vulnerability?”
So how did we arrive at the model above? We use a Poisson model because we are modeling counts, and that is what these models are for. We do it monthly because CVEs don’t show up at a regular pace; things like “Microsoft Patch Tuesday” tend to make any finer cadence intractable. We use a segmented model because we are looking for a change point in the data. We select the best model using the Bayesian Information Criterion.
That is why this approach is useful, but why is it wrong? Some CNAs have a cadence that is not monthly but more like quarterly or somewhat random (looking at you Oracle). Because Poisson models use a log link and we are just modeling a growth in time, the log link means we are expecting the published CVEs to grow (or decline) exponentially—something unrealistic in the long run. Indeed, all logistics start looking like exponentials, and that is what we are looking at here. This also means these models cannot predict future numbers very well.
Methodological tangent addressed. You can go back to the main post to see what this means.
3Note that we take a look at the top 100 CNAs in this post and evaluate them one by one, unlike others who have selectively picked a set of CNAs, aggregated them, declared AI a spike.
4Note we are shifting to a cumulative rather than monthly view here.
