AUC or GTFO

In hacker culture, there’s a long-standing phrase: “POC || GTFO.” Short for Proof of Concept or Get the F** Out*, it’s the community’s way of saying: don’t just talk, prove it. If you claim to have an exploit but can’t show the code, the demo, or the actual work, your words don’t carry weight. Put up, or shut up.

Fast forward to today, and I can’t help but feel we need a similar mantra in vulnerability management. Every day, I see vendors making sweeping claims: this vulnerability is critical, that one isn’t worth your time, and so on. More often than not, the arguments are cherry-picked, framed with selective data, or spun in a “How to Lie with Statistics” kind of way. Instead of advancing the conversation, it muddies it.

I’m not here to call out specific examples. Instead, I want to propose a shift in how we approach these debates - one that could actually move the industry forward.

Prediction is Inevitable

At its core, vulnerability management is about prioritization. We fix one issue before another because we’re making an implicit prediction: this one matters more right now. Whether you rely on CVSS, threat intel, vendor feeds, or predictive models, you’re making a bet.

This post isn’t about arguing which methodology is “best.” My request is simpler: show your work.

Enter Area Under the Curve

In data science, we have several standard ways of evaluating how well a predictive model performs such as precision and recall or ROC curves. These metrics are often visualized with a line plot. Precision and recall metrics are especially useful in cases of class imbalance - where positives are rare compared to negatives. Sound familiar? In vulnerability management, only a tiny fraction of vulnerabilities ever get exploited.

These two metrics make the trade-offs crystal clear:

• ROC measures how well you can tell good from bad (or in the case of vulnerabilities the “urgent” from the “less urgent”

• Precision and Recall measures how often you’re right and how much of the bad stuff you found (we also call these efficiency and coverage respectively)

Once you see performance plotted out, you can decide what matters more for your organization: minimizing noise, or minimizing risk. More importantly, you can start improving the model.  With these metrics you can compare two models against each other too - a common way to that measurement is called the “Area Under the Curve” or simply just AUC.  Without performance metrics, you’re flying blind - you can’t know if your approach is good, bad, or just average.

The Proposal

So here’s my challenge: let’s stop relying on hand-wavy claims and selective statistics. If you’re going to argue that one approach to vulnerability management is better than another, back it up with performance metrics. Show how well it works, what the trade-offs are, and where it can be improved.

In other words:

AUC || GTFO.