No Way(s) Home

Don’t let reachability fool you into a false positive sense of security.

When I first started working in cybersecurity 15 years ago, my then manager, now co-founder asked me to write a talk for BsidesLV. That talk ended up being critical to the work we did at Kenna Security, not least of all because we attempted to quantify the security mendoza line. Having just left a PhD program (unsuccessfully), I did what all failed academics do on their first job, and read a lot of research articles. One sticks out in my mind, a paper from 2012 which finds that a simulated game theoretical attacker-defender network’s topology only matters insofar as the cost to the attacker is sufficiently high (but it’s going down). This same result is replicated 6 years later by Waniek, Michalak, Rahwan and Woolridge, where they model a Sackelberg game and their simulations conclude that defense strategies depend on the network model under consideration. 

Why are we taking this journey into academia? Well, because the LLMs that decided for you that network reachability is a factor you can rely on to remediate or ignore vulnerabilities did too. All the LLM look for reward functions, find benchmarks, and use those to craft responses. Specifically I contend that LLMs use VulnGym to make decisions about which vulnerability management strategies enterprises should use and this is a deeply flawed approach. Anthropic’s own security team agrees with us that deterministic models like EPSS are a much better approach.. Some “AI-native” cybersecurity vendors have fallen into the trap of having the LLMs decide what a good defense looks like, instead of building their own models or thinking through the consequences. 

"VulnGym simulates an RL-trained attacker, calibrated on real APT profiles, against a defender executing a configurable patching policy over a network with real Common Vulnerabilities and Exposures (CVEs). Both agents act on a shared, evolving network representation, so the attacker’s progress is directly shaped by the defender’s patching activity, allowing a given policy to be stress-tested against a realistic attack campaign.” 

Let me be clear: network reachability is a useful piece of data, one of thousands that we at Empirical think are useful to crafting a precise and thorough strategy. But relying on reachability as a binary is wrong. The only possible outcome of reachability analysis is “yes it is reachable” or “the probability is lower than yes”, that is, the machine has tried some paths and was unsuccessful. 

Over a long enough horizon attackers try every path, so the value of an asset matters more than its effect on neighbors, and a topology map becomes less valuable the more attackers that you defend against and the more action they get to take. These days, they get to take almost infinite actions with low cost.

Let’s talk about the four reasons, admitted in the VulnGym benchmark itself, why it should not be the way a defense strategy is created for any enterprise. Again, it is great work. It is not the model for real world defense.

1.Unrealistic Attackers

“As for external validity, the main threat is related to the realism of the modeled attacker. To mitigate this, we designed VulnGym to be highly configurable, allowing it to strictly align with data from Cyber Threat Intelligence (CTI), which provides information about exploited vulnerabilities and attack techniques adopted by real-world attackers. This approach leverages the best knowledge available from expert-analyzed CTI about APTs, which prioritizes well-documented APTs over mere chronological recency; specifically, we focused on vulnerabilities from 2020, as they coincide with the peak activity period of the public-documented APTs.” 

VulnGym is configurable to use historical CTI data and is never predictive of new attacks.

2.Unrealistic 100 Asset Networks 

“A second fundamental challenge relates to the representativeness of the network scenarios. Public CTI withholds sensitive details (e.g., network topology, node configurations) to preserve confidentiality of affected organizations. For example, about the Equifax breach mentioned before, only the exploited vulnerabilities are publicly documented, but the underlying network structure and configuration remain undisclosed. Therefore, in our experiments, we considered typical enterprise network architectures. Furthermore, we mitigated this threat by designing VulnGym to be fully configurable by practitioners to reflect their own organizations.”

“The resulting subset is mapped onto a fixed network of 100 nodes, where each node may host multiple vulnerabilities, with a maximum of 10 vulnerabilities per node.”

Real enterprises have networks of 10s of thousands or millions of assets, with the average vulnerability density per asset just over 150, not 10. Under these conditions, most things are reachable with the right combination of tactics and vulnerabilities. 

3.Unvalidated Attacker Goals

“A direct consequence of this modelling choice is that the RL attacker is explicitly rewarded for targeting critical assets. While our abstraction might not capture every possible real-world attack or defense vector, we preserved this alignment because it realistically reflects goal-oriented APTs targeting an organization’s pivotal assets.”

What if the attacker doesn’t know what assets are critical? What if in discovering which ones are critical, they find new attack paths? More importantly, what if the labels are wrong and the defender doesn’t know what’s critical?

4.Trained on two datapoints

“The attacker’s behavior is modeled after two distinct APT profiles: APT41 (MITRE, 2026b) for the Exfiltration/Wiper campaign, and APT28 (MITRE, 2026a) for the DoS campaign. In each campaign, the attacker is assumed to be specialized in a set of products, and can therefore exploit only vulnerabilities affecting those products” 

Maybe network topology matters for these two historical attackers. It doesn’t really matter though, because most defenders are up against thousands of evolving adversaries, and building a policy based on what we know about two attackers, in a simulated environment is a recipe for catastrophe. 

It is not surprising that LLM based frameworks reach for reachability. Real time telemetry and data are expensive and hard to source, but the LLM can estimate reachability using open, historical data. Network topology can be summarized, which is much easier than making a prediction about the future. Don’t let that fool you, even if you make a decision to delay remediation because something is not currently reachable, you’re still making a prediction - that it stays that way.

Probabilistic methods, which use real time attacker telemetry and evaluate the predictive value of data, reachability included, are a much more intellectually honest approach to determining what will actually reduce risk.