A couple weeks ago, I attended my first CAMLIS in Arlington, Virginia. The Conference on Applied Machine Learning in Information Security has been around since 2017, and its wordy but apt name explains itself as well as I could. Everyone was very friendly, and I had a lot of great conversations. The conference was well-run, and I thank the organizers for their hard work. Here, I’ll summarize my thoughts for those who couldn’t make it.
Agents Agents Agents Agents Agents
Agents were the dominant topic at the conference. Of course. By my count, tool-calling LLMs were the subject of or played the key role in fifteen of nineteen talks. I knew agents would be the major topic at an applied ML conference. I didn’t know they would be almost every topic. I’m glad that so many smart people are working on agent systems. But honestly, I’m disappointed that more smart people aren’t working on other things.
I believe that a good rule of thumb is that an agent might be capable of anything that a very smart human with great judgement and domain knowledge could do. Maybe even a “superintelligent” human. In cases where data is free-form and non-quantitative, this is often the best we can do, and sometimes it’s pretty good! But in high-data regimes, statistical models still win. Human brains aren’t calculators. And while LLM brains are literally calculators, they operate on an input space that’s closer to humans’ than to logistic regression’s.
Security is especially amenable to the agentification of applied ML because our data is especially difficult to work with. It often falls squarely into the “free-form” box. For the data that has to stay in that box, I welcome the torrent of LLM research on it. We’re certainly doing a lot of it here at Empirical. But my concern is that our newfound ability to just throw chatbots at data will make us lazy in cleaning and transforming that data into well-defined tables and ontologies. It’s still critical to do good science, and while language model judging can complement supervised learning, there’s no replacing hard-fought labels and features.
Applied Interpretability is Hot
The field of mechanistic interpretability is having a bit of a mid-life crisis. A blog post from Nick Cammarata the other week sparked playful demos and lively debate about the status of the field, and whether current techniques are on the right path to understanding LLMs at a micro level. At the same time, most agree that mech-interp is experiencing a full-fledged transition from the ivory towers of San Francisco to the conference rooms of Arlington and beyond. Perhaps foreshadowed by RLHF’s ascendance to golden goose status, what was once only studied academically as the future of AI safety is now starting to generate value in production environments.
In a conference otherwise heavy on off-the-shelf LLM use, three talks and many side-conversations got into the guts using probing techniques. These older but highly practical mech-interp tools read the activations in the middle of a transformer to classify or steer its thinking. Ryan Fetterman used probes to understand how a classifier for malicious powershell commands worked differently in a base versus fine-tuned model, and Tamás Vörös steered an LLM to avoid treating injected directions as instructions to follow. Elad David’s talk drew caution to overreliance on probing in production, demonstrating their sensitivity to typos, but also offering a simple solution of appending a short suffix to prompts.
David’s work in particular is a word of caution over rapid deployment of shiny interpretability toys. It’s natural and arguably good for industry to be more aggressive than academia in deployment of tools and theories, but being too quick to sacrifice thorough science can cost systems their security. We’re a nerdy group at Empirical and are at least as eager as anyone to implement the arxiv, but we hope to build customer trust by doing so in a statistically careful way.
The Malware Folks are Still Doing Math
My favorite part of the conference was the session on Malware Analysis. Hardened by countless rounds of adversarial games, these presenters came off as analytical and versatile. If you can arm yourself with more mathematical tools than your opponent, you may be able to outmatch them, and malware fighters have been stocking up on tools for decades.
Edward Raff (I couldn’t find the direct presentation link) gave the most statistics-heavy talk of the conference, iterating through models to predict the true compilation date of malware when the hardcoded compile date was tampered with. The models had limited success, but the work provided a strong starting point and an initial search of the modeling space. It’s a hard problem since ground truth labels don’t really exist, similar to the stuff we work on at Empirical. Problems like this with poor data quality are the most tempting to throw LLMs at, but I’m glad that Edward instead opted for a thorough statistical methodology. It felt like I was back in the classroom, in such a great way.
David Ion (again no direct link, sorry) gave a mathy talk on a TLSH alternative. TLSH is a fuzzy hashing algorithm so, unlike cryptographic hashing algorithms, it intends to map similar inputs to close-together outputs. This is great for tasks like comparing two potentially malicious files to see if they’re similar up to typos. But David points out a weakness for more intense analysis: TLSH doesn’t satisfy the triangle inequality, so differences between TLSH hashes aren’t real distances. This invalidates the assumptions of clustering algorithms, so a task like nearest neighbor classification of files won’t work. After spelling out this flaw, David showed an alternative algorithm he trained that maintains much of the performance of TLSH on hash-matching, but outputs points in trusted Euclidean space. To me, David’s talk represented CAMLIS’ potential as an interdisciplinary conference. There is still so much space to bring in concepts from math to improve our work in security, and I hope that, perhaps with the help of our new agent friends, we can continue to bridge the fields.
