OWASP 2025 Global AppSec EU

Current methods for evaluating the safety of Large Language Models (LLMs) risk creating a false sense of security. Organizations deploying generative AI often rely on automated “judges” to detect safety violations like jailbreak attacks, as scaling evaluations with human experts is impractical. These judges—typically built with LLMs—underpin key safety processes such as offline benchmarking and automated red-teaming, as well as online guardrails designed to minimize risks from attacks. However, this raises a crucial question of meta-evaluation: can we trust the evaluations provided by these evaluators?

In this talk, we examine how popular LLM-as-judge systems were initially evaluated—typically using narrow datasets, constrained attack scenarios, and limited human validation—and why these approaches can fall short. We highlight two critical challenges: (i) evaluations in the wild, where factors like prompt sensitivity and distribution shifts can affect performance, and (ii) adversarial attacks that target the judges themselves. Through practical examples, we demonstrate how minor changes in data or attack strategies that do not affect the underlying safety nature of the model outputs can significantly reduce a judge’s ability to assess jailbreak success.

Our aim is to underscore the need for rigorous threat modeling and clearer applicability domains for LLM-as-judge systems. Without these measures, low attack success rates may not reliably indicate robust safety, leaving deployed models vulnerable to unseen risks.