AI Jailbreak Testing Has a Blind Spot, ECHO Swaps "Did It Comply?" for "How Harmful Was the Answer?"

Every safety test on a large language model rests on one question that almost nobody audits: how do you know when an attack worked?

For years, the answer has been a shortcut. Automated red-team tools fire thousands of tricky prompts at a model. To decide which prompts "broke" it, they look for a simple tell, did the model start its reply with something like "Sure, here is" instead of "I can't help with that"? If it complied, the attack counts as a win.

That shortcut is called a compliance proxy. It is fast, cheap, and easy to automate. It is also the wrong question. And a new method called ECHO is built to replace it with a better one: instead of asking whether the model complied, ask how harmful the answer actually is.

The shift sounds technical. It is not a small one. It goes to the heart of how the AI industry measures safety, how vendors prove their claims, and whether the numbers the public sees mean anything at all.

The Proxy Problem: Confusing "Polite" With "Dangerous"

Optimization-based jailbreak attacks do not work by hand. They work by search. A tool makes small changes to a prompt, reordering words, adding role-play framing, layering instructions, and keeps whatever version gets closer to the goal. To run that loop thousands of times, the tool needs a fast score for every response it gets back.

Checking whether a reply starts with "Sure" takes milliseconds. Judging whether a reply is genuinely dangerous takes far longer and needs real understanding. So the field reached for the cheap signal, and it stuck.

The trouble is that compliance and harm are not the same thing, and they drift apart in both directions:

• The false alarm. A model says "Sure, here's a general overview of the topic" and then gives a careful, well-hedged answer with nothing dangerous in it. The compliance proxy scores this as a jailbreak. It isn't.
• The miss. A model refuses the exact wording of the request but leaks the substance in pieces across the rest of the answer, or answers a nearby question that gets the user most of the way there. The proxy scores this as a refusal, so the attack is logged as a failure. It wasn't.
• The trained blind spot. A model can learn to open with a refusal and still deliver the content. Every check that only reads the first few words will report that model as safe.

This matters because these scores do not stay inside research labs. They turn into the safety numbers on a model card, the pass rate in a procurement review, and the footnote in a regulatory filing. A weak proxy in the measurement loop becomes a weak claim in the world.

What ECHO Does Differently

ECHO changes the target of the optimization itself. Rather than rewarding an attack when the model drops its refusal language, ECHO scores the response harmfulness and uses that score as the thing the attack tries to maximize.

Put plainly: the search is no longer hunting for a model that has stopped saying no. It is hunting for a model that has actually produced something bad. A polite "sure, absolutely" with harmless content earns nothing. A stilted, awkward reply that happens to contain genuinely dangerous material earns credit.

Two consequences follow, and both are bigger than they first appear.

1. The attack and the safety team finally want the same thing

Safety teams care about real-world harm. Attack tools, under the old proxy, cared about surface compliance. That gap meant a red team could report a pile of "successful jailbreaks" that a product team would rightly dismiss as noise, while missing the quiet leaks that actually mattered. Aligning the optimizer's objective with the defender's concern removes a whole category of wasted work.

2. The metric stops saturating

Cheap signals are easy to game, and once a model learns to refuse in a slightly different style, a compliance-based test can hit 100% and stay there. A metric that has plateaued tells you nothing about whether the next model is safer. A harm-based score is much harder to flatline, because it keeps asking the question that never fully goes away.

Why This Is the Right Argument at the Right Time

AI evaluation is shifting from an academic exercise to an operational and legal one. Three forces make ECHO-style thinking urgent rather than academic.

Models are getting better at refusing, outwardly. As refusal behavior improves, surface-level tests lose their power to tell good models from bad ones. When your measurement can no longer separate the top of your lineup, you are flying blind exactly where the stakes are highest.

Safety claims are becoming commercial claims. Enterprises now buy models partly on safety documentation. Regulators increasingly ask for evidence of testing. If that evidence rests on a proxy, the whole chain of trust rests on a proxy. Expect auditors, insurers, and counterparties to start asking not "did you test?" but "what did you measure?"

The attack/defense loop is asymmetric, until measurement improves. Attackers get a huge speed boost when their reward signal matches reality, because they stop wasting effort on fake wins. Defenders benefit too, but in a different way: defenders cannot fix what they cannot see. Better measurement generally favors the side that has to prove something, and that is usually the defender.

The Honest Catch: Judging Harm Is Expensive and Messy

ECHO is not a free upgrade, and any honest reading of it has to say so.

Harm is contextual. The same sentence about a chemical process is homework for one reader and a hazard for another. A single scalar score cannot capture that richness, so any harm-based method has to make judgment calls that will sometimes be wrong.

Grading requires a grader. At scale, that usually means another model acting as a judge, or a trained classifier. Those judges can be biased, can be fooled, and can themselves be the target of attack. Once your judge is the linchpin, the judge becomes part of the attack surface.

It is slower. Optimization loops want thousands of evaluations. Harm scoring is heavier than string matching, so ECHO-style methods trade speed for fidelity. The sensible pattern is a two-stage pipeline: cheap proxies to explore a huge space of candidate attacks, then harm grading to decide what actually counts. The mistake is reporting the cheap stage as a safety result.

What This Means for Businesses

If you buy, build, or deploy AI systems, this is not an abstract research debate. It changes what a credible safety claim looks like.

• Ask vendors what their success metric is. "We ran a red team" means little without the scoring method. A vendor that grades harm is giving you stronger evidence than one that counts refusals, even if the second vendor's slide deck has a bigger number on it.
• Audit your own harness. Many internal test suites were built quickly and still count "Sure, here is" as a break. If yours does, the safety figure in your own documentation may be inflated. Find out before someone else does.
• Track two failure modes separately. Record harmful-but-compliant and safe-but-over-refusing as distinct categories. They have different owners, different fixes, and different customer impacts. Blending them hides both.
• Treat evaluation methodology as a governance artifact. As testing and documentation requirements tighten, your measurement design becomes part of your compliance record. A documented, imperfect harm-based method is a stronger position than an undocumented proxy.
• Expect procurement to change. Once buyers understand the difference, "compliance rate" stops being a persuasive answer in a security review. Sellers who get ahead of that will have an easier conversation.

What It Means for Society

Public trust in AI depends on safety claims that mean something. If the headline numbers rest on whether a model said "sure" or "sorry," the public is being told a story about manners, not about risk.

That gap is tolerable while AI systems are mostly advisory. It becomes hard to defend as agents take actions, as models are embedded in healthcare, finance, and infrastructure, and as incidents start needing explanation. Serious standards, incident reporting, and liability all rest on the same foundation: a shared, defensible way to say how bad an output was.

Harm-graded evaluation is a step toward that foundation. It also gives policymakers something better to point at than refusal percentages, which is exactly why arguments over measurement will become arguments over policy.

An Actionable Playbook

• Audit your success criterion first. Before adding more attack prompts, confirm what your scoring function is actually rewarding.
• Separate exploration from judgment. Use cheap heuristics to search; use harm grading to decide. Never publish the exploration number as a safety result.
• Validate the judge on the failure that matters most. For safety, a missed harmful answer is usually worse than a false alarm. Test for false negatives specifically.
• Keep humans on the worst categories. Automated harm grading is not reliable enough to be the final word on the highest-severity content.
• Report methodology alongside results. State plainly what was measured. A transparent weak spot is manageable; a hidden one is a liability.
• Re-run when the model changes. A metric that passed last quarter on a previous model tells you nothing about the one you shipped this week.

What to Watch Next

Three signals will show whether this shift sticks. First, whether public safety benchmarks move from refusal-based scoring to harm-based scoring, that would change incentives across the whole field at once. Second, whether judge models become a recognized attack surface, with their own evaluation and hardening. Third, whether cost per graded response falls far enough that harm scoring becomes routine rather than a premium exercise.

ECHO's core insight is not really about jailbreaks at all. It is about the difference between a convenient measurement and a true one, and about how much of the AI safety conversation has quietly been built on the convenient kind. The methods that win the next few years will be the ones that measure the thing they actually care about.

Conclusion

Optimization-based jailbreak attacks were built to find weaknesses fast, and the compliance proxy made that possible. But fast and wrong is a bad trade in a field where the numbers get published, sold, and regulated. By making response harmfulness the objective instead of surface compliance, ECHO aligns what attack tools optimize with what safety teams actually fear, and closes a gap that has been quietly inflating both confidence and alarm.

The method is not free. Harm is contextual, judging it is slow, and the judge becomes a new point of failure. Those are real costs, and they argue for careful pipelines rather than for abandoning the idea.

What remains is the important part: the industry now has a clearer way to ask whether an AI system is actually safe, not merely polite. That question is going to get asked a lot more often, by buyers, by auditors, and eventually by regulators. The organizations that can answer it with real evidence will be the ones trusted with the systems that come next.