Agentic security | Levanto

Agentic security

Guardrails no longer have to choose between smart and fast.

Sage reads the whole conversation and decides in the time a classifier takes.

77% Harm prevented
95.5% Harmless allowed
208 ms Median latency
~5× Faster than GPT-5, similar quality

The trade-off

Fast and blind, or smart and slow.

Classifiers are fast, but blind

Most score only the last message, so harm built up through roleplay, personas or fake authority slips past.

LLM judges are smart, but slow

A frontier model catches that harm, at 1–1.5 seconds per call.

So teams pick a failure mode

Too strict blocks real users. Too loose lets harmful tasks finish.

LLM judgment at classifier speed.

LLM-grade judgment
Reads the whole conversation and judges intent, not keywords.

Classifier-grade latency
Fast enough for an agent loop or a checkout.

Generalized, not domain-trained
Your instructions are the policy. No per-attack training.

Safety–utility Pareto on AgentHarm

Harmless allowed (x) vs harm prevented (y). Top-right is best.

868890929496981005060708090% of harmless requests allowed → higher is better
% of harmful prevented → higher is better
← ideal

System	Harm Prevented	Harmless Allowed	Median Latency
Levanto Sage	77%	95.5%	208ms
OpenAI GPT-5	76.3%	96%	1009ms
Google Model Armor	89.7%	87.5%	195ms
OpenAI GPT-5-mini	56.1%	100%	1114ms
Meta PromptGuard	57.3%	95.5%	301ms
InjecGuard	55.8%	95.5%	262ms
Protect AI	54.7%	95.5%	312ms
AWS Bedrock LOW	58.8%	88.1%	191ms
AWS Bedrock MED	51.6%	96%	188ms
Unguarded GPT-4o	51.1%	100%	-

The hot path

Median latency

All 352 prompts, one at a time, from AWS us-east-2. Successful calls only.

System	Median Latency
AWS Bedrock MED	188ms
AWS Bedrock LOW	191ms
Google Model Armor	195ms
Levanto Sage	208ms
InjecGuard	262ms
Meta PromptGuard	301ms
Protect AI	312ms
OpenAI GPT-5	1009ms
OpenAI GPT-5-mini	1114ms

Why AgentHarm

Most jailbreak benchmarks are already saturated.

Most jailbreak benchmarks judge static strings, sit in classifier training sets, and never score what an agent actually does.

AgentHarm (Andriushchenko et al., ICLR 2025, arXiv:2410.09024), developed by the UK AI Security Institute and Gray Swan AI, instead gives a real tool-using agent multi-step malicious tasks and scores how much harmful work actually gets completed.

The unguarded baseline is the paper's original model, GPT-4o.

What this chart does and doesn't show

Thresholds are calibrated in-sample: ≤ 5% FPR on the same 176 benign prompts used for evaluation, not a held-out set (Bedrock and Model Armor use fixed confidence settings).
Conversations are live and non-deterministic. Harm score and over-block mix the gate decision with a tool-using agent and two GPT-4o judges, so results move run to run; expect stable ranking, not bit-identical numbers.
The open-weight classifiers (Protect AI, PromptGuard, InjecGuard) were measured on CPU on the same host, not a GPU-optimized deployment; their latency would likely drop on GPU.
GPT-5 and GPT-5-mini ran at reasoning_effort=minimal.

Drop-in guardrail

Score the transcript, branch on confidence.

import os, httpx

resp = httpx.post(
    "https://sage.levanto.ai/decide",
    headers={"Authorization": f"Bearer {os.environ['SAGE_API_KEY']}"},
    json={
        "content": transcript,
        "question": {"id": "needs_refusal", "kind": "yesno",
                     "instructions": "Is the user working the assistant "
                                     "toward an outcome it should refuse?"},
    },
)
result = resp.json()["result"]

if result["probability"] >= 0.716:
    refuse(transcript)                   # matches the AgentHarm operating point
elif result["answer"] is None:
    escalate_to_human(transcript)        # not sure? a human decides
else:
    run_agent(transcript)