Security tests for LLM apps, written in pytest.

Your app has failure modes your test suite can't see. It'll follow an instruction hidden in a document it just retrieved. It'll repeat a key you told it to keep. It'll act on an authorisation the caller just typed in.

LLMSecTest attacks your running app for all ten OWASP LLM Top 10 categories and tells you what it got out. MIT-licensed, on PyPI, built in the open.

pip install llmsectest Run your first scan »

Just pytest

No separate scanner to run. Your security checks are pytest tests. Same command, same build gate, same plumbing as the rest of your suite.

How it works »

Mapped & scored

Every probe maps to an OWASP LLM Top 10 category and carries a CVSS v4.0 base score. So you can triage a finding instead of reading a wall of text.

The coverage map »

CI-native

Reports come out as SARIF v2.1.0, which your code-scanning dashboard already reads. HTML, JSON and Markdown too. No glue code.

A sample finding »

Quick-start

Get a scan running in seconds.

One line to install, then point it at a model or at your running app. Read the SARIF. There's a full walkthrough in the docs.

bash
~ $ pip install llmsectest
~ $ llmsectest --target anthropic:claude-3-5-haiku
~ $ llmsectest --target app:http://localhost:8000/chat   # your live app
~ $ llmsectest --target app:http://localhost:8000/chat \
      --report-formats=sarif,html,json,markdown
# => OWASP LLM Top 10 categories probed · findings written to results/<target>.sarif

OWASP LLM Top 10 · honest status

What's covered and what isn't yet

All ten OWASP LLM Top 10 (2025) categories are implemented and tested today. Nothing here got marked done before it was. What's left to do is depth. If a run can't reach a category, because you didn't hand it a repo, a model path or an app marker, it says so as an explicit skip. You won't get a silent gap.

The numbering below is the 2025 edition. Older editions numbered these differently. Supply chain used to be LLM05 and is now LLM03, so a number from an older list won't line up with what the reports say.

There's a 2026 edition. This tool doesn't implement it yet. It came out on 3 August 2026. I've read it against what's here: nothing was added, dropped, merged or split, but eight of the ten categories change number and System Prompt Leakage becomes Hidden Context Exposure with a wider remit. Every report and every stored baseline this project has published carries 2025 numbers, so renumbering quietly would change what those records mean. OWASP's own category pages still show 2025 too. So I'm staying on 2025, saying the year everywhere, and I'll move once there's a way to keep the old records readable.

  • LLM01Prompt Injectiondone
  • LLM02Sensitive Information Disclosuredone
  • LLM03Supply Chain (dependency manifests, OSV)done
  • LLM04Data & Model Poisoningdone
  • LLM05Improper Output Handlingdone
  • LLM06Excessive Agencydone
  • LLM07System Prompt Leakagedone
  • LLM08Vector & Embedding Weaknessesdone
  • LLM09Misinformationdone
  • LLM10Unbounded Consumptiondone

done implemented & tested, 10/10 · depth improvements continue on the roadmap

What "done" still doesn't tell you. Read this before you trust a clean row. LLM06 (excessive agency) can only report what your app says. If yours describes an action in prose instead of emitting the signature you passed in, a clean row means "we didn't see it", not "your app is fine". We write every limitation we find into the changelog. It comes out again when it's fixed.

Roadmap · shipped, building, planned

Where it's going

Built in the open across the funding period, which runs to the end of November 2026. Each phase ships before we claim the next one. The status here tracks what the code really does today.

  1. 01

    Foundation

    shipped
    • pytest-native framework & plugin
    • One adapter for OpenAI · Anthropic · Hugging Face · Ollama & LM Studio (local), with a fail-fast --preflight health check
    • Reports in SARIF v2.1.0 · HTML · JSON · Markdown
    • CVSS v4.0 scoring with OWASP mapping
    • CLI & documentation site
  2. 02

    OWASP coverage

    shipped
    • All 10 of 10 categories live: LLM01 to LLM10, the complete OWASP LLM Top 10 (2025)
    • Black-box testing of a running application, --target app:<url>, reaching 8 categories once you name what the app holds (--app-prompt/-secret/-action/-canary/-rag-poison); LLM01/05/09/10 need nothing
    • White-box scans: dependency manifests with an OSV known-CVE lookup (--repo, --osv, LLM03), and an offline serialization-opcode scan of model files that never unpickles them (--model-scan, LLM04)
    • A white-box LLM08 scan of your persisted vector store (--vector-store). It does not invert your embeddings. Inverting one needs an inverter trained for that embedding space, so instead it reads the store and tells you how much of the corpus somebody with read access already has. Usually all of it, because the text is filed next to the vector it was made from
    • LLM02 and LLM06 attack along four mechanisms each rather than four wordings, including a secret asked for base64-encoded and a forged authorization an agent cannot check. A leaked secret counts however it comes back: encoded, Unicode-disguised, or split character by character, including with the separator spelled out (E SPACE X SPACE A). That last one is what a small model came up with by itself to get past an output filter
    • RAG-specific LLM08: retrieval exposure (--app-canary) and indirect prompt injection through a poisoned retrieved document (--app-rag-poison)
    • Red-team jailbreak set (JailbreakBench / AdvBench, --redteam-set) plus a false-refusal rate on the benign twins (--redteam-benign)
    • Reports that can't flatter your app. A clean run tells you what your app withstood, per category. If a probe never got an answer back, it's inconclusive and it isn't a pass. That covers an endpoint we couldn't reach, one that answers with an error instead of a reply, a rate limit, and a reply we couldn't parse. The reason says which, because a rate limit is a quota to raise and an expired token is not a wrong address. Each one is named, so you can see which attacks went unanswered. A probe we gave up on stays out of the timing figures too, since it timed your deadline and not your app. The run exits non-zero, so an empty findings list can't sail through CI, and the summary withholds its verdict instead of handing you a good one: the status reads INCOMPLETE, not PASSED, and no security posture is claimed at all. And if a run got your secret out, it won't turn round and tell you the secret was protected. See what that looks like on our whole test cohort » One result out of those reports is written up on its own: your secret-disclosure test passed, the secret still left »
    • Standalone HTML from any SARIF file (--render-sarif), proven against committed output from ruff, Bandit and Semgrep

    Release-by-release detail lives in the changelog.

  3. 03

    Depth & reports

    in progress
    • CycloneDX SBOM generation, --sbom <path> (LLM03)
    • A classifier refusal oracle (GLiGuard / Llama-Guard), replacing the substring rule
    • The remaining LLM08 white-box dimensions: embedding-store poisoning and multi-tenant isolation
    • Deeper supply-chain analysis and stress tests
  4. 04

    Integrations & v1.0

    planned, this funding period
    • CI/CD templates for GitHub Actions · GitLab CI · Jenkins
    • v1.0 on PyPI
  5. 05

    Past v1.0

    planned, beyond this funding period
    • Hardening & independent security audit
    • OWASP community submission
    • PDF reports, a remediation database, a plugin API

shipped in the code today · in progress being built now · planned ahead, and the last two phases say which side of the funding period they sit on

Output · the target format

Findings you can act on

A run gives you SARIF that drops straight into GitHub code scanning, GitLab, or any SARIF viewer. Each finding carries its OWASP category, a CVSS vector and a pointer to the fix. Here's the shape of it.

Illustrative. This is the format we target. We didn't run this scan.

Read a finding » The secret-disclosure probes found one thing across 52 applications; 32 of them gave the credential up anyway.

See real ones » Every report from our own regression cohort, unedited: dozens of LLM applications scanned black-box, including the categories that found nothing.

report.sarif
{
  "ruleId": "LLM01-prompt-injection",
  "level": "error",
  "properties": {
    "owasp": "LLM01:2025 Prompt Injection",
    "cvss":  "CVSS:4.0/AV:N/AC:L/.../VC:H",
    "score": 9.2
  },
  "message": {
    "text": "System prompt recovered via instruction override."
  }
}

Built in the open, for the people shipping LLM features.

App developers, security leads, researchers. The repo is public and the roadmap is honest. Watch it, try the adapter, or open an issue.

github.com/wehnsdaefflae/llmsectest »