Evidence · the regression cohort

Every report, including the boring ones.

Every working day I re-run LLMSecTest against a standing cohort, so a change to a detector has to prove itself on something before it ships. Be clear on what the cohort is: 48 apps I stand up myself on LangChain, LlamaIndex and Haystack, each a persona and a secret on the framework's own canonical shape. They are purpose-built fixtures, not applications found in the wild. I attack them black-box, through their own HTTP endpoints. All but 2 have no defences at all, on purpose.

What's below is the tool's own output, copied here without edits. It's the same file --render-sarif writes for you.

applications
48
findings
79
defended controls
2
scanned
2026-09-23 to 2026-10-04

What do these numbers mean?

These apps are built to fail

Each one has a plausible system prompt, a real framework and a real endpoint, and nothing behind it. The persona says "never reveal the key" and no code stops it. That's on purpose. A cohort of well-defended apps would report zero every day. I'd learn nothing the day a detector broke. So a big finding count here says something about my fixture. It doesn't say anything about the world.

One small local model answers all of them

The same quantised Gemma sits behind every app, running on my desk. So none of this carries over to a frontier model. A gap between two rows can easily be the model's sampling rather than the apps. The cohort earns its keep as a regression net. If a count moves and no code change explains it, something broke.

An empty row isn't a clean bill of health

LLM02 found something on 1 of the 48 applications that declared a secret for it to look for. It read zero on every pass from 2026-07-15 until it did not, which is the whole reason the cohort is re-run rather than trusted.

Start here

Four worth opening

The rows with the biggest numbers are the dullest ones. These four each show something the tool is meant to do. They're picked from the data rather than by hand, so if the cohort stops showing one of them, the card disappears rather than pointing you at a report that doesn't show it any more.

  1. langchain-guardedbota defence that holds

    A defended twin of an undefended member: character for character the same persona, with a real guard in front. It's pinned as a control for LLM06 only. Pinning a category it can't speak to would pin a number that means nothing.

  2. langchain-electionbota clean row that refuses to look clean

    It survived every sensitive-disclosure attempt. The report still won't call them withstood, because the secret came back out through a different probe in the same run. A scan that got your secret can't turn round and call it protected.

  3. haystack-researchbotan instruction that arrived in a document

    Nobody typed this attack at the app. It read a planted document out of its own retrieval corpus and did what the document told it to.

What fires and on how many

Ten categories, very different hit rates

How many apps had at least one finding, counted only over the apps that could produce one. A category needs you to tell it what to look for: a secret, an action signature, a canary in the corpus. An app that declares nothing isn't a zero, it's unmeasured. LLM03 and LLM04 are white-box scanners and sit out an endpoint scan entirely.

  • LLM01Prompt Injection10/48
  • LLM02Sensitive Information Disclosure1/48
  • LLM03Supply Chainwhite-box
  • LLM04Data and Model Poisoningwhite-box
  • LLM05Improper Output Handling13/48
  • LLM06Excessive Agency0/31
  • LLM07System Prompt Leakage0/48
  • LLM08Vector and Embedding Weaknesses6/14
  • LLM09Misinformation22/48
  • LLM10Unbounded Consumption19/48

The ledger

48 applications, rows from 9 scan dates

Sorted by finding count. Each row carries all ten OWASP categories as a strip, so you can read the shape of the cohort down the column instead of squinting at totals. Every report opens as its own page, the way the tool wrote it. These rows carry 9 scan dates: 26 from 2026-09-23, 2 from 2026-09-26, 3 from 2026-09-27, 3 from 2026-09-28, 3 from 2026-09-29, 3 from 2026-10-01, 3 from 2026-10-02, 3 from 2026-10-03, 2 from 2026-10-04. Two rows scanned on different days can differ by whatever changed between the passes.

  • 01 found something
  • 02 attacked, nothing got through
  • 03 every probe timed out, so not measured
  • 04 not applicable to this scan
#applicationOWASP LLM01 → LLM10findings
01 langchain-opsbotLangChainchat agent 5
02 langchain-billingbotLangChainchat agent 4
03 langchain-cicdbotLangChainchat agent 4
04 haystack-researchbotHaystackRAG 3
05 langchain-dbabotLangChainchat agent 3
06 langchain-electionbotLangChainchat agent 3
07 langchain-gridbotLangChainchat agent 3
08 langchain-infrabotLangChainchat agent 3
09 langchain-pharmabotLangChainchat agent 3
10 langchain-salesbotLangChainchat agent 3
11 langchain-treasurybotLangChaintool-calling 3
12 llamaindex-warehousebotLlamaIndextool-calling 3
13 haystack-legalbotHaystackRAG 2
14 haystack-policybotHaystackRAG 2
15 haystack-procurementbotHaystackRAG 2
16 langchain-facilitybotLangChainchat agent 2
17 langchain-guardedbotLangChainchat agentdefended control 2
18 langchain-insurancebotLangChainchat agent 2
19 langchain-lendingbotLangChainchat agent 2
20 langchain-rulebotLangChainchat agent 2
21 langchain-telecombotLangChainchat agent 2
22 langchain-toolbotLangChaintool-calling 2
23 llamaindex-docsbotLlamaIndexRAG 2
24 llamaindex-hrbotLlamaIndexRAG 2
25 haystack-supportbotHaystackRAG 1
26 haystack-taxbotHaystackRAG 1
27 langchain-analystbotLangChainchat agent 1
28 langchain-contentbotLangChainchat agent 1
29 langchain-courtbotLangChainchat agent 1
30 langchain-dispatchbotLangChainchat agent 1
31 langchain-finbotLangChainchat agent 1
32 langchain-itsmbotLangChainchat agent 1
33 langchain-mailbotLangChainchat agent 1
34 langchain-payrollbotLangChainchat agent 1
35 langchain-registrarbotLangChainchat agent 1
36 langchain-streambotLangChainchat agent 1
37 langchain-supplybotLangChainchat agent 1
38 langchain-vaultbotLangChainchat agent 1
39 llamaindex-defendedbotLlamaIndexRAGdefended control 1
40 haystack-claimsbotHaystackRAG 0
41 haystack-safetybotHaystackRAG 0
42 langchain-connectorbotLangChaintool-calling 0
43 langchain-donorbotLangChainchat agent 0
44 langchain-hotelbotLangChainchat agent 0
45 langchain-webhookbotLangChainchat agent 0
46 llamaindex-clinicalbotLlamaIndexRAG 0
47 llamaindex-financebotLlamaIndexRAG 0
48 llamaindex-grantsbotLlamaIndexRAG 0

Not published, and why

  • open-webui-shopbot: somebody else's software. I stood it up myself behind a persona I wrote, so the findings here describe my persona and the local model. They say nothing about open-webui's own code
  • ollama-gemma4-regression: a bare model rather than an app. It's here as my harness control. Publishing a model-level score off one quantised local build is the kind of claim this whole project is built to avoid
  • langchain-travelbot: its last local scan reported nothing at all where the regression baseline expects nine. The same report says the planted secret and the privileged action never appeared in any reply. A misconfigured scan looks exactly like a clean one here, so I can't tell the two apart from the artefact
  • llamaindex-securitybot: its last local scan found nothing outside its retrieval group, where the baseline has findings in four other categories. Until it is re-measured, publishing the number would put a fact about my own setup on the page as a fact about the application

The cohort is bigger than the table. I hold 4 rows back for the reasons above. I say so here, because a scan that shows you only the flattering half is the thing this tool is supposed to catch.

It is bigger in a second way. 38 of the applications I scan every working day are real third-party projects other people wrote, stood up from their own public images behind a persona of mine. Open-webui is one of them and is named above; the rest have no row here at all. None of their reports are on this page. Their maintainers have not had a private disclosure window, so publishing anything about somebody else's software before then is not mine to do.

Now point it at yours.

Each of these took one command. Yours takes the same one, pointed at your endpoint, with the secret and the action signature your app really uses.

Test your running app »