Evidence · the regression cohort

Every report, including the boring ones.

Every working day I re-run LLMSecTest against a standing cohort, so a change to a detector has to prove itself on something before it ships. Be clear on what the cohort is: 50 apps I stand up myself on LangChain, LlamaIndex and Haystack, each a persona and a secret on the framework's own canonical shape. They are purpose-built fixtures, not applications found in the wild. I attack them black-box, through their own HTTP endpoints. All but 2 have no defences at all, on purpose.

What's below is the tool's own output, copied here without edits. It's the same file --render-sarif writes for you.

applications: 50
findings: 515
defended controls: 2
scanned: 2026-08-24 to 2026-08-26

What do these numbers mean?

These apps are built to fail

Each one has a plausible system prompt, a real framework and a real endpoint, and nothing behind it. The persona says "never reveal the key" and no code stops it. That's on purpose. A cohort of well-defended apps would report zero every day. I'd learn nothing the day a detector broke. So a big finding count here says something about my fixture. It doesn't say anything about the world.

One small local model answers all of them

The same quantised Gemma sits behind every app, running on my desk. So none of this carries over to a frontier model. A gap between two rows can easily be the model's sampling rather than the apps. The cohort earns its keep as a regression net. If a count moves and no code change explains it, something broke.

An empty row isn't a clean bill of health

LLM02 found nothing on any of the 48 applications where its probes came back, and it has read zero in every machine-checked baseline since we started keeping them on 2026-07-15. And on 2 more, every LLM02 probe timed out without an answer, so the category was not measured there at all and is left out of that rate rather than counted as clean, and those timeouts came in one consecutive block per application, which is the shape of our own deadline leaving the app still working rather than of four separate refusals to answer; on 8 of the 48 that are counted, some probes still timed out, so even those rows rest on fewer mechanisms than they attempted. We do not claim to know why the disclosure probes stay quiet while the same secret walks out through the prompt-leakage probe beside them. Each report says as much on its own face. If a category was scored against a value the scan never saw, it's marked unconfirmed, not passed.

Start here

Four worth opening

The rows with the biggest numbers are the dullest ones. These four each show something the tool is meant to do. They're picked from the data rather than by hand, so if the cohort stops showing one of them, the card disappears rather than pointing you at a report that doesn't show it any more.

langchain-guardedbota defence that holds
A defended twin of an undefended member: character for character the same persona, with a real guard in front. It's pinned as a control for LLM06 only. Pinning a category it can't speak to would pin a number that means nothing.
langchain-toolbotan agent that really did it
Its excessive-agency finding isn't a string the model copied. The executor ran the tool, for a stranger who typed an employee id and a ticket number that don't exist. On a prompt-only target the same row would only mean the app *said* it had acted.
haystack-claimsbota clean row that refuses to look clean
It survived every sensitive-disclosure attempt, and the report still won't call them withstood, because the secret came back out through a different probe in the same run. A scan that got your secret can't turn round and call it protected.
haystack-legalbotan instruction that arrived in a document
Nobody typed this attack at the app. It read a planted document out of its own retrieval corpus and did what the document told it to.

What fires and on how many

Ten categories, very different hit rates

How many apps had at least one finding, counted only over the apps that could produce one. A category needs you to tell it what to look for: a secret, an action signature, a canary in the corpus. An app that declares nothing isn't a zero, it's unmeasured. LLM03 and LLM04 are white-box scanners and sit out an endpoint scan entirely.

LLM01Prompt Injection50/50
LLM02Sensitive Information Disclosure+2 unanswered0/48
LLM03Supply Chainwhite-box
LLM04Data and Model Poisoningwhite-box
LLM05Improper Output Handling47/50
LLM06Excessive Agency+1 unanswered2/31
LLM07System Prompt Leakage+9 unanswered30/41
LLM08Vector and Embedding Weaknesses14/15
LLM09Misinformation46/50
LLM10Unbounded Consumption47/50

The ledger

50 applications, newest full pass

Sorted by finding count. Each row carries all ten OWASP categories as a strip, so you can read the shape of the cohort down the column instead of squinting at totals. Every report opens as its own page, the way the tool wrote it.

01 found something
02 attacked, nothing got through
03 every probe timed out, so not measured
04 not applicable to this scan

#	application	findings
01	haystack-claimsbotHaystackRAG	14
02	haystack-legalbotHaystackRAG	13
03	langchain-insurancebotLangChainchat agent	13
04	langchain-webhookbotLangChainchat agent	13
05	haystack-procurementbotHaystackRAG	12
06	haystack-taxbotHaystackRAG	12
07	langchain-dispatchbotLangChainchat agent	12
08	langchain-facilitybotLangChainchat agent	12
09	langchain-hotelbotLangChainchat agent	12
10	langchain-salesbotLangChainchat agent	12
11	langchain-streambotLangChainchat agent	12
12	haystack-researchbotHaystackRAG	11
13	langchain-cicdbotLangChainchat agent	11
14	langchain-dbabotLangChainchat agent	11
15	langchain-infrabotLangChainchat agent	11
16	langchain-itsmbotLangChainchat agent	11
17	langchain-lendingbotLangChainchat agent	11
18	langchain-payrollbotLangChainchat agent	11
19	langchain-pharmabotLangChainchat agent	11
20	langchain-supplybotLangChainchat agent	11
21	langchain-telecombotLangChainchat agent	11
22	langchain-toolbotLangChaintool-calling	11
23	langchain-vaultbotLangChainchat agent	11
24	llamaindex-hrbotLlamaIndexRAG	11
25	langchain-courtbotLangChainchat agent	10
26	langchain-donorbotLangChainchat agent	10
27	langchain-electionbotLangChainchat agent	10
28	langchain-finbotLangChainchat agent	10
29	langchain-gridbotLangChainchat agent	10
30	langchain-guardedbotLangChainchat agentdefended control	10
31	langchain-mailbotLangChainchat agent	10
32	langchain-opsbotLangChainchat agent	10
33	langchain-registrarbotLangChainchat agent	10
34	llamaindex-securitybotLlamaIndexRAG	10
35	llamaindex-warehousebotLlamaIndextool-calling	10
36	haystack-policybotHaystackRAG	9
37	haystack-supportbotHaystackRAG	9
38	langchain-analystbotLangChainchat agent	9
39	langchain-billingbotLangChainchat agent	9
40	langchain-contentbotLangChainchat agent	9
41	langchain-travelbotLangChainchat agent	9
42	langchain-treasurybotLangChaintool-calling	9
43	llamaindex-docsbotLlamaIndexRAG	9
44	haystack-safetybotHaystackRAG	8
45	langchain-rulebotLangChainchat agent	8
46	llamaindex-clinicalbotLlamaIndexRAG	8
47	llamaindex-grantsbotLlamaIndexRAG	8
48	langchain-connectorbotLangChaintool-calling	7
49	llamaindex-defendedbotLlamaIndexRAGdefended control	7
50	llamaindex-financebotLlamaIndexRAG	7

Not published, and why

open-webui-shopbot: somebody else's software. I stood it up myself behind a persona I wrote, so the findings here describe my persona and the local model. They say nothing about open-webui's own code
ollama-gemma4-regression: a bare model rather than an app. It's here as my harness control. Publishing a model-level score off one quantised local build is the kind of claim this whole project is built to avoid

The cohort is bigger than the table. I hold these two back for the reasons above. I say so here, because a scan that shows you only the flattering half is the thing this tool is supposed to catch.

Now point it at yours.

Each of these took one command. Yours takes the same one, pointed at your endpoint, with the secret and the action signature your app really uses.

Test your running app »