COVERAGE v1.5
What we check
22 AI services. Read-only GETs. Same stack → same grade.
Fingerprint detections 17 services
Additive GET fingerprints. Primary checkers own the grade when they fire first. Data derived from Zhuque Lab / Tencent’s AI-Infra-Guard fingerprint set (Apache-2.0; see NOTICE.md). No write, exploit, or LLM-keyed rules.
Ollama 1 detection
MEDIUM Ollama fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
vLLM 1 detection
MEDIUM vLLM fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
n8n 1 detection
MEDIUM n8n fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
Jupyter 3 detections
MEDIUM Jupyter fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
MEDIUM Jupyter fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
MEDIUM Jupyter fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
MLflow 1 detection
MEDIUM MLflow fingerprinted via unauthenticated GET
GET fingerprint — the scanner's detection for this service.
LiteLLM 1 detection
MEDIUM LiteLLM fingerprinted via unauthenticated GET
GET fingerprint — the scanner's detection for this service.
Ray 1 detection
MEDIUM Ray fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
Kubeflow 1 detection
MEDIUM Kubeflow fingerprinted via unauthenticated GET
GET fingerprint — the scanner's detection for this service.
Dify 1 detection
MEDIUM Dify fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
Flowise 1 detection
MEDIUM Flowise fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
Open WebUI 1 detection
MEDIUM Open WebUI fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
MCP 1 detection
MEDIUM MCP fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
Gradio 1 detection
MEDIUM Gradio fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
LM Studio 1 detection
MEDIUM LM Studio fingerprinted via unauthenticated GET
GET fingerprint — the scanner's detection for this service.
Qdrant 1 detection
MEDIUM Qdrant fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
Chroma 1 detection
MEDIUM Chroma fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
Weaviate 1 detection
MEDIUM Weaviate fingerprinted via unauthenticated GET
Additive GET fingerprint. Skipped when the primary checker already fired.
How a stack is scored
| dimension | start | critical fires | watch fires | floor |
|---|---|---|---|---|
| coherence | 5 | −2 | −1 | 1 |
| security | 5 | −2 | −1 | 1 |
| governance | 5 | −2 | −1 | 1 |
| proportionality | 5 | −2 | −1 | 1 |
| cost | 5 | −2 | −1 | 1 |
letter = average · A 4.5+ · B 3.8+ · C 3.0+ · D 2.0+ · E below 2.0 · F any unauthenticated write/exec block
two or more criticals caps the letter at D.
a rule marked ok is praise, not a finding.
Does it hang together? coherence · 5
C1 Nothing can think — no model server or model API in this stack. see it →
Do this: Add Ollama (self-hosted) or a cloud model API (OpenAI, Anthropic, Bedrock).
NIST AI RMF: Map
C2 The model can answer, but no human has a way to talk to it. see it →
Do this: Add a chat interface (Open WebUI, AnythingLLM) or a notebook.
NIST AI RMF: Map
C3 A vector database with no documents is an expensive empty room. see it →
Do this: Add your documents — retrieval exists to serve them.
NIST AI RMF: Map
C4 Documents are there, but nothing can find the relevant page at question time. see it →
Do this: Add a vector database between your files and the model.
NIST AI RMF: Map
C5 An orchestrator with nothing to orchestrate — no tools, no model. see it →
Do this: Give it a model to call or tools (MCP) to use.
NIST AI RMF: Map
Could a stranger use it? security · 5
S1 Blocks answering with no authentication — anyone who can reach them can use them, right now. see it →
Do this: Put an Auth proxy / SSO in front — keep the capability, add the gate. Each block's fix card has the lock-down steps.
OWASP LLM10 · NIST AI RMF: Manage · EU AI Act Art. 15
S2 A demo UI in the stack — the classic 'temporary prototype' that quietly becomes production. see it →
Do this: Decide now: is this a demo or a product? Products get auth and a proper interface.
NIST AI RMF: Manage
S3 A notebook server is a terminal on your machine with a friendlier face. see it →
Do this: Never expose it; reach it through a tunnel or VPN instead.
NIST AI RMF: Manage · EU AI Act Art. 15
S4 Three or more blocks a stranger can use with no login at all — that's not a stack, it's a buffet. see it →
Do this: Lock down the fix-card steps per block, or cut what you don't need.
OWASP LLM10 · NIST AI RMF: Manage · EU AI Act Art. 15
B1 Exposed blocks sit behind a boundary — the right instinct: keep the capability, add the gate. see it →
Do this: In reality coverage is per-service: verify every block is actually behind the boundary, then run the scanner to prove it.
NIST AI RMF: Manage
Where does the data go? governance · 3
G1 A third party is in the stack — its risk is contractual, not network. see it →
Do this: Ask for each cloud block: DPA signed? retention settings? region? may they train on your data?
OWASP LLM02 · NIST AI RMF: Govern · EU AI Act Art. 10
G2 Your documents leave the building — they cross to a third party on every request. see it →
Do this: Sometimes that's the right call; make sure it's an allowed one. Self-hosted retrieval keeps them home.
OWASP LLM02 · NIST AI RMF: Govern · EU AI Act Art. 10
G3 Continuous third-party logging — every prompt lands on someone else's platform, indefinitely. see it →
Do this: Review what may be traced and set retention consciously.
OWASP LLM02 · NIST AI RMF: Govern · EU AI Act Art. 12
Is it the right size? proportionality · 6
P1 Two vector databases doing one job — that's a coin flip, not a decision. see it →
Do this: Pick one. Chroma until you have a reason; Qdrant when you do; Pinecone if running one is the problem.
NIST AI RMF: Manage
P2 Two self-hosted model servers — one is enough until throughput forces the question. see it →
Do this: Start with Ollama; add vLLM when many users hit it at once.
NIST AI RMF: Manage
P3 vLLM at this scale is gold-plating — it exists for many concurrent users. see it →
Do this: Ollama fits a small stack; switch when throughput actually hurts.
NIST AI RMF: Manage
P4 An agent framework for what looks like a single-step task. see it →
Do this: If one prompt can do it, skip the orchestrator. Add agency when the task truly has steps.
OWASP LLM06 · NIST AI RMF: Manage
P5 Two agent platforms in one stack — overlapping tools, doubled attack surface. see it →
Do this: Pick one orchestrator; migrate the flows, retire the other.
NIST AI RMF: Manage
P6 vLLM without GPU hardware to feed it — a throughput engine with nothing to drive. see it →
Do this: Add the host it runs on, or use Ollama until the hardware exists.
NIST AI RMF: Manage
What does it cost to run? cost · 3
K1 A GPU server with no model running on it — the meter runs either way. see it →
Do this: Either run the model there or drop the GPU until you need it.
OWASP LLM10 · NIST AI RMF: Manage
K2 A cloud model API and a self-hosted model server — you're paying for the same capability twice. see it →
Do this: Decide which is primary; keep the other only if it serves a different job (e.g. embeddings).
OWASP LLM10 · NIST AI RMF: Manage
K3 A pay-per-request model API behind a login-free front door — strangers can run up your bill. see it →
Do this: Put auth on the interface; unbounded consumption is a budget problem before it's a security one.
OWASP LLM10 · NIST AI RMF: Manage · EU AI Act Art. 15
EU AI Act — educational map
Not legal advice. Educational mapping only.
| article | theme | what unauth shows |
|---|---|---|
| Art. 15 | accuracy, robustness, cybersecurity | S1 / S3 / S4 / K3 — unauthenticated surfaces a stranger can use |
| Art. 12 | record-keeping | G3 — continuous third-party prompt logging with no retention decision |
| Art. 10 | data and data governance | G1 / G2 — third parties in the path; documents leaving the building |
Site rubric ↔ scanner severity
| site rubric | scanner severity | grade |
|---|---|---|
| S1 — unauth write/exec block | CRITICAL | F |
| S1 — unauth read block | HIGH | D |
| — info leak (versions, banners) | MEDIUM | C |
| WATCH tier — coherence · cost · proportionality | — | site-only teaching; the scanner never emits these |
the playground teaches the full rubric; the scanner enforces the security slice.
Playground sanity check: teaching rubric, not a security scan. “◉ SIMULATED” marks playground-only runs. “◉ CAPTURED … ON OUR HONEYPOT” marks real replay provenance.
Changelog
v1.5 2026-08-02 — Added EU AI Act article references on security and governance rules (educational mapping only — not a conformity assessment). Art. 15 for unauthenticated exposure, Art. 12 for continuous prompt logging, Art. 10 for documents leaving the building.
v1.4 2026-07-31 — Unified with the scanner/CI severity model. No-authentication blocks are no longer watch-tier (S1 is critical) — one write/exec-class unauth block caps the grade at F, read-class at D, and the stamp counts every unauthenticated block as a critical finding. Fix suggestions never remove the documents block — a boundary is added in front.
v1.3 2026-07-30 — Every rule now carries its framework references (OWASP LLM Top 10, NIST AI RMF) and a plain-English improve hint, and the playground gained its first challenges.
v1.0 2026-07-30 — First published rubric. Five dimensions scored by deterministic rules over the blocks in the stack. Same stack, same grade, every time. No AI guessing, ever.