Nothing can think — no model server or model API in this stack.
Do this: Add Ollama (self-hosted) or a cloud model API (OpenAI, Anthropic, Bedrock).
NIST AI RMF: Map
The public rulebook
The rules v1.3
Every grade the playground gives comes from these rules and nothing else. No model, no mood, no mystery: same stack, same grade, every time. If you think a rule is wrong, the argument starts here.
How a stack is scored
Each of the five dimensions starts at 5 points. A critical rule that fires costs 2 points in its dimension, a watch costs 1, and no dimension drops below 1. The letter is the average across dimensions: A at 4.5+, B at 3.8+, C at 3.0+, D at 2.0+, E below that. One exception: two or more critical rules firing caps the letter at D, because a tidy average must not hide a disaster. A rule marked ok is praise, not a finding — it costs nothing.
Does it hang together? coherence
The model can answer, but no human has a way to talk to it.
Do this: Add a chat interface (Open WebUI, AnythingLLM) or a notebook.
NIST AI RMF: Map
A vector database with no documents is an expensive empty room.
Do this: Add your documents — retrieval exists to serve them.
NIST AI RMF: Map
Documents are there, but nothing can find the relevant page at question time.
Do this: Add a vector database between your files and the model.
NIST AI RMF: Map
An orchestrator with nothing to orchestrate — no tools, no model.
Do this: Give it a model to call or tools (MCP) to use.
NIST AI RMF: Map
Could a stranger use it? security
Blocks that ship with no authentication by default — normal, and exactly what attackers scan for.
Do this: Each block's fix card has the lock-down steps; run the scanner on the real server.
OWASP LLM10 · NIST AI RMF: Manage
A demo UI in the stack — the classic 'temporary prototype' that quietly becomes production.
Do this: Decide now: is this a demo or a product? Products get auth and a proper interface.
NIST AI RMF: Manage
A notebook server is a terminal on your machine with a friendlier face.
Do this: Never expose it; reach it through a tunnel or VPN instead.
NIST AI RMF: Manage
Three or more blocks a stranger can use with no login at all — that's not a stack, it's a buffet.
Do this: Lock down the fix-card steps per block, or cut what you don't need.
OWASP LLM10 · NIST AI RMF: Manage
Exposed blocks sit behind a boundary — the right instinct: keep the capability, add the gate.
Do this: In reality coverage is per-service: verify every block is actually behind the boundary, then run the scanner to prove it.
NIST AI RMF: Manage
Where does the data go? governance
A third party is in the stack — its risk is contractual, not network.
Do this: Ask for each cloud block: DPA signed? retention settings? region? may they train on your data?
OWASP LLM02 · NIST AI RMF: Govern
Your documents leave the building — they cross to a third party on every request.
Do this: Sometimes that's the right call; make sure it's an allowed one. Self-hosted retrieval keeps them home.
OWASP LLM02 · NIST AI RMF: Govern
Continuous third-party logging — every prompt lands on someone else's platform, indefinitely.
Do this: Review what may be traced and set retention consciously.
OWASP LLM02 · NIST AI RMF: Govern
Is it the right size? proportionality
Two vector databases doing one job — that's a coin flip, not a decision.
Do this: Pick one. Chroma until you have a reason; Qdrant when you do; Pinecone if running one is the problem.
NIST AI RMF: Manage
Two self-hosted model servers — one is enough until throughput forces the question.
Do this: Start with Ollama; add vLLM when many users hit it at once.
NIST AI RMF: Manage
vLLM at this scale is gold-plating — it exists for many concurrent users.
Do this: Ollama fits a small stack; switch when throughput actually hurts.
NIST AI RMF: Manage
An agent framework for what looks like a single-step task.
Do this: If one prompt can do it, skip the orchestrator. Add agency when the task truly has steps.
OWASP LLM06 · NIST AI RMF: Manage
Two agent platforms in one stack — overlapping tools, doubled attack surface.
Do this: Pick one orchestrator; migrate the flows, retire the other.
NIST AI RMF: Manage
vLLM without GPU hardware to feed it — a throughput engine with nothing to drive.
Do this: Add the host it runs on, or use Ollama until the hardware exists.
NIST AI RMF: Manage
What does it cost to run? cost
A GPU server with no model running on it — the meter runs either way.
Do this: Either run the model there or drop the GPU until you need it.
OWASP LLM10 · NIST AI RMF: Manage
A cloud model API and a self-hosted model server — you're paying for the same capability twice.
Do this: Decide which is primary; keep the other only if it serves a different job (e.g. embeddings).
OWASP LLM10 · NIST AI RMF: Manage
A pay-per-request model API behind a login-free front door — strangers can run up your bill.
Do this: Put auth on the interface; unbounded consumption is a budget problem before it's a security one.
OWASP LLM10 · NIST AI RMF: Manage
Changelog
v1.3 2026-07-30 — Every rule now carries its framework references (OWASP LLM Top 10, NIST AI RMF) and a plain-English improve hint, and the playground gained its first challenges.
v1.0 2026-07-30 — First published rubric. Five dimensions scored by deterministic rules over the blocks in the stack. Same stack, same grade, every time. No AI guessing, ever.
Rules change when the world does: new products, new attack patterns, arguments won in public. Every change lands here with a date.
the playground grades with these rules · the scanner checks the real thing · build a stack · scan your server