Verity: one assistant, and an answer shaped by what you're allowed to read
Most company knowledge assistants face an ugly choice: index everything and risk leaking the salary spreadsheet, or index only the safe documents and be useless. Verity takes neither. It asks who is asking, then answers only from documents that person is cleared to read — and when nothing they can read supports an answer, it says so instead of guessing. The same question, asked by two people, correctly produces two different answers.
- Type
- Permission-aware RAG · multi-tenant
- Stack
- Python · FastAPI · Postgres + pgvector · Azure OpenAI · Next.js
- Tests
- 371, plus a 44-question evaluation harness
- Security
- 0 of 48 cross-tenant attempts succeeded
Permission belongs in the query, not after it
The tempting way to build this is to search everything, then remove the results the user isn't allowed to see. It works right up until it doesn't. That design has already fetched the restricted passage, already ranked it, already got it sitting in memory next to the answer — and the only thing standing between it and the user is one filter that a future refactor can forget. It also lies quietly about relevance, because a restricted document can crowd out the permitted one that would actually have answered.
Verity puts the permission check inside the database query that does the search. Your groups become part of the SQL predicate, so a document you may not read is never a candidate in the first place — there is no ranked-but-hidden chunk to leak. The visible consequence is the one in the project image: ask for the Severity 1 escalation codeword as a support engineer and the runbook holding it is never searched, so the refusal is structural rather than a policy the model was asked to follow.
The bug hunt that found two worse bugs
An early review caught that the document *listing* endpoint filtered by company but not by group — so you could see the titles of restricted files. Fixing it meant reading every database query in the project, and that read found two holes considerably worse than the one I set out to fix. Fetching a single document checked only the company, which meant anyone could delete a restricted document they were never allowed to open. And re-uploading a document overwrote its permission list, so any user could quietly strip the protection off a restricted file.
Both are now closed at the database layer, including a guard clause that refuses the update rather than silently widening access. The lesson I keep from it: the reported bug is a symptom, and the fix is worth nothing next to the audit it justifies. A suite of 48 adversarial tests now acts as a hard gate — it deliberately tries to read, delete, and overwrite another tenant's data, and every attempt has to fail before anything ships.
Refusing well is a feature, not a fallback
It is easy to build a system that never hallucinates: make it refuse everything. That scores perfectly on the metric everyone quotes and is completely worthless. So the evaluation set measures both directions — 32 questions the documents genuinely answer, 8 the documents don't cover at all, and 4 whose answer exists but sits behind a permission the asker lacks. Wrongly refusing an answerable question is tracked as its own failure, with equal weight to inventing one.
In the interface, this shows up as a design decision rather than an error. A refusal isn't a red toast that reads like a malfunction — it's a stamped "insufficient evidence" note, because declining is the product working. The current measurements: every answerable question retrieves the right document first, no answerable question is wrongly refused, no claim appears that isn't traceable to a retrieved passage, and none of the four restricted questions leaked.
Measure first, and distrust the results you like
The evaluation harness was built before any tuning, so every retrieval decision could be adopted or rejected on a number instead of a hunch. Some were rejected: a more sophisticated chunking strategy that respects document structure measured no better than plain fixed-size chunks on this corpus, so it stayed switched off and the negative result got written down rather than buried.
The habit that paid off most was suspicion of clean results. An early run reported that searching the top 1 result was exactly as accurate as searching the top 10 — an implausible coincidence that turned out to be the test endpoint applying a relevance cutoff before returning, which made "not found" and "found but trimmed" indistinguishable. Later the same instinct had to be pointed the other way: a query-rewriting feature appeared to reduce accuracy, and I wrote a confident explanation for why. Then I noticed three unrelated metrics had each fallen by precisely the same amount — the signature of three questions returning nothing at all, not of degraded quality. They were transient API errors. A re-run scored perfectly. Errors are now a first-class field in the scorecard with a loud banner and a non-zero exit code, and runs can repeat to show their spread. A believable failure gets less scrutiny than a suspicious success, which is exactly backwards.
A stop button that actually stops
Answers stream a word at a time over Server-Sent Events, with the details that decide whether streaming survives real infrastructure: buffering and compression explicitly disabled, a heartbeat so load balancers don't close a connection while retrieval is still thinking, and mid-stream failures delivered as an event, because once the first byte is sent the HTTP status is already 200 and returning a 500 is no longer available.
Proving the stop button worked is what found the most interesting bug in the project. Cancellation reached the server instantly, but the cleanup that saves the partial answer and closes the connection to the model didn't run for ninety-four seconds. The cause isn't logic — it's that a Python generator suspended at a yield never has anything thrown into it, so its cleanup block waits for garbage collection, and iterating a stream doesn't close it. An explicit close on the way out brought ninety-four seconds down to milliseconds. I only found it because the exit criteria demanded log evidence rather than a plausible mechanism.
Seams that earned their keep
The architecture was planned in writing before any code, with the irreversible decisions — how tenants share tables, where the permission check lives, what identifies a chunk — settled first and recorded as decision records, and the reversible ones deliberately deferred. Every model call goes through one interface, so the whole pipeline ran against a deterministic fake before any cloud credentials existed, and that fake is now the path the test suite uses, which is why 371 tests cost nothing to run and never flake on an API hiccup.
Two moments proved the seams were real rather than decorative. Adding full authentication changed the body of exactly one function, and the evaluation fingerprint afterwards was byte-identical — measurable proof the change was behaviour-preserving. Adding an AI reranking stage never touched the retrieval code at all. Both were adopted only after measurement said they helped.