Context

Prototype built for a mid-sized services company evaluating whether to buy or build. Budget was one sprint. No access to their production systems, so the corpus was a redacted export.

The problem

New hires were re-asking the same policy questions in chat. The existing handbook was 200 pages of PDF that nobody searched, and the team had already tried pasting it into a general chatbot — which confidently invented policy that did not exist.

Approach & decisions

  1. Chunked by heading structure rather than fixed token windows — the handbook was already organised by policy section, so honouring those boundaries kept each chunk semantically whole. Fixed-size chunking split single policies across three chunks and was measurably worse in spot checks.
  2. Embedded and stored in SQLite with a vector extension instead of standing up a vector database. At 900 chunks the whole index is a few megabytes; a server-side vector DB would have been pure operational overhead for the client.
  3. Required the model to return a citation or the literal string "not in the handbook". A refusal that a user can verify is worth more than a confident guess they cannot.
  4. Added a 20-question evaluation set written from real support tickets, run on every change. Without it, every prompt tweak felt like an improvement and we had no way to tell.

Result

On the internal evaluation set, grounded answers with a correct citation went from 61% (naive paste-the-PDF baseline) to 89%. Refusal behaviour on out-of-scope questions was correct in 24 of 25 cases. Delivered as a runnable prototype plus a writeup of what a production rollout would require.

Why this is marked self-initiated

This was built on a redacted sample corpus without a signed contract, so it is not presented as client work. It is here because it shows the shape of the problem I solve and how I measure whether it is actually solved.

What a production version would need

The prototype deliberately skipped three things that would matter in production, and I said so in the handover: access control per document (the handbook has manager-only sections), a re-indexing trigger when the source document changes, and cost monitoring per conversation. None of those are hard, but all of them are scope.