How I split an AI assistant between code and a model

The assistant on this site answers questions about my work. Someone might ask what I built for a particular project, how I handle authorization, or where a claim on the site came from. The source material is a versioned set of approved records. Each record has a summary, a fuller claim, a link, and any caveat the answer needs to preserve.

The first version was straightforward. I gave the model the full set of records and asked it to choose what mattered, write a useful answer, and cite its sources. That made the answers pleasant to read, and it let me build an initial version quickly.

When I checked the answers against the records, I found citations that did not quite hold up. Sometimes the cited record was related to the sentence but did not establish it. Sometimes a sentence blended details from two records and cited only one. The result looked careful to a casual reader, which made the mistakes harder to spot.

That matters here because the assistant speaks about my own work. A visitor should be able to follow a statement back to the record behind it. If the source cannot support the sentence, a numbered citation adds confidence without adding evidence.

I tried to separate two jobs that the first version had given to the model. Finding relevant records is a ranking problem with inputs I can inspect. Writing a clear answer from those records benefits from the model. I could tighten the prompt and keep both jobs together, but the citation errors had already shown me how little a firm instruction proves about a particular answer.

I also considered rendering the records directly. That would make every word traceable to approved copy, but it would force a reader to do the work of turning several claim summaries into an answer. A second model call to review the first was another possibility. It might catch errors in an evaluation set, but it would add latency and cost to every live answer while leaving the final check dependent on another model judgment.

The assistant now selects at most four records before the model writes. That makes the evidence set easier to inspect, although a ranking mistake can leave out a useful record. When the model response fails validation, the site shows the selected summaries directly. That fallback is less conversational, and it gives me another response path to maintain and test.

There is also a limit to what the validator can establish. It checks that the model returned the expected structure and that each sentence has citations to records it was given. It cannot determine from a citation number alone whether the record actually supports the sentence. That kind of error still requires examples, evaluation runs, and review against the source material.

The current assistant uses code to rank the approved records, asks the model to compose from the selected set, and checks the structure of the answer before showing it. If the response fails that check or the model is unavailable, it returns the selected claim summaries with a clear fallback label. The reader can also inspect the evidence and the generation mode for the answer.

That arrangement solved the part of the problem I could enforce in code: the model cannot cite a record it was never given or omit citations from its structured answer. Whether a cited sentence is a fair reading of its source remains a separate quality question. I treat that as an evaluation task, and I would rather describe that limit plainly than suggest a citation validator has settled it.

The distinction between what the model produces and what the application can verify is central to this design. I can narrow the evidence and check the response structure in code. I still have to read examples against their sources to judge whether the answer means what its citations suggest.