When two users see different evidence

I use a hypothetical analyst and executive to think through a problem I have encountered in systems with tiered data. The analyst can see public filings and team notes. The executive can also see a board deck, a forecast, and deal terms. Both ask an assistant whether the company should pursue an opportunity.

The first design requirement is straightforward: the assistant should retrieve only records the person is allowed to see. I have built authorization into retrieval paths before. The user's identity and permissions have to travel with the request, including any tool calls the assistant makes while preparing an answer.

The two users may receive different recommendations because they are working from different records. The analyst may have enough information for a preliminary view while the executive has facts that would change the decision. For example, the analyst's records might support further diligence but say nothing about margin or contract risk. The executive's terms might settle those questions. The analyst still needs an answer that is useful within the evidence available to them.

An assistant that varies its warning based on hidden records can reveal something about those records. If it says "there may be more to this" for some questions and stays silent for others, repeated questions could expose the shape of information the analyst cannot access. There is a second problem even when retrieval is correctly scoped. A model can turn general knowledge into a plausible guess that sounds like a conclusion from the supplied records. Citations help a reader inspect the answer, but their presence alone does not prove that the conclusion follows from the evidence.

One option is to let the model see the executive's records and ask it to write a redacted answer for the analyst. I would avoid that design. Once those records are in the model's context, the lower-tier answer depends on the model successfully suppressing their influence in its wording, emphasis, and confidence.

Another option is to warn every user that other information may exist. A constant warning avoids the selective signal, but it gives the analyst little help in deciding whether the present answer is sufficient. I would show the evidence used for every answer and make the limits of those records clear. That rule would apply consistently, whatever the user's tier.

The analyst's answer may be less decisive than the executive's. Sometimes that is frustrating for the person who has to do the first pass on a decision. The system also needs a route for questions whose consequences exceed the evidence available to the person asking. That route requires an agreed human decision owner and a workable review process.

Evidence receipts and citation checks create their own maintenance work. They need tests that compare answers with the underlying records, including questions designed to coax the model toward a restricted conclusion. A validator that merely checks citation numbers would leave the central problem untouched.

For this design, I would scope retrieval and tools to the user's permissions, give every answer the same kind of evidence receipt, and use consistent language when the authorized records do not support a recommendation. For consequential decisions, I would offer a path to a person with the authority and context to review them. I would keep restricted records out of the lower-tier model context throughout that flow.

I have not deployed this exact workflow. Before calling it effective, I would test paired analyst and executive questions, repeated probes, and answers that sound reasonable but reach beyond their cited records. If those tests showed that the analyst could infer restricted facts or that the assistant made unsupported recommendations, I would revise the answer policy before putting it in front of users.

The broader lesson is that an LLM's ability to produce a smooth answer says little about whether it was allowed to see the records behind that answer. Access has to be enforced in the retrieval and tool path before the model is asked to write.