Key takeaways
- The AI should never see more than the person asking is allowed to see.
- Log prompts, sources and outputs so you can investigate later.
- Evaluate on your own questions, not on a vendor demo.
- Put a human in the loop where errors are costly, and decide data residency early.
A team connects an AI assistant to the shared drive and the ERP over a weekend. On Monday it answers questions brilliantly, including one about salary bands that the asker should never have seen. Nothing was hacked. The assistant simply had more access than the people using it. That is the core risk, and it is manageable.
Access: inherit permissions, do not bypass them
The safest design makes the assistant act on behalf of the signed-in user and applies that user's existing permissions at query time. Avoid a single service account with broad read access, because it turns every user into an administrator through the chat window.
- Retrieve documents through the same access controls your file and application systems already enforce.
- Exclude sensitive repositories by default and add them deliberately.
- Review connectors the way you review any integration with production data.
- Remove access when people change roles, as you would for any system.
Logging: be able to reconstruct what happened
When an answer is wrong or a data exposure is suspected, you need to see who asked what, which sources were retrieved, which model version responded and what came back. Keep those records with sensible retention and restricted access, since the logs themselves may contain sensitive content.
Logging also helps improvement. Repeated questions that the assistant fails on show where documentation or data is missing.
Evaluation: test on your own questions
Vendor demos use friendly examples. Build a small evaluation set from real questions your staff ask, along with the answers a knowledgeable colleague would give. Run it before launch, after every model or prompt change, and on a schedule.
Include hard cases: questions with no answer in the data, questions that try to pull restricted information, and instructions hidden inside documents. Score not only correctness but also whether the assistant admits uncertainty and cites its sources.
Human review where mistakes are expensive
Not every output needs approval. Drafting an internal summary is low risk. Sending a message to a customer, changing a record in the ERP or making a financial commitment is not. Classify tasks by consequence and require confirmation for the higher tier. Make the review easy, showing the sources and the proposed action side by side, or people will click approve without reading.
- Read-only answers with citations: low risk, light review.
- Drafts a person edits and sends: moderate risk, review by the sender.
- Actions that change records or spend money: explicit approval, and logging of who approved.
Data residency and third parties
Decide early where data may be processed and stored. Check where your provider runs the model, whether prompts are retained, and whether your data is used for training under your agreement. Terms differ by product and plan, so read them for the specific service you buy rather than assuming.
Also consider contractual and regulatory obligations that apply to your industry or your customers, such as restrictions on moving certain personal or financial data across borders. Involve legal and security early; it is much cheaper than retrofitting.
Start small and widen deliberately
Begin with one department, a limited set of documents and read-only access. Measure the evaluation results, review the logs and fix gaps. Then expand sources or actions one step at a time. Companies get into trouble when a pilot quietly becomes a platform without anyone revisiting the controls.
If it would help, we can review your planned data connections and suggest a staged rollout with the right checks at each step.
