02 · Case study · Production AI
AI answers are easy. Trust is the engineering problem.
A retrieval-augmented assistant that answers tax and HR questions from real enterprise documents. Building the RAG pipeline was the easy part. The actual work was making the answers trustworthy enough to use.
A demo that retrieves documents is a weekend project. A system employees can trust with real questions is an engineering problem.
Context
Enterprise documents are messy, and a wrong answer about tax or HR policy is worse than no answer. A demo that retrieves documents is a weekend project. A system employees can trust with real questions is an engineering problem.
Constraint
The system could not ship on vibes. Retrieval quality and hallucination rates had to be measured and acceptable before anyone depended on it.
Architecture
- DocumentsTax + HR policy corpus
- Chunking / indexingSemantic embeddingsretrieval quality
- Azure AI SearchHybrid retrieval
- Re-rankingPrecision before generationgrounding
- LLM orchestrationLangGraph pipeline
- EvaluationHallucination checks pre-shiphallucination checks
- AnswerCited, grounded, measured
Decision
Semantic search with re-ranking over the document corpus, orchestrated as an explicit pipeline (LangGraph) so every step could be inspected. Before deployment, retrieval quality and hallucination rates were evaluated automatically. The system didn't ship until the numbers justified it.
Result
A production assistant grounded in the company's own documents, with measured retrieval quality and evaluated failure modes — deployed because the evaluation supported it, not because the demo looked good.
Lessons
- Retrieval quality is the product. The chat interface is the easy part.
- Evaluate before people depend on it, not after.
- A wrong answer about tax or HR policy is worse than no answer.
Stack
Azure AI Search · Azure OpenAI · Copilot Studio · LangGraph