The challenge
The client's support team was drowning. Their product had grown across three distinct product areas, and every release added more surface area to support. Customers waited too long for answers, and the team was spending most of its time on questions that were already answered somewhere in the docs.
They'd seen AI demos and were curious — but they were also worried about hallucinations, security, and the long tail of "weird" support questions that real customers send.
What we did
We ran the engagement in three phases.
Phase 1 — Feasibility (2 weeks)
We instrumented their existing ticket flow with anonymized eval data, prototyped a retrieval-augmented system on Anthropic Claude, and ran it against 500 historical tickets. The numbers showed clear potential — but also clear risks. We presented both to leadership before continuing.
Phase 2 — Build (10 weeks)
We built a retrieval pipeline that indexed their docs, release notes, and historical resolved tickets into a pgvector store. The copilot:
- Searches the knowledge base with hybrid lexical + semantic retrieval.
- Cites every claim with linked source documents.
- Hands off to humans when confidence is low — never bluffing an answer.
- Logs everything to Langfuse so quality is visible, not hoped-for.
We also built a guardrail layer that handled prompt injection, PII redaction, and content safety — and an internal admin UI for the support team to review, correct, and re-rank responses.
Phase 3 — Rollout (4 weeks)
We rolled out gradually: 5% of inbound tickets, then 25%, then full deployment with human-in-the-loop. We worked closely with the support team to tune escalation thresholds and build trust.
Outcome
After three months in full production:
- 38% of inbound tickets are now fully deflected by the copilot.
- Average response time for tickets that need humans is down 52% — the agents start with summaries and suggested replies.
- CSAT increased 11 points — customers especially appreciate getting answers at 3am without waiting.
- The engineering team can now add a new product area's coverage in a single day instead of weeks.
Just as importantly, the support team's morale improved. They went from firefighting to spending real time on the genuinely hard customer problems that need a human.