I’ve been running an AI agent on a B2B technical support inbox for a few months. The interesting part isn’t the answering — it’s what happens when the model *doesn’t* know. Sharing the architecture in case it’s useful.
The flow
1.Gmail trigger on the support address.
2.A filter node kills auto-replies, out-of-office, and no-reply noise *before* anything hits the LLM. Skipping this cost me an infinite loop on day one: an OOO bounced back, the agent replied to it, and the two of them talked to each other.
3.Agent step: LLM with the knowledge base attached as a tool (a spreadsheet, \~1,200 Q&A rows) and a structured output parser returning {answer, confidence, reasoning}.
4.Confidence gate. This is the whole design.
-Confident:the draft goes to a human for approval via a send-and-wait node. Approve → it sends. Reject → drops into the manual path.
-Not confident:*the customer immediately gets “an expert will be in touch” (sets expectation, buys time), while a human gets pinged for the real answer. Their answer goes back out.
5.Both paths converge into an enrichment step. A second agent takes the exchange, normalizes it (strips pleasantries, generalizes customer-specific details, one question per row) and appends it to the knowledge base.
So the KB grows specifically from the cases the agent couldn’t handle. Escalation rate drops over time without anyone maintaining documentation.
Things I got wrong first
-Asking the model “are you confident?” is useless — it’s confident about everything. I had to force explicit criteria into the structured output: did you find a matching KB entry, does it cover the *full* question, not a vibes score.
-Raw email threads make terrible KB entries. The enrichment agent isn’t optional — without normalization the KB fills with noise and retrieval degrades.
-Send-and-wait is the most underrated node in n8n. Human-in-the-loop without building an app.
-A flat spreadsheet as KB works better than expected at this scale. I’d move to a vector store past a few thousand rows.
Stack:n8n, Gemini Flash, Google Sheets, Gmail.
Curious how others handle the confidence gate — self-reported confidence still feels like the weakest link. Anyone scoring retrieval quality instead?
