Most AI chat demos try to answer every message. In a real business inbox, that is the wrong goal. A good agent knows what it is allowed to do, and it knows when to step back and bring in a person.
This post explains how I designed the AI inbox agents for a client.
The problem
Agencies use the CRM to talk with their clients' customers on WhatsApp and email etc, They wanted AI agents that could:
- • answer common questions from the business's own knowledge base,
- • capture details such as an email or a company name, and tag and qualify leads,
- • hand the conversation to the right person when the AI should not continue
The hard part was not the reply. It was the decision around the reply.
The shape of the system
There are two services. The main platform is a .NET modular monolith. It owns conversations, contacts, and billing. A separate Python service (FastAPI, LangGraph, pgvector) owns everything AI: agent settings, knowledge retrieval, model calls, and usage records.
The platform never calls an LLM directly, and the AI service never writes to the CRM directly. Each side does one job, and they talk over a signed service-to-service API.
What happens when a message arrives
- 1. The platform saves the message. A WhatsApp or email webhook delivers it, and the AI trigger fires only after the save commits.
- 2. It waits 4 seconds. Customers often send three short messages in a row. The background job checks whether a newer message arrived, so the agent answers all three with one reply instead of three.
- 3. It reserves credits first. If the client's wallet is empty, the agent stays quiet and the thread stays with the human team.
- 4. It calls the AI service. The request carries the conversation, the contact, and the credit reservation.
- 5. The agent thinks once. The AI service loads the agent, retrieves knowledge from pgvector (only from the sources this agent may use), and makes one structured LLM call.
- 6. Rules check the result. More on this below.
- 7. The platform applies the decision. It sends the reply, updates the CRM, routes any handoff, and pushes the change to the live inbox. The AI service reports the real token usage, and the platform settles the reserved credits.
One call, one decision
Instead of a chain of tool calls, the agent makes one call that returns everything it wants to do as a single JSON object:
{
"reply": "Yes, we deliver on Friday. Can I take your email to confirm?",
"actions": [
{ "type": "update_contact_field", "field": "company", "value": "Acme Studio" },
{ "type": "qualify", "score": "warm", "reason": "Asked about delivery dates" }
],
"handoff": null,
"confidence": 0.82
}One call keeps latency and cost predictable. It also gives me one place to check everything before anything happens. If the model returns invalid JSON, it gets one retry with the error message.
The model suggests, the rules decide
The model's output is only a proposal. Plain code then applies rules that the model cannot talk its way around:
- • Allow-listed actions. The agent can only update the fields, apply the tags, and qualify leads that its settings permit. Anything else is dropped.
- • Valid targets only. A handoff counts only if it points to a target that the business configured.
- • Forced human handoff. The conversation goes to a person when the customer asks for one, when the message touches a blocked topic, when the agent reaches its turn limit, or when the model's confidence is below the threshold.
When a rule forces a handoff, the agent also writes a short summary for the teammate, so nobody has to read the whole thread from the start.
Handoffs between agents
A business can run several agents, for example a receptionist, a sales agent, and a support agent. Each handoff target is written as a routing rule, such as "route here when the customer asks about pricing". The model reads these rules and picks a target.
When the target is another AI agent, the transfer happens in the same turn: the receiving agent reads a handover note and answers the customer at once. I limited this to one hop. The receiving agent can still escalate to a human, but never to another AI agent, so two agents can never pass a customer back and forth.
If the target agent is switched off or missing, the handoff falls back to a human. Human handoffs assign the conversation round-robin across the teammates the business selected.
Failure modes I designed for
- • The AI service fails. The credit hold is released, the job retries temporary errors, and the conversation stays visible to the team.
- • A CRM action fails. The reply still goes out. A lost tag can be fixed by hand; a lost reply cannot.
- • Data isolation. Every query is scoped to the agency and the client account, and the AI service connects with a database role that can only read and write rows.
Pricing as data
Each model and its price live in a catalog table, not in code. One model router serves OpenAI, Gemini, and Claude. The team can switch an agent's model or change a price without a deploy, and every run records its tokens, provider cost, and credits.
The main lesson
Let the model propose, and let your guardrails decide. That one rule is what makes an AI agent safe enough to trust with a real business inbox.