Building AI systems that people can trust, verify and improve

5 production principles that determine whether an AI system actually changes how a company works.

Across my work with different enterprises and use cases, I’ve found that the hardest problems rarely sit inside the actual AI engineering. Instead, they sit in the decisions around it. When those decisions aren’t thought through carefully, it’s easy to end up with a sophisticated AI system that changes very little about how work actually gets done.

I’ve distilled my experience into 5 principles for building AI systems that can operate inside a business, earn people’s trust and improve over time. This article explains each one through a production system I built for Ntiva, a $100M+ managed IT provider, to address one of its most pressing problems at the time: client churn.

1. Tailor the product shape to the business workflow
2. Give each workflow step to the simplest mechanism that handles it well
3. Redesign the human’s job at the agent handoff
4. Design the agent handoff for human verification and feedback
5. Build a governed self-improvement loop from human feedback

The case study: Detecting churn risk hidden across service tickets

Ntiva runs IT operations for its clients and was facing unprecedented client churn at the time. One of the clearest warning signs is a service issue that keeps coming back for the same client. The problem is that this warning almost never arrives as a warning.

Instead, it arrives as separate incidents spread over weeks or months. Each incident looks small and self-contained. The commercial risk only becomes visible when someone notices that the same underlying problem keeps returning.

Spotting that pattern meant relying on busy people to connect the dots. Support technicians saw individual service tickets raised by clients. Account managers reviewed client touch points and dashboards. For the risk to surface, someone had to remember the earlier incidents, judge whether they formed a pattern, gather the evidence and alert the person who owned the client relationship.

Across many clients and a high volume of service tickets, connecting those dots by hand was prohibitively manual. Ntiva couldn’t keep tabs on every client and every risk signal consistently at scale.

Tailor the product shape to the business workflow

The common instinct I see most teams follow is to put an AI system behind a chat interface. Ntiva tried that route, too. The team first investigated churn through its enterprise data agent, a separate system I’d previously built for them so employees could ask questions about company data in natural language.

But in my experience, a chat interface is often the wrong product shape for most enterprise workflows, for 2 reasons:

1. It waits for someone to think of the right question at the right moment. Business work usually starts because something happened, not because a person remembered to ask about it. In this case, early churn detection has to begin when new evidence arrives and keep running in the background.
2. It makes employee adoption a prerequisite for the business outcome. For adoption to just work, the AI system has to run without anyone needing to drive it. Ideally, it should also require minimal change to existing behaviour.

Ntiva’s workflow also depends on patterns that build up over time. To run efficiently, the system should retain what it has already learned about each client’s ticket history instead of rebuilding that history in response to every question. For example, a stateful background process can store each client’s issue history and continue from where it left off, so every new run handles only the tickets that arrived since the last one.

You could bolt those capabilities onto a chat-based agent with an agent skill that explains how to store findings and process only new tickets. However, once that capability carries persistent state and fires on a business event, it already operates as a separate system. Hiding it behind chat doesn’t simplify the system, it only disguises a background workflow as a conversation.

As such, I guided Ntiva away from treating churn detection as another question-and-answer use case for its chat-based enterprise data agent. Instead, I built a separate, event-driven AI system. This choice shifts the system’s job from helping employees inspect churn risk to owning the detection work itself. The system watches for new evidence, carries each client’s ticket history forward and starts the right workflow without waiting for someone to prompt it.

Give each workflow step to the simplest mechanism that handles it well

The mistake I see teams make most often is handing an agent the entire workflow. A value needs comparing, so the agent compares it. A result needs routing, so the agent routes it. Putting everything in one prompt can look simpler, but in production it folds exact rules, ambiguous investigation and commercial judgement into one opaque LLM run.

The strongest production systems I’ve seen and built don’t use the most sophisticated agent at every step. They give each step to the simplest mechanism that handles it well, so the complete workflow stays accurate, understandable and cheap to run at scale.

The real design work is deciding which steps belong to deterministic code or a specialised method, which need an agent and which have to stay with a person. The agent is one decision-maker inside the system. It goes where judgement is needed, not everywhere by default.

For Ntiva, the workflow takes this form: The system watches each client’s ticket history and groups incidents that might describe the same underlying issue. When a group crosses Ntiva’s definition of recurrence, an agent investigates whether the tickets represent one continuing problem and whether the problem deserves an account manager’s attention. Only a finding that clears both bars becomes an alert for the account manager who owns that client.

The AI system I built runs this workflow in 2 linked stages:

1. A once-off preparation builds a compact picture of each client’s recent issues.
2. A scheduled weekly run compares new tickets with that picture, investigates recurring patterns and escalates only the ones that earn attention.

A summarizer agent handles language judgement

During the preparation stage, the system pulls each client’s last 3 months of service tickets from Ntiva’s ticketing system. A single ticket can hold a long email thread between the client and a support technician, which is far more context than the rest of the workflow needs. To cut context size and noise in every downstream step, I compress each ticket to only its useful signal.

This is the type of task an LLM shines at. A summarizer agent reads the full ticket and reduces it to one concise issue summary while keeping enough detail to identify and compare the problem.

Semantic similarity groups tickets without using an agent

The next step groups tickets that might describe the same underlying issue. Similarity is probabilistic, so no exact rule can draw the line. But the system also doesn’t need an agent to reason through every possible pair, which would be slow and expensive at volume.

As such, I use semantic similarity here. The system turns each issue summary into an embedding, a numerical representation of its meaning, then uses clustering to build the initial issue groups separately for each client. This way, the system handles similarity directly, cheaply and through a repeatable method.

Deterministic code owns the recurrence threshold

The once-off preparation creates the initial issue groups. The scheduled weekly run comes next. Every Monday morning, the system pulls the previous week’s new tickets for each client and puts each one through the same summarize-and-embed steps. It then compares each new embedding with that client’s existing issue groups. A close enough match joins the ticket to that group. A ticket with no close match starts a new group.

Deterministic code checks Ntiva’s rule for X similar tickets within Y days. A threshold like that is exact by definition, so it belongs in code rather than an agent that might apply it inconsistently.

If the threshold is crossed, the system doesn’t yet send an alert to the account manager. Semantic similarity is only a cheap, quick first pass at grouping tickets that might be related. Crossing the threshold means the group might contain a recurring issue, not that the similarity match proves the client has one continuing problem. An investigation agent has to make that call.

An investigation agent judges the evidence

When an issue group crosses the threshold, the system starts an investigation agent for that group. The agent pulls the full history of every ticket in the group from Ntiva’s ticketing system. From there, it decides what else it needs and retrieves it from the same source, such as details about the client or the IT system at fault.

No rule or embedding can settle this part. The agent has to read conflicting or incomplete evidence and make 2 judgements: whether the tickets represent one continuing issue and whether the pattern warrants an account manager’s attention.

The 2 judgements stay separate because a repeated pattern and a useful risk signal aren’t the same thing. For example, planned maintenance can generate several related tickets without pointing to a damaged relationship, or a recurring technical fault can be genuine and still not deserve an account manager’s time. Only a finding that’s both real and commercially relevant becomes an account manager alert.

A human handles the decision to act

Deterministic routing in code emails the alert to the account manager assigned to that client. The AI system doesn’t prescribe what the account manager should do next. The account manager keeps the relationship judgement because the right response depends on the client and relationship context outside the technical evidence.

Across the complete AI system, code handles exact rules; embeddings handle semantic similarity; agents handle messy evidence and investigations; humans keep the decision with real commercial consequences. This division gives each step to the simplest mechanism that handles it well, so the complete workflow stays accurate, understandable and cheap to run at scale.

Redesign the human’s job at the agent handoff

Adding an AI system to a business workflow changes what the people in it do. One of the most important product decisions is where the system stops, what it hands over and what the humans still own.

An AI system usually ends in one of 2 ways:

• It completes the workflow, while people spot-check its work from time to time
• It stops before the final decision and gives a person all of the context needed to decide and act quickly

In the second case, the AI system is respnsible for filtering what needs human attention, surfacing only those cases and bringing the relevant context with them. Done well, that turns 30 minutes of a human reconstructing what happened into a 1-minute decision. The system completes the searching, analysis and evidence assembly that come before the judgement call, then stops where a person’s authority still matters.

This is Ntiva’s case, where the account manager still owns the client relationship. The system emails a consolidated, fully investigated risk alert to whoever is assigned to that client. This means the account manager no longer has to dig through tickets or piece together what happened. They can spend their attention on protecting the relationship using the evidence the system has already assembled.

But this handoff only works if the account manager can trust the system’s findings.

Design the agent handoff for human verification and feedback

The agent handoff is the final interface a person sees of the agent’s work.

Verification starts at product design, not at the end

A background AI system raises the bar for verification. With a chat-based agent, the user sees each response as it arrives. But a background system can repeat a bad judgement across many cases before anyone notices.

Teams often put most of their effort into the AI engineering, and the handoff gets whatever attention is left. That misses one of the highest-leverage parts of the system. A sophisticated agent is useless if the people receiving its output can’t trust it enough to use it, and trust comes from verification and traceability. If the person receiving the agent handoff has to redo the agent’s work to confirm its conclusion, the system has exposed the wrong evidence.

That leads to a simple design principle: Make the agent’s work easy to verify before building evaluations around its output. Verification can’t be added at the end as a score around an otherwise opaque answer. It has to shape what the system produces, how the output is broken down and which evidence travels with every conclusion.

For Ntiva, the agent handoff is the account manager alert. I designed that alert so reviewable evidence is an intentional output, instead of just stating a risk level and leaving the account manager to reconstruct why.

Start with how a human does the work

In this case, don’t start by asking what an alert normally looks like. Start with how the person receiving it already checks whether a finding is correct. For Ntiva, that means understanding how an account manager confirms that a recurring client issue is real.

They need to see what the system believes is recurring, how often it happened, which tickets support the conclusion and what each ticket says. They also need access to the source record. This checking process determines the handoff interface design.

The evidence packet therefore opens with the client, the recurring issue and its frequency. It then shows the tickets that support the finding. Each ticket has a concise summary and a direct link to its full record in Ntiva’s ticketing system.

Here’s what an example alert looks like:

Reveal the evidence without overwhelming the human

This interface uses progressive disclosure. The account manager sees the finding and the most useful ticket evidence immediately. It shows 2 representative tickets and makes clear that 3 more belong to the group. The reviewer doesn’t have to read every raw ticket before deciding whether the pattern makes sense, but every conclusion stays linked to its source of truth in Ntiva’s ticketing system.

This design reduces reading without hiding provenance. When the evidence is incomplete or contradictory, the agent states that beside the affected ticket instead of presenting its conclusion as settled. The account manager can see what the system knows, what remains uncertain and where each claim came from.

Collect feedback at each agent decision point

Feedback helps most when it identifies which part of the AI system got something wrong. When a system contains several agent decision points, a generic thumbs-up or thumbs-down on its final output can’t tell you where the mistake occurred. That makes the system hard to improve.

Products such as ChatGPT and Claude use a generic rating on the final response because one interface has to support everything from financial analysis to legal work and frontend design. But a system built for one specific enterprise workflow has an advantage. Its important decisions are known ahead of time, so its interface can ask for feedback exactly where each judgement occurs.

The design goal is to make such evaluation part of normal use. Account managers don’t have to leave their normal workflow to label abstract AI outputs or repeat the searching and analysis that produced the alert. They make 2 quick calls while reviewing a finding they already care about:

1. Is this ticket part of this recurring issue? This checks whether the system grouped the right client incidents together.
2. Is this issue worth flagging to you? This checks whether a confirmed recurring pattern deserves the account manager’s attention.

The 2 questions stay separate because the AI system can group the tickets correctly and still be wrong that the pattern matters commercially. They also produce 2 different evaluation signals. Ticket-level answers show whether the system groups incidents correctly, and issue-level answers show whether a confirmed pattern deserves attention. Instead of one rating on the final alert, the team learns which decision needs improving.

Clicking any Yes or No opens an optional free-text box that captures the reason behind the human judgement, which is what the improvement process needs.

The easier this feedback is to give during normal work, the more likely people are to give it. Breaking one large finding into smaller units also makes annotation less expensive and produces more useful signals for future evaluation.

Let verification become lighter as trust grows

The right amount of visible evidence can change as users build confidence in the AI system. During an early rollout, the handoff interface can show ticket evidence and agent reasoning by default so people can inspect how the system behaves. As the system earns trust, some detail can collapse by default while the full evidence remains available.

The product should reduce the effort required to review the work as confidence grows, but it shouldn’t remove the ability to check the work.

Build a governed self-improvement loop from human feedback

Once the system has collected feedback, a separate feedback-ingestion workflow turns it into proposed changes for future runs.

In this case, I trace every production run in LangSmith. When an account manager answers Yes or No and leaves feedback, that response attaches to the exact agent run that produced the alert. This link keeps the correction connected to the evidence, agent decisions and output it refers to.

The feedback-ingestion workflow reads the runs that carry feedback, the feedback itself, and the agent’s current instructions. It then determines what reusable instruction follows from the correction and proposes a focused change to the instruction that governs that decision. These instructions form the agent’s procedural memory, which is the guidance it carries into future runs about how to perform the work.

Feedback can point to code or workflow changes too. But in practice, I’ve found that most useful improvements land in procedural memory. Updating one instruction is often the simplest change available that produces outsized gains.

This memory work belongs in its own background workflow. The production path should stay focused on detecting client risk, not slow down while another workflow decides what to remember. The feedback-ingestion workflow can consider the complete run and the current instructions together instead of rewriting production behaviour in the middle of a client-risk investigation.

For example, feedback about planned maintenance might produce a proposed instruction not to flag an issue when the ticket history confirms that planned maintenance caused it.

[Current instructions here...]

Importantly, I designed this feedback-ingestion workflow so it can’t push its own proposal into production directly. Each proposal goes into LangSmith Context Hub, where a human compares the current and proposed agent instructions.

This controlled loop lets the production AI system improve through use. Real use produces decision-specific feedback. The background workflow turns that feedback into a proposed change to agent procedural memory. A person decides what becomes future production behaviour.

This way, the company improves the system without giving it permission to rewrite itself. Product and operations leaders can see the instructions that shape future runs, what changed and why, without opening the application code.

The business result

Within weeks of going live, this AI system surfaced a recurring network connectivity problem for a high-value client that had been hiding across months of ticket data. Every ticket had looked resolved on its own. Together, they showed a problem that still needed fixing, and Ntiva caught it while there was still time to act.

Ntiva’s Director of Product described the impact this way:

“Analyzing client churn risk has always been brutally manual for us, as it was hard to keep close tabs on many clients or risk signals. Sheila turned that into an AI system that surfaces repeated patterns over time, making churn risk visible in a way we couldn’t do before. Within weeks, it flagged a client issue that had been hiding in our ticket data for months.”