Our Framework for Implementing Agentic AI in the Enterprise
Coding agents now write, deploy, and fix code faster than most organizations can decide what they want built. This means that the hard work, the real work, has moved upstream. That work is articulating what the business needs clearly enough to build it. In a world where articulating something is only a few minutes or hours away from seeing it fully built, the articulation itself becomes both development and quality control.
Most leaders I talk to have sat through some version of this meeting:
A delivery team built an AI agent in 2 weeks, and the demo went well, as demos do. Then someone digs a bit deeper. How often does this agent hallucinate? What is its cost per task? Would hosting it on a different platform affect the tokenomics? The implementation team glances at each other and formulates a cautious response. LLMs are probabilistic machines. Token costs will continue to fluctuate. This is the most state-of-the-art architecture in {insert current month of current year here}… Etcetera, etcetera. Now it’s time for a Go-Live decision, and agents need access to production systems, pretty please!
The problem is that the team spent its 2 weeks building and almost none of it articulating or proving. Nobody wrote down what good looks like, so nobody can say whether the agent is good. To address this, my team at Praecipio works in a loop we call Specify, Predict, Evaluate.
How does the loop work?
Specify starts with the stakeholders who own the process we are augmenting/implementing. We record conversations with business owners and executives, then use Claude to turn the transcripts into a spec that defines business and functional requirements. Nobody picks a model or a platform yet. Often the most useful thing a spec produces is a list of disagreements. In accounts payable, the AP manager may want small price variances approved automatically, while the controller wants every variance in front of a human for the first quarter. Better to discover that kind of misalignment in conversation than in production.
Predict is the step most tempting to skip. But it’s crucial in the era of AI. Today, someone on the delivery team usually sits beside the coding agent, approving its steps and steering it with instinct and taste. That judgment matters (a lot!), but nobody records it, so nobody knows whether that human judgment is getting better or getting worse over time. Therefore, as a crucial step even before running evals, our team proposes an architecture and we document how we expect that architecture to perform. These are often “What if?” questions phrased as predictive statements. For example: What if we gave the agent fewer tools and turned on prompt caching? Becomes: With fewer tools and prompt caching enabled, this workflow will complete 90% of tasks correctly in under 6 seconds. The statement form makes the prediction testable.
Evaluate is where reality joins the loop. An eval is a repeatable test that runs the agent against a fixed set of tasks and scores the quality, response time, and cost of its work - often measuring the latter in terms of tokens, compute, and/or infrastructure uptime. Most teams use evals to compare one configuration against another. But we think it’s also important to compare results against our own predictions over time, so that we can measure not only the AI’s performance in real-world environments, but also measure our own instincts and judgment against reality as AI systems continue evolving. So: an eval measures the agent, and the prediction measures the judgment of the people who built the agent.
Our learnings then go back into the spec, and the loop starts again with a new round of predictions.
Over a few rounds, the gap between predicted and measured performance starts to shrink, which shows the team understands the workflow it is building. When the gap grows after a model release or a change in a connected system, it is time to run the loop again, before users notice.
SPE in Practice
Early in an engagement, the usual sticking point is access. With no production credentials, no test users, and sometimes lacking a charter/SOW for a full implementation, there is nothing real to run an eval against. To work within these constraints, we sometimes leverage a technique we call “synthetic evals.” In synthetic evals, we use AI models to role-play the APIs of enterprise systems for the purpose of exploring architectural options. One model reads your ticketing system’s real API documentation and answers requests the way that system would, error messages included. Another plays a frustrated customer who leaves out half the details. By keeping each of these in their own subagents, with system-based personas, the agentic workflow we are designing is able to “encounter” a very realistic simulation of the use case the client wants built.
Everyone gets something positive out of this innovative approach. The delivery team gets several rounds of the loop and identifies gotchas and ahas before ever touching a live system. The customer gets to see how a proposed workflow handles its own rules and edge cases, with real quality and cost numbers, before it grants access or goes live. Of course, latency and error metrics are only estimates until we can eval the real deal, but this way, by the time we request systems access, everyone can agree to a design that makes sense.
All of this depends on being able to measure the AI, and that is where AI wrapper products struggle. Many chatbots and AI features built into existing platforms do not let you measure each task’s cost and response time, or wrap your own tests around them, so the loop cannot close. The vendor chooses the model, tools, and prompts, leaving your team nothing to predict. Models change on the vendor’s schedule, and quality drops go unmeasured until users complain. When the tool is replaced, nothing carries forward. These tools can help individual productivity, but a workflow the business depends on should run on AI it can measure.
What does the business get out of it?
For our business partners, this SPE loop changes how an AI project feels from their side of the table. Disagreements between stakeholders surface in the spec during the first week. Architecture decisions arrive with numbers for quality, speed, and cost. After go-live, every model release or system change triggers a fresh eval, so the workflow keeps performing better and better over time. And the judgment, taste, and human element is measured and tuned to facilitate an exceptionally clear and helpful handoff process.
If your team is building agentic AI and wants to know it’s actually working before it touches production, we’d love to talk. Praecipio helps organizations put a measurable loop around AI delivery, so quality and cost are known quantities, not surprises at go-live.
This blog was written by Clayton Chancey, Director of Professional Services at Praecipio.
