The previous part of this chapter ended with the most critical of all questions, namely how you know that an agent reliably fulfills its task. Reliability is not a property one asserts; it has to be demonstrated. Only through measurement does an agent become a tool that reliably fulfills a specific task.

Here, a property pays off that runs as a guiding idea through the second half of this series: the agent runs locally and within the process, it is not an external, closed system that cannot be observed, or that can be observed only to a limited extent. An embedded agent can be instrumented in such a way that every relevant step becomes visible, not just its result. Most agent tools deliver the agent, but no built-in function for systematically measuring and refining its reliability. Precisely that belongs here to the runtime.
 

An Evaluation Framework That Runs Along

From this observability a measuring instrument emerges. An evaluation framework runs defined test scenarios, i.e. typical tasks from the simple case through multi-stage plans all the way to error recovery, and lets the agent work through them one by one. For each run, a structured observation log emerges, which is evaluated by a checking agent in order to judge the quality of the result and of the path leading to it.

The procedure is a loop. Before the run, it is determined which aspects are observed and in what depth. After the run, the logs are evaluated against success criteria and check questions defined in advance. These include questions such as: was the task correctly understood? Was the appropriate tool chosen on the first attempt? Were there unnecessary iterations?

From the findings, internal prompts, tool guidance, context tailoring, or code are refined; then the run is repeated. Reliability therefore does not arise from a one-time test, but from this empirical loop: measure, refine, measure again. The same scenarios can also be used to compare the agent’s behavior cleanly under different language models.

This is not mere dry theory: it is precisely with this evaluation framework that PROSTEP evaluates its own agent. The PROSTEP AI Workbench is measured against these scenarios and refined round after round. The methodology described here is thus the tool with which we further develop our own agent architecture. Important for productive operation is that the logging required for this can be switched off, so that no measurement overhead arises in live operation.

What Is Measured

What is measured is not only whether a plausible answer stands at the end. What is observed is the entire causal chain of the behavior, with around a dozen aspects, among them:

• Task understanding: did the agent break down the task correctly, or misunderstand a requirement?
• Tool choice: did it choose the appropriate tool, or correct a wrong choice in a controlled way?
• Result interpretation: did it read a tool result correctly, or overlook an error?
• Planning, decision points, dead ends: where did it replan, where did it waste iterations?
• Result quality: does the result fulfill the criteria defined in advance?

Added to this are sober operating quantities per run such as iterations, tool calls, tokens, runtime. Only these step-level observations show whether an agent works reliably or has merely been lucky in the individual case. And only through this can it be said, with any solidity, where an agent works stably and where not. On basic tasks, many systems lie close together. The difference only shows itself where the architecture plays out its particular capabilities, for example on tasks that have to retain state, decisions, and intermediate results across many steps, or with tools that process more than text.
 

Where APL Is the Right and the Wrong Answer

For the sake of honesty, the conclusion of this chapter should also include the acknowledgment that APL is not the right choice for everything. Where a task is simply rule-based, classical script programming suffices. A compiled process language with AI steps would be over-dimensioned. And where a task is open and exploratory, without a recurring form and without any claim to reproducibility, an externally operated general-purpose agent can be the more pragmatic answer.

APL pays off precisely in between: for processes that have to deliver the same thing productively, repeatably, and verifiably, and that at the same time contain genuine cognitive steps. That is not a narrow special case, but the wide field of industry-capable AI applications. But it is a specific field, and naming that clearly is itself part of industry capability.

The Chapter Closes

The aspiration of APL is to embed AI in such a way that its work can be described, bounded, observed, and improved: a deterministic backbone, selectively deployed AI, an agent that works within the application, and a reliability that can be measured. How these principles function in real applications, e.g. in a traceability assistant or in contract and requirements specification analysis, is shown by the third chapter of this series. It brings the aspects together from the product perspective.