An agent can return the expected result and still perform poorly. It may repeat a failed tool call, waste time and tokens, or take an execution path you wouldn't want repeated. The final answer can hide all of that.

Then you change the prompt, switch the model, or give it a different tool. The next run looks better. Did the change help, or did the agent just get an easier task?

We're building Softprobe to help teams answer that question. We want evaluation to lead to a useful decision about what to change, followed by evidence that the change helped. Our first step is making an agent's work available for inspection.

What is AI agent evaluation?

AI agent evaluation checks how well an agent performs a task against explicit criteria. It can assess the outcome, the actions taken, and the cost or time required. Teams use manual reviews and automated tests to find failures, compare versions, and check for regressions.[1]

The criteria matter. Finishing quickly is useful only if the agent did the required work. A correct final answer doesn't excuse an unauthorized action along the way. The team responsible for the task has to decide which outcomes and constraints count.

Observability and debugging help you investigate the result. Consider a repeated tool call: retrying after a temporary outage may be sensible; sending the same invalid request again deserves a closer look. The input, error, and surrounding steps help you distinguish the two.

A trace also has limits. An agent saying it updated a record is not proof that the right record changed. You may need to check the resulting state in another system.[1] If the evidence is missing, the conclusion should stay open.

How do you know whether the next version is better?

Compare the current and candidate versions on representative tasks using the same success criteria and comparable conditions. Include the failure you're trying to fix and cases that already work. Repeat runs where model variability could affect the result, and examine quality alongside cost and latency.[1]

This can start small. Keep the failed run, write down what should have happened, and identify a change to test. A prompt change that stops a retry loop is worth investigating. It still needs to complete the task and avoid breaking other cases.

Keep the comparison result with the change, whether that's in a ticket or a pull request. The next person should be able to see why you made it and what remains uncertain. After release, check whether the improvement holds on new work.

The point is to make the next engineering decision easier. An evaluation that produces a score without helping you understand what to do next leaves too much of the job unfinished.

What we're launching

Today, Softprobe captures OpenCode Sessions and lets you inspect their details. A Session is the captured record of one agent run. You can start with a run you want to understand and review what happened before deciding what to investigate further.

Our free plan includes unlimited Session ingestion, unlimited agents, and 365 days of Session history.

Next are LangChain support and a limited first set of Agent Diagnosis checks. We're taking a deterministic-first approach: start with observable execution problems, and link each finding to its supporting evidence. These checks will not, on their own, prove business correctness.

Our longer-term direction is to connect evaluation evidence, human decisions, changes, and controlled verification.

We think of that longer-term workflow as performance reviews for AI agents. Repeated over time, it could support a broader practice of AI agent performance management. It starts with a much more immediate job: understand the run in front of you.

We want teams to spend less time reconstructing failures and more time improving what runs next.

Start free and inspect your first Session.

Sources

[1] https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents — Anthropic: Demystifying evals for AI agents