AI agent testing and evals: test suites, regression runs and answer scoring
We build a test suite for your agent from real conversations and risky scenarios. Every change to the prompt, model or tools runs through it before production. You see what got better and what broke.
What hurts
You changed the prompt to fix one answer, and three others broke. You found out from customers a week later. Your team cannot recheck the agent by hand after every change.
A new model is cheaper or faster, but nobody knows if it is worse on your tasks. The comparison is five examples and a gut feeling. Model decisions are made blind.
In a bank, a clinic or a call center, an agent error is expensive. The agent may promise too much, quote the wrong price or break a policy. You need proof that it behaves correctly, not just the confidence of a developer. Documented testing also helps with EU AI Act obligations.
What we build
- Review of the current agent and collection of risky scenarios
- Test cases from real chats and calls, anonymised
- Scoring criteria: facts, tone, policy, actions in systems
- LLM-as-judge calibrated against human labels
- Tool mocks so tests never create real orders
- Regression runs in CI before every release
- Model and prompt version comparison on the same suite
- A report of failures with recommended fixes
What you get
- A test suite in a format your team can extend
- Test runs set up in your CI or on our platform
- A quality report on the current agent with failure examples
- A comparison of models by quality, speed and cost
- A guide for adding new cases after incidents
How we measure the result
- Share of test cases the agent passes, per version
- Number of regressions caught before release, not by customers
- Agreement between LLM-as-judge scores and human scores
- Cost and time of one test run
Systems and stack
Industries
Questions
How much does agent testing cost?
From $5,000. The price depends on the number of scenarios, the agent type (text or voice), the number of tools to mock and the reporting requirements. Voice agents cost more to test, because cases are built from call recordings, not only text. You get a fixed estimate 2–3 days after discovery.
Do you only test agents you built?
No. We test agents on any platform, as long as there is an API or access to conversations. Voice agents too. Agent test suites are part of our Happ platform, so we work with this every day and know the usual traps.
What is LLM-as-judge and can we trust it?
It is a model that scores agent answers against clear criteria. It can be wrong too, so we check its scores on a sample labelled by people. For critical rules such as prices or amounts we use exact checks instead of model scoring.
What data do you need?
Conversation exports or call recordings, the agent prompt, a tool description and your policies. We anonymise personal data before work starts and sign an NDA and a GDPR DPA before access. Recordings are analysed by our own LLM, so data does not go to third-party models.
What happens after handover?
Your team runs the tests before every release. We can maintain the suite under an SLA: add cases after incidents, test new models and keep the tests in line with how the process changes.
Related services
More in this section
Describe the task. We reply within one business day, the first call is with the founder.
Describe the task. The first call is with the founder.
We reply within one business day. 30 minutes to tell you whether it is worth doing and what it costs.