Before hiring an AI Agent developer: plan the pilot and acceptance tests

A useful pilot uses representative inputs, verifiable outcomes and agreed criteria for proceeding, revising or stopping. Talvyn Labs starts AI Product and Agent development with the business workflow and defines evaluation criteria with the people responsible for the work.

Navigating the Jagged Technological Frontier, published in 2026, tested 758 consultants. AI improved speed and quality on tasks within its capabilities, but reduced correctness on a different task beyond the tested system’s capabilities.

Those findings concern the tasks and model studied, not a score for every current model. For a business, they support assessing quality alongside speed for each proposed use.

Choose one observable goal: a reviewed document prepared sooner, an enquiry routed correctly or complete information ready for approval. “More AI use across the company” is not enough to show whether the work improved.

Name the user, the input and the result a reviewer must accept. In an illustrative service-team pilot, an agent retrieves manuals and prepares an answer with sources. Staff review and send it. That differs from authorising a system to send messages or change a job's status.

Start with authorised read access and activity records. Changes to records, commercial terms and customer commitments require human approval. List the tools the agent may use and the actions outside its scope.

Assign a reviewer who understands the work and can participate. Without someone able to establish whether an output is correct, a finished interface cannot establish operational readiness.

Include cases where the correct outcome is “insufficient information.” Scoring only how many questions receive an answer can reward guessing over an appropriate stop.

For Thai and English use, review results separately by language. Include actual abbreviations, product names and mixed-language requests. A combined average can hide weak performance in one language.

Establish a baseline for the current process before comparison. Use work of similar difficulty and record relevant conditions, such as legible documents or missing fields. Comparing easy AI examples with difficult manual cases produces a misleading result.

Measure preparation, waiting, review, correction and handoff. An answer generated quickly may save little time if staff must independently reconstruct its sources. Record both total time and the steps consuming it so the team knows what to improve.

Track quality alongside time: accepted work, work needing corrections, work returned to the original process and the reason for each outcome. Separate errors that alter meaning or lead to a wrong decision from wording changes. A single score can conceal a commercially significant failure.

Include service usage, retries, data preparation, review time and operating effort in the cost assessment. Relate those costs to successfully completed, accepted work. Pilot results apply to the tested scope; they are not a promise about every workflow in the business.

Suppose one case involves gathering information, preparing a draft and reviewing and recording the result. The following figures are hypothetical teaching examples, not customer results or a Talvyn benchmark.

The example saves four minutes per qualifying case. Across 100 such cases, that returns 400 minutes, or six hours and forty minutes of capacity. The next question is what the team can actually do with that time.

If salaries remain unchanged, saved time does not automatically reduce cash spending. It may help clear a backlog, reduce overtime or create room for sales follow-up. Each outcome needs its own measure.

Include cases where AI makes mistakes and review takes longer. Reporting only the smooth cases would misrepresent the workload the business actually handles.

Review at least four dimensions, using measures suited to the task:

• Time: Human effort and elapsed time until completion.
• Quality: Whether the result meets the required standard, including corrections and rework.
• Cost: System and AI usage, connections, data maintenance and review effort.
• Business outcome: Whether the change improves the backlog, customer response or capacity to take on work.

The OECD’s SME report distinguishes employee performance, workload and revenue outcomes. These should not collapse into one claim of “improvement.” Its evidence is survey-based, which also needs distinguishing from experimental results.

If revenue is the goal, follow the relevant outcome and account for other conditions such as enquiry volume, promotions and lead quality. A quicker response is one step in a sales journey; it does not establish that AI caused higher sales.

Use one set of examples to improve the system and reserve another set for evaluation. Testing only the cases used during development can overstate how well the product will handle new work.

Record the data, configuration and model versions used in each evaluation. When a change improves one failure, check that previously successful cases still work. A demonstration of the latest fix is not evidence that the whole workflow remains dependable.

Use inputs the business is authorised to share, minimise unnecessary detail and restrict access. Do not circulate confidential customer documents to several prospective suppliers merely to obtain quick demonstrations. If synthetic inputs are used, record the limits of what they establish about real data.

Proceed when the agreed work succeeds, quality meets the business owner's criteria, permissions and approvals operate correctly, and someone owns ongoing support. Document the limitations the business has accepted.

Revise and retest when the cause is understood and can be addressed within scope. Examples include inconsistent catalogue records or a review screen that makes corrections awkward. Specify the change and the cases that will demonstrate improvement.

Stop or change approach when the system exposes unauthorised data, performs actions without required approval or leaves staff rebuilding nearly all the useful output. Those failures should not be averaged away by correct answers elsewhere.

There is no universal accuracy threshold for every business workflow. Suggesting a document and preparing commercial information have different consequences when wrong. Agree on criteria based on the intended use and the impact of errors. The NIST AI Risk Management Framework offers voluntary risk-management guidance; it does not certify that an individual system is safe or has passed evaluation.

Agree what success would mean before testing. One example is a lower total task time while meeting the required quality standard and keeping maintenance within the team’s capacity. Set thresholds around the task and its consequences, rather than applying one percentage to every business.

If only some work benefits, expand within those conditions first. Frequent corrections elsewhere may call for better information, a revised review process or a narrower AI responsibility, followed by another assessment.

Ask for a clear scope and permission model, evaluated examples, known limitations, activity records, failure-handling procedures and named responsibilities for the connected systems. The handover should help the team decide what can expand and what must remain unavailable.

It should also distinguish defects from new requirements. A failure to follow an agreed approval rule is different from adding another department or data source. Keeping that distinction visible helps prevent a pilot from becoming an undefined implementation.

For a scope discussion with Talvyn Labs, bring representative time-consuming work, examples the team considers acceptable and constraints in existing systems. Start with buying AI software or developing a custom agent if the delivery approach is unresolved. If users need assistance within an existing product, see adding AI features to existing software.

1. Dell’Acqua et al. (2026): Navigating the Jagged Technological Frontier, Organization Science · Full text hosted by HBS.
2. OECD (2025): Generative AI and the SME Workforce.
3. NIST (2024): Generative Artificial Intelligence Profile, NIST AI 600-1.
4. Talvyn Custom CRM demo, used for the interface illustration rather than experimental data.