Measure AI Work at the Finish Line

A colleague produces a polished briefing in twelve minutes with an AI assistant. The old method took an hour. The result looks like a clear productivity gain—until another colleague spends forty minutes checking references, restores a missing exception and rewrites the recommendation because the draft used the wrong audience assumptions. The first person became faster. The work did not necessarily finish sooner.

This distinction matters as organisations decide whether AI is improving knowledge work. The easiest moment to measure is the dramatic one: a prompt goes in and a draft appears. But a useful unit of productivity is not generated text, code or slides. It is work that has reached an agreed standard, can be relied upon and has been accepted by whoever carries responsibility for it.

The thesis is simple: measure AI-assisted work at the finish line, not at the first draft. That means counting review, correction, integration and recovery as part of the job. It also means accepting that the same tool can accelerate one task and slow another without either result being contradictory.

Generation and verification are different queues

Most knowledge work contains at least two linked processes. The first produces a candidate: a memo, translation, analysis, design, code change or decision note. The second establishes whether that candidate is fit for use. It checks facts, tests assumptions, reconciles the output with local constraints, obtains approval and repairs errors.

Generative AI can make the first process much faster. It can also send more material into the second process. If review capacity does not grow with generation capacity, work accumulates at the handoff. The bottleneck moves rather than disappears.

This is not just a matter of hallucinated facts. AI can create subtler review work: a source that is real but does not support the claim, a correct rule applied to the wrong case, fluent code that ignores a repository convention, or a sensible recommendation optimised for a goal nobody selected. The reviewer must often reconstruct the task before deciding whether the answer is acceptable. Fluency lowers the cost of producing a candidate, not the cost of knowing that it is right.

Alkemata has argued that verifiability is a property of the route from a claim to inspectable evidence. The same logic applies to productivity. A workflow is not productive merely because its visible output arrives quickly; the route to accepted work must become shorter or more capable.

The evidence is deliberately untidy

The strongest counterargument is that AI really does save time in many settings. It should not be dismissed because some managers measure it badly. In a preregistered experiment published in Science in July 2023, 453 college-educated professionals completed occupation-specific writing tasks. Access to ChatGPT substantially reduced completion time and improved average quality for the tasks studied. The authors also found that performance differences between workers narrowed. Those are meaningful gains, not merely faster typing. Read the study.

A field study of 5,179 customer-support agents likewise found that access to a generative-AI assistant increased issues resolved per hour by an average of 14%, with much larger gains among novice and lower-skilled workers and minimal effects among the most experienced. The published analysis suggests that the tool helped distribute practices associated with stronger performers. Read the study and publication record.

Yet a randomised trial by Model Evaluation & Threat Research, published in July 2025, produced the opposite result in a different setting. Sixteen experienced open-source developers completed 246 real tasks in mature repositories they knew well. Allowing early-2025 AI tools increased measured completion time by 19%, even though the developers believed the tools had made them faster. Read the paper.

The number should not be universalised. It describes particular developers, tools, repositories and dates. METR itself reported in February 2026 that a later experiment was affected by selection and time-measurement problems, and that newer tools may have become more helpful, while the available data could not support a confident estimate. Read the methodological update.

Together, these studies do not say that AI is productive or unproductive. They show that the task boundary matters. Bounded writing and support tasks, where quality can be judged relatively quickly and good patterns can be reused, may gain greatly. Work embedded in a mature technical system can demand so much context, testing and correction that the assistance creates additional friction. Expertise also changes the baseline: a novice may gain access to a useful pattern, while an expert may spend time steering a system away from plausible but locally unsuitable choices.

Why first-draft metrics distort behaviour

What an organisation measures becomes a target. If teams are praised for the number of AI-generated documents, suggestions accepted or hours apparently saved, they have an incentive to move work across the first handoff. The uncounted burden lands on reviewers, maintainers, editors, legal teams, support staff or future users.

That distribution matters. The person who receives the productivity credit may not be the person who absorbs the verification cost. Junior staff may produce more while senior staff become a hidden correction service. Contractors may be paid for output volume while permanent staff carry integration and liability. A manager may see shorter drafting time while workers experience more fragmented review.

Human oversight is not a magic phrase that resolves this problem. It is a staffed activity with time, competence and authority requirements. The NIST AI Risk Management Framework playbook recommends comparing AI systems with human or simpler baselines, assessing generated outputs against ground truth and defining responsibilities and training for human reviewers. If review is essential, its capacity belongs in the system design and the productivity calculation.

This also changes the approval interface. A change preview, as Alkemata has proposed for acting agents, reduces review work when it shows the exact modification, uncertainty and recovery path. Provenance beside a claim is more useful than a detached list of links. Tests that run automatically are more useful than a request to “check carefully”. Better AI productivity often comes from making verification cheaper, not generation faster.

Run a twelve-task finish-line trial

A small team can test the claim without buying a monitoring platform. Choose twelve low-risk, recurring tasks that already have a recognisable acceptance point: short internal summaries, routine data transformations, test cases, draft replies or documentation updates. Exclude confidential, legally consequential or safety-critical work unless the organisation already has an authorised process for it.

Before starting, write down what “accepted” means for each task. It might require factual checks, a passing test suite, approval by a named role, conformity with a template or confirmation that the recipient can act on the result. This definition prevents the finish line moving after the result is seen.

Pair similar tasks and randomly assign one in each pair to the usual process and one to the AI-assisted process. For every task, record only four durations:

• preparation and generation;
• review and verification;
• correction and rework;
• waiting for acceptance or integration.

Also record the outcome: accepted without correction, accepted after correction, rejected, or reopened later. Do not score employees or capture prompts and keystrokes. The purpose is to inspect a workflow, not to turn a small experiment into worker surveillance.

At the end, compare median time to accepted work, not average drafting time. Then examine where the minutes moved. If generation fell by thirty minutes but review rose by twenty-five, the improvement is small and the reviewer may now be the bottleneck. If both generation and correction fall, the workflow probably found a good task-tool fit. If quality rises at the same total time, that may still be a valuable gain—but it should be named as capability, not disguised as speed.

Choose the task, not the slogan

The useful question is not whether an organisation is “AI-first”. It is which tasks benefit, for whom, under what acceptance standard and with whose review capacity. A first draft is an event. Accepted work is an outcome.

The next practical step is therefore modest: choose one recurring workflow and move its stopwatch to the real finish line. If AI saves time there, the gain will survive review. If it merely relocates effort, the diary will show where a better interface, narrower task or simpler tool is needed.