Journal · Reliable deployment

The cost of a successfully completed AI task.

The useful unit is the accepted business outcome. Retries, external tools, review and failed work belong in its cost, even when the model call looks cheap.

Conclusion

The useful unit is the accepted business outcome. Retries, external tools, review and failed work belong in its cost, even when the model call looks cheap.

A ledger for one accepted outcome

  1. Define the assigned taskSpecify the business object, deadline and acceptance rule before execution.
  2. Accumulate every attemptAttach model, retrieval and tool costs to the same task identifier; retries add cost.
  3. Record interventionAdd review, corrections and delivery support; preserve the cost of failed work.
  4. Verify the final stateCheck whether the business result meets the agreed standard and within what time.
  5. Divide cohort cost by accepted tasksCount successful tasks once and show failed and abandoned tasks alongside the ratio.
  6. Stress-test sustainabilityAdd setup allocation and error-loss sensitivities before comparing price with cost.
Original accounting flow. The article's numerical comparison is hypothetical; this flow specifies what a real measurement would need to include.

A cheap attempt can still be expensive work

An AI task is economically successful when the customer receives an acceptable outcome at a sustainable delivery cost. My conclusion is that cost per accepted task deserves more attention than cost per model call. A falling inference bill can coexist with expensive retries, manual corrections and unresolved work. The denominator matters as much as the infrastructure price.

For a hypothetical invoice assistant, producing extracted fields is an intermediate result. The business may need a correctly matched invoice, an authorized posting and an auditable exception when something is missing. If a reviewer repairs the vendor identity and purchase-order match, the workflow might still be valuable. Its economics must include that review rather than assigning the entire outcome to automation.

Write the acceptance rule before calculating the ratio

Anthropic's evaluation guide distinguishes an agent's transcript from the resulting environment state. That distinction supplies a useful discipline: check what actually changed. An assistant saying a booking is complete does not establish that the reservation exists. For the invoice task, inspect the posted record and supporting evidence rather than the reassuring final message.

Commercial definitions also vary. Intercom's Fin documentation distinguishes confirmed and assumed resolutions, and treats a configured Procedure handoff as a billable outcome. It also describes deducting a resolution if the customer returns to that conversation seeking more help. These are vendor billing rules, not a universal measure of task quality. They illustrate why an investor should read the outcome definition before comparing reported volumes.

Count the work that failed as well as the work that passed

For an observed cohort, I would calculate total attributable delivery cost divided by accepted completed tasks. The numerator includes inference across all attempts, paid tools, retrieval infrastructure, human review, correction and routine delivery support. Failed attempts remain in that numerator. Count each accepted business task once, even if the system needed several model calls or retries. Report the completion rate beside the ratio so abandonment stays visible.

Implementation deserves a separate view. Show the initial setup bill and a sensitivity analysis allocating it over a stated customer lifetime or task volume. Do not quietly spread it over an optimistic future denominator. Product research and general sales overhead belong in broader company economics; keeping them separate from delivery cost makes the comparison interpretable without making those expenses disappear.

A hypothetical example changes the apparent winner

Consider two imaginary systems each assigned 1,000 tasks. System A spends $200 on models and tools, $600 on review and $100 on delivery support. It completes 750 tasks to the agreed standard. Its $900 delivery bill means $1.20 per accepted task. System B spends $350 on models and tools, $150 on review and $100 on support, completing 900 tasks. Its $600 bill means about $0.67 per accepted task. These are illustrative inputs, not observed results or supplier quotes.

System B has the higher machine bill and the lower completed-task cost. That conclusion depends on consistent task difficulty, acceptance rules and labor costing. Neither figure includes initial implementation, selling expenses or expected losses from material errors. Those omissions must remain explicit; a serious misposting can overwhelm savings across many routine invoices.

Averages hide the cases that consume the team

Segment performance by task difficulty, customer configuration and exception type. A tidy average can be driven by easy cases while a small difficult group absorbs expert attention. Report the proportion completed without intervention, assisted completion and failure separately. Measure elapsed time as well as labor minutes: a two-minute review after a two-day queue may fail the buyer's deadline.

There is practical infrastructure for explicit intervention. LangGraph documents interrupts that pause execution for external input and resume from saved state. This establishes an implementation pattern, not low review costs. I would propose logging every intervention against the task, then comparing a representative cohort with the customer's current process. That comparison remains a proposed evaluation here.

Owning a model is a hypothesis about total cost

The strongest objection is that early unit economics are unstable: model prices, routing and product design can change quickly. Rejecting a pre-seed product because its first implementation is labor intensive could miss substantial learning. I agree, but a credible improvement path should name the expensive failure class and how it will disappear.

Sequoia's own-intelligence guide presents cost, speed and proprietary data as possible reasons to control more of the stack, while explicitly allowing frontier APIs to remain appropriate. My inference is to compare architectures against the same accepted outcomes and include the engineering burden of ownership. A cheaper custom model is not automatically a cheaper operating system.

Underwrite the route to less expensive completion

I would favor teams that can reconcile task traces, intervention records and customer acceptance with the delivery ledger. At pre-seed, the evidence may be a transparent small sample and a testable explanation of its expensive exceptions. At seed, repeated cohorts should show whether more volume reduces delivery effort or merely recruits more operators.

The investment question becomes: which identifiable change lowers the cost of acceptable work without shifting risk to the customer? Better routing, clearer exceptions or fewer integration failures may matter more than another inference discount. The next milestone should specify the task class, acceptance rule and cost component it intends to improve.

Source register

  1. Anthropic: Demystifying evals for AI agents · primary-technical-research
  2. Intercom: Fin AI Agent outcomes · primary-product-documentation
  3. LangGraph: Interrupts · primary-product-documentation
  4. Sequoia: Own Your Intelligence: A How-To Guide · investor-framework

← Back to journal