Why enterprise AI stalls at the demo: Prove one workflow before scaling

Pick a workflow tied to a customer pain point, define permissions, accountability, baselines, and evaluation before building, and use real data for evidence before expanding. What a pilot leaves behind—context, metrics, and operating know-how—makes the next workflow easier to start.

By Mark Ajzenstadt09-05 · 16 min read
🎧 Listen to the podcast0:00 / --:--
0:00 / --:--
Deep dive · XiaoHu Explains
One-minute overview
  • Enterprise AI gets stuck between demo and production not because models can't answer, but because no one picked a workflow with clear boundaries, agreed on evidence in advance, or tied pilot results to a concrete decision.
  • The first workflow should come from where customers wait, complain, or churn. Candidates that lack real users, real data, a signable baseline, or a clear owner should be dropped before development begins.
  • Set the baseline, golden set, and failure criteria before any agent is built. Then run the new approach in parallel with the old one for two to four weeks, measuring throughput, quality, and economics before deciding to continue, adjust, stop, or expand.
  • Permissions and accountability must be settled before development. Only after evidence supports expansion should you harden production deployment and handoff. The context, evaluations, and operational lessons left behind mean the next workflow doesn't start from zero.

The most common illusion companies have about AI is confusing "can be demonstrated" with "is worth deploying." Over six weeks, Mark Ajzenstadt spoke with 20 operating partners at private equity firms and found that while most portfolio companies already use AI to draft emails and summarize meeting notes, few can articulate how to move it into workflows that actually touch customers and business outcomes.

This gap operates on two timescales. A polished demo can take a week to build, but proving production quality can take six months. The problem usually isn't whether the model can answer; it's that companies skip three foundational steps: they don't first choose a process with clear boundaries, fail to agree on what counts as success before starting development, and never set up pilot results to feed into an explicit go/no-go decision.

Ajzenstadt is the founder of Limestone, an AI delivery firm. His "AI transformation loop" starts with a customer pain point: pick a process with clean boundaries, agree on baseline metrics and evaluation methods before development, validate a minimal slice against real data, and only then decide whether to scale up investment. Data access and ownership must be defined from day one. Only after the evidence supports expansion should the team refine production deployment and operational handover. Each time the loop runs, the accumulated context, evaluation suite, infrastructure, and business knowledge lower the cost of the next initiative.

Step one: find a viable process where customers are waiting

Asking business teams "what AI use cases do you have?" rarely yields good answers. Operators spend their days serving customers, maintaining relationships, and handling exceptions. Customers, meanwhile, don't proactively ask for AI next quarter. Asking frontline staff to pause their work and invent an AI roadmap from thin air is the wrong starting point.

A more reliable approach is to work backwards from the customer: identify where customers wait, complain, or churn, then examine the work, data, evidence, and ownership behind that experience. The original article breaks this examination into five lenses.

Five lenses for screening candidate workflows

  1. Customer interaction: Look for places that change customer outcomes. This could mean offering insights they didn't have before, compressing multi-day waits into minutes, or improving the product between releases. Ajzenstadt specifically notes that onboarding moments—account opening, document review, new-customer activation—are often undervalued; when customers can't use a service quickly, that's a signal to investigate further.
  2. Underlying work: Break steps into categories: deterministic, judgment-intensive, exception-driven, and relationship-dependent. Machines should handle deterministic steps and prepare materials for judgment calls; humans remain responsible for decisions, approvals, and exceptions. If most of the work depends on complex judgment or client relationships, save it for a later pilot and choose processes where acceptance criteria are easier to define up front.
  3. Context: Verify where facts live, who owns them, what fields actually mean, and what sensitivity levels apply. If no one internally can explain what the data means, defining the data is the first job.
  4. Evidence: Check whether the company can currently measure work units, volume, cycle time, hands-on time, rework, exception rates, and unit costs. Without a current baseline, there's no way to prove what changed after the pilot.
  5. Ownership: Pre-assign a process owner, data owner, and executive sponsor. If no one will take over running the process after the pilot, and no one is willing to make a go/no-go decision based on evidence, the process is not viable.

Companies should identify 3–5 candidate processes, rank them by value, feasibility, risk, and reuse potential, and then choose just one to start. Anything without real users, real data, a signable baseline, or clear ownership should be eliminated before development begins. Goals like "redesign the entire organization" that can't be validated in a short cycle also fail the test.

Once selected, walk through the process with the people who actually do the work. Collect representative cases, exceptions, permissions, and undocumented expert judgment. Interview first, then inventory technology; before reading code, ask who owns each dataset and what each field represents. Ajzenstadt also requires writing down a single business outcome and explicit non-goals, because teams easily expand a small pilot into a full system by week two.

Step two: agree on baseline and evaluation criteria before building the agent

The critical part of Ajzenstadt's ten-step method isn't the number ten; it's the ordering. If you pick metrics after seeing model outputs, you'll naturally favor whatever the model does well. If you build the baseline only after the pilot ends, you can't tell where changes came from. Baseline metrics, a golden set, and critical failure categories must exist before the agent does.

Both sides must agree on current workload, cycle times, hands-on processing time, quality and rework rates, exception rates, operating costs, and the measurement window. How things are done today, what performance level is being achieved, and how improvement will be calculated all need to be settled before the trial run.

The "golden set" here means a batch of real tasks whose results have been confirmed by the customer's own experts. Evaluation must also cover the failures the business cannot tolerate—wrong numbers, protected data leaks, or confident behavior when data is missing. Only when both successes and critical failures are baked into the criteria do trial results qualify for decision-making.

This preparation requires involvement from people inside the company. Under Limestone's "proof month" structure, executive sponsors dedicate 30 minutes a week and make decisions at checkpoints, process owners spend 2–4 hours weekly on walkthroughs and acceptance reviews, and data owners commit 1–3 hours at the start. A vendor can build the system, but business goals, field definitions, and correct answers must be confirmed by the company itself.

This also defines the budget boundary. Limestone reports its delivery mix as one embedded developer with an agent plus a part-time architect at US$15,000–20,000 per month; infrastructure for one operational agent project handling four groups of customer data runs about US$250 per month. These are different figures: the infrastructure bill doesn't include development, acceptance, or organizational handover costs. Their policy of no up-front diagnostic fees, monthly billing, and no payment if the first month doesn't satisfy is that vendor's commercial commitment—it can't substitute for the customer's own acceptance criteria.

Ajzenstadt adds one more discipline to the proof month: every activity must yield a decision, a tested hypothesis, a working increment, or a reusable asset. Anything that produces none of those isn't worth the team's time. This keeps walkthroughs, meetings, and development focused on actual results.

Step three: run a 2–4 week parallel trial and let evidence make the call

A pilot shouldn't start by making final decisions for people, nor should it write directly to systems of record. A safer minimal slice looks like this: the agent recommends, a named person decides, real data is used, the process runs in parallel with the old way, and every execution leaves a trace others can inspect.

These 2–4 weeks must answer three distinct questions at once: is it faster, is it better, and is it worth it once all costs are counted?

Metric What to actually look at What it can't be reduced to
Throughput Work completed per cycle, cycle time, hands-on time "Running more" alone doesn't prove better delivery
Quality First-pass rate, rework, human coverage, incidents, critical failure categories Average accuracy can't hide tail risks like data leaks or false confidence when data is missing
Economics Unit operating cost, plus verified value of capacity, rework avoided, incremental output, and risk-adjusted losses prevented Hours saved don't automatically convert to cash; subtract cloud, model, support, and organizational change costs, and report as a range

The article cites Faros AI's telemetry on 22,000 developers from 2026, noting that while task throughput rose 33.7%, defects per developer increased 54% and incidents per pull request rose 242.7%. Those numbers don't predict any single company's outcome, but they make it clear that speed and quality must be measured separately.

The pilot has four exits: continue, adjust, stop, or scale. If results don't clear the baseline agreed up front, stopping at four weeks preserves the budget for later. Adjustments should modify the implementation or the process approach, then be verified against pre-agreed criteria; lowering the bar when results disappoint isn't acceptable. If the scope of the task itself changes, the verification conditions need to be redefined—old and new results can't be treated as one comparison.

The figure places the trial run and all three measurement categories in the same phase: evidence accumulates continuously during execution, and the business and delivery teams together decide where the next investment goes.

How pilots guide the next step

Evidence before investment

Define permissions and owners at the start. A pilot has four exits.

  1. Choose one workflow

    Trace customer pain points to real users, data, owners, and explicit non-goals.

    Output: a bounded workflow and one business outcome
  2. Set baselines and tests

    Agree on current performance and the measurement window. Customer experts confirm the golden set and critical failure criteria.

    Output: acceptance criteria set before development
  3. Run and measure together

    Run alongside the old process for 2–4 weeks. The agent recommends; a human decides. No system-of-record writes yet.

    Output: throughput, quality, economics, and readable execution logs

Decide together against agreed criteria

Go on
Keep validating within scope and gathering evidence.
Adjust
Revise the implementation or workflow, then rerun. Poor results do not justify a lower bar.
Stop
End the pilot, retain evidence, and stop further investment.
Scale
Evidence supports more investment: harden production deployment and hand over operations.Continuous evaluation · reliable hosting · fallbacks and cost caps · operator training

After scaling and handover, reuse context, evaluations, and operating experience in the next workflow.

Measure all three dimensions throughout the trial. Four decisions lead to different actions; only one leads to production expansion.

After evidence supports expansion, harden production deployment and the operational handover

An agent that clears the pilot still needs reliable scheduling, observability, integration, failure fallbacks, cost ceilings, and continuous evaluation before it can operate daily. Every change requires re-evaluation, and nothing should depend on one engineer's laptop. The original article's seven-level adoption ladder describes the capabilities required to move from a personal tool to an organizational system.

Moving up the ladder doesn't mean basic permission management can wait. Ajzenstadt is explicit: knowledge structures need role-based access control from day one; data owners, sensitivity levels, and pilot leaders are identified before development starts. The extra hardening comes after a pilot passes—it's about production operations and scale, not about deferring data and permission boundaries.

Seven-level AI adoption ladder

Level Added capability Typical state
1 Tool adoption People start AI tools one by one; automation disappears when they take leave or close their laptops
2 Context layer Department knowledge managed in domains, queried across functions through defined interfaces, reducing semantic noise
3 Evaluation, routing, quality gates Define standards and routing first, then choose models; swapping models doesn't require rebuilding the whole system
4 Hosted execution off personal machines Triggered by schedule or conditions without someone logging in daily
5 Governance Knowledge structures have role-based permissions; agents have lifecycles; unused or underperforming capabilities get retired
6 Agent-to-agent handover One agent's output feeds the next, with quality gates in between
7 Semi- or fully automated workflows Agents handle routine tasks at scale; people focus on exceptions and high-risk judgment

Many companies are still at level 1 yet buying capabilities from level 7. The real bottleneck is usually level 4: moving scripts off personal computers and onto services that trigger reliably, log their activity, handle failures, and have clear ownership. Without stable hosted services, multi-agent handoffs can't become reliable daily operations either.

Governance also needs concrete retirement rules. Ajzenstadt suggests giving every agent a lifespan—enforce lifecycle management once tool count passes 100, retire anything that hasn't run in 90 days, and retire anything that falls below its evaluation threshold. These are management rules he proposes; the point is that someone must own decommissioning stale capabilities so obsolete agents don't keep valid credentials.

Finally, train the operators, hand over a runbook, and embed the capability into existing workflows. People's roles shift from starting tools one by one to accepting results, exercising judgment, and handling exceptions; every expert correction flows back into the golden set. Naming an owner is just the beginning—the operational handover is complete only when the team actually uses, evaluates, and maintains the capability.

Three cases show what the loop can do—and what it can't prove

Ajzenstadt offers three customer cases. They illustrate how the loop operates in practice but they can't by themselves prove the method will produce similar returns at every company: clients aren't named, and the original baselines, full evaluation sets, and third-party review materials haven't been published.

Scenario How the loop ran Author-reported results What can't be extrapolated
Medical billing denial queries Built a golden set from customer's real questions and expert answers; customer's product tech lead accepted after six weeks 59/60 on first deployment, 15/15 on denial cases; 97% accuracy maintained after online database schema migration, key questions traceable to the cent Agent only suggests, human decides, nothing writes to the system of record; evaluation set not public
Logistics platform invoice rebuild First week had no code; every feature spec'd first, human gate on every merge In 90 days, two engineers merged 122 PRs, 104 with fewer than 5 review comments, delivered 50% earlier than customer estimate Software has relatively clear inputs and acceptance gates; evidence depth exceeds most operational work
Hardware manufacturer pilot Four-week parallel trial against another firm roughly ten times Limestone's size Customer CTO judged Limestone superior, planned 50/50 joint delivery first, then 80/20 across ten other Scrum teams No standardized scoring published; "performed better" is the customer CTO's assessment relayed by the author

The article also admits plainly that no company transforms itself in one to three months. Software engineering has the deepest evidence because inputs, tests, and human gates are relatively well defined. Outside software and simple repetitive processes, there are no universally "obvious" use cases—every judgment- or relationship-intensive workflow has to establish its own baseline, golden set, and failure criteria from scratch.

These limitations don't weaken the loop; they explain its purpose. It doesn't promise answers in advance. It gives the company a controlled period to gather enough evidence to decide whether to keep investing.

The real compounding effect: the next process doesn't start from zero

The first process pays off in more than just possible savings. When the loop is run completely, the company keeps four kinds of assets that carry straight into the next pilot:

  • Context layer: data definitions, owners, sensitivity levels, and cross-department boundaries are already mapped.
  • Evaluation framework: the golden set, failure categories, routing logic, and quality gates can be extended.
  • Hosted infrastructure: scheduling, logging, fallbacks, security review, and cost ceilings don't need to be rebuilt from nothing.
  • Business ownership: runbooks, approval relationships, and owners are already embedded in daily operations; expert corrections keep flowing back into the golden set.

Models will change. Today's popular agent framework may be obsolete next quarter. But workflow knowledge, evaluation standards, hosting capabilities, and operational responsibility stay with the company. Because those assets persist, the second process starts from where the first one ended.

Find 3–5 candidate processes where customers are waiting, complaining, or churning, and use the five lenses to pick one. Define permissions, responsibility, baselines, and evaluation before development. Run a 2–4 week parallel trial on real data, and let throughput, quality, and economics jointly determine whether to continue, adjust, stop, or scale. Once the evidence supports expansion, refine production operations and handover, and carry the assets from each proven process into the next.

Source
The AI transformation loopMark Ajzenstadt·2026-09-03·View primary source
Site note
Mark Ajzenstadt is also the founder of Limestone, an AI delivery provider. Customer cases, cost figures, and evaluation numbers in this article are self-reported.
↑ End of article
✓ This article is free to read in full

Members can copy or download the full Markdown, and let their AI subscribe and ask.

Read the original ↗
📮 New explainers, straight to your inbox

Found this useful? Leave your email — every new explainer, the moment it's out. Unsubscribe anytime.