TL;DR: Define meaningful success, create a comparison, run the test or rollout for a specified time, and analyze the data.

If I were to ask ten people what percentage of AI pilots succeed, I’d get ten different answers: 95% fail, 80% show no value, 42% get abandoned, etc. If I ask them how their number was actually measured, I doubt any would be able to (accurately) tell me. The lack of a standard (and good!) definition for “success” is a problem.

Why AI Success Rate Studies Disagree

Let’s start with the number you’re most likely to have seen quoted: Project NANDA’s report last year (“State of AI in Business 2025”) that 95% of generative AI pilots fail. The report was built on an analysis of 300+ AI projects, 153 leader surveys, and 52 companies. The definition of success was 1) reaching production, and 2) demonstrable impact. Fair enough, but what’s missing is: how did they know P&L impact was caused by the AI project? I have the same question for any other quoted number for impact, and I’ve never seen that question answered.

McKinsey had a study from November of last year where they surveyed almost 2,000 people around the world, and their numbers show how big a swing in “results” comes from changing the definition. Just 39% of survey respondents credited any impact on earnings to AI at all, and only about 6% said the earnings impact of AI was “significant” and had a 5%+ impact on earnings. I could see either number being reported on its own as either showing substantial (39%) earnings impact, or almost no (6%) impact. Too easy to hand-wave and quote numbers.

I should also mention that the 6% number from McKinsey is oddly close to the NANDA project number (95% failure = 5% success), and some people point to the similarity as validation. However, the bigger point is that NONE of these numbers comes from validation that AI is responsible for either the measured change (NANDA study) or the self-reported survey numbers (McKinsey survey).

Metric Improvement Isn’t Proof

Suppose a company launches a chatbot and average handle time drops 20% that quarter. Did the chatbot cause the improvement? Not necessarily. Other things may have been rolled out at the same time, as the company tries many tactics to handle what’s perceived as a new problem or a new emphasis. Roll out a chatbot. Improve the self-serve content on the site. Fix the things that people were contacting the company about. Seasonal lull in communications from customers, correlated with simpler and faster questions. All of these happening at the same time would make it impossible to assign credit to one or more of these factors.

Five Questions to Measure AI Impact

You don’t need a data science team to measure the impact of AI (although, it helps). Answer these questions:

1. What’s the one metric for measuring success, and what’s the level required to be a “success”? This needs to be decided and written down before launch, not settled on after the fact to make the project look good.
2. Compared to what? What would have happened without AI? Is there an actual counterfactual? You need a holdout group, a staggered rollout, a matched comparison, something that indicates what the AI actually caused. Again, this has to be answered (and set up) prior to launch, as it’s largely unfixable after. Sure, you could try to do some fancy statistics, but a randomized controlled trial is the gold standard for measuring impacts (and I say this as a proud purveyor of fancy statistics to measure impact).
3. What else changed at the same time? Was there seasonality, another initiative, a policy change? Get clear on what else is moving, and if at all possible make the other changes stop. If “stopping” is not practical, you are leaning harder on the comparison set up in #2.
4. Is the metric chosen an actual outcome that matters? Tool usage or adoption are not indications of value to the organization. Choose revenue increase, cost savings, more leads, lower customer churn, something real.
5. When will “success or not” be evaluated? Set this date beforehand, then stick to it.

Measuring an AI Pilot (Example)

Suppose you plan to roll out an AI-powered assessment and decisioning tool to enhance your existing chatbot and insurance product offerings. Let’s run through the five questions:

1. What’s the metric? In this case, quote volume might be the appropriate choice, or total value of quotes if you want to guard against a shift toward more quotes at lower value (or profitability). For the “how much” part of this question, I’d compare this to the effort required to roll out the change. If it cost $300,000 in hard- and soft-costs to make this change, you want to see at least that much improvement in quote profitability. If the change was easy (one developer spent six hours building out the change with Claude), then the threshold for success should be much lower.
2. Compared to what? The natural comparison group here is the traffic: there are plenty of modern testing tools that can assign a persistent ID to visitors and shift half the visitors to the existing setup (with no AI-powered tool) and half to the new page with the tool. That’s your comparison.
3. What else is changing? If you’ve got the comparison group in #2, this question is far less important. Ideally, it’s still good to keep other changes to a minimum, but even if you change pricing, or staffing, or something else that would hit both populations, the differential still shows what difference the tool made.
4. Does the metric actually matter? Yes! Quote volume, or quote value, is a great, meaningful metric that would definitely impact the business.
5. When will success be evaluated? In this setup, even a relatively short trial of 2-4 weeks should generate a sufficiently powerful test to see what the impact is. When setting the length of time to evaluation, it is handy to have a data scientist or statistician determine the length needed, which will depend on how many quotes you typically get each day, how much that varies, and how small an improvement you want to detect. If you’re getting hundreds or even dozens of quotes per day, pretty steadily, you’ll only need weeks not months. For example, if you have 60 quotes per day with mild variation, and you split that 50% into current page and 50% into tool-enhanced page, and you want to set the length so that an increase from 30 to 35 quotes would be significant (with 95% confidence and 80% “Power”, assuming variance of 30), you’d only need three weeks.

So: answer the questions and do the setup right, run the project, then analyze the data. Do it this way and you’ll KNOW what the impact of your AI project was, with no guessing.