Why Do AI and Data Science Projects Fail? 7 Reasons and How to Prevent Them

How common is failure?

According to RAND Corporation’s 2024 research, more than 80 percent of AI projects fail, which is twice the failure rate of IT projects that do not use AI. The number speaks for itself, but what is really interesting is what the research says about the causes of failure.

RAND names five root causes: management misunderstands what problem needs solving, or optimises for the wrong metric; there is not enough high-quality data for training; the organisation puts the newest technology ahead of the real business problem; data management and operating infrastructure are underfunded; and sometimes the task simply goes beyond what the technology can do today.

Four out of five reasons are not technical. This matches what we see in our own projects, and what this article already argued in 2019: the problem is almost never the choice of tool.

The seven reasons in a table

1. No one knows what data means

This is the most common and least visible obstacle. Even in multinational companies, it happens that they cannot answer how many customers they have, because different divisions mean different things by "customer". The same is true for almost every field: measured value or calculated value? Who produced it, and according to what rule?

Until this is clarified, the project cannot move forward, because someone will question every result by saying "that is not how we understand it". The solution is not technological: you need to find data owners and write down what the data means. This is also the first step of data asset assessment.

2. Nobody has an interest in a result being produced

In large organisations, data is power. Departments are reluctant to share their data because they fear being held accountable, or because they see ownership as an advantage. In such cases, the data scientist can easily become a common enemy: their work makes visible what did not need to be shown before.

Only one thing really protects against this: the project must also be good for the data owner. If the result of the analysis is only a risk for them, there will be no cooperation, no matter what management says.

3. There is no trust in the result

People cling to their intuition more than we think, and when the data contradicts their gut feeling, they usually blame the data. "Surely analytics broke." "It must be a measurement error."

There is also a less well-known reason for this: whether we regret a decision is determined not primarily by its correctness, but by how we made it. A bad decision made on intuition is easier to live with than one made from data. We wrote more about this in our post on data-driven decision-making.

The defence here is to fix the success criterion before the result is known, and to make the method transparent.

4. No decision is made based on the data

There is data, it is reliable, the analysis is done, and yet nothing happens. The reasons are usually mundane: there is no budget for it, there is a legal obstacle, there is no one to carry it out, or an internal rule forbids it.

This is the most demoralising version, because the team quite rightly asks why we are measuring if we are not going to do anything with it anyway. You can prevent this by naming, right at the start of the project, who will make the decision based on the result, and what intervention will follow. If there is no answer to that, the project is not ready to launch.

5. The business question is wrong

The ideal data project starts from a concrete business problem. In practice, however, things often go the other way around: data accumulates first, and then we look for a use for it. That is not always bad, but it is a much harder road.

The difference is that in a project starting from a business question, we know what we will do differently once we have the answer. That is why the CRISP-DM methodology also starts with understanding the business goal, not the data.

6. We started with expectations that were too high

The idea that "data is the new oil" breeds excessive optimism. It is often said that "twenty percent loss can definitely be extracted" or that "there is twenty percent profit in the data". Then it turns out that the data is stored badly, the sought-after relationship is not in it, or one of the previous five problems gets in the way.

That does not mean there is no value in it. It means the value must be estimated before we commit to it.

7. Skills are missing

The ideal data scientist, who simultaneously understands the domain, the business, change management and technology in depth, does not exist. Anyone looking for such a person will be looking for a long time.

This is solved by a team, not a hero. And it is worth honestly checking which piece is missing from among us: if the domain knowledge is there but the analysis is not, that is a different situation than the other way around.

One extra success criterion: feedback time

There is a rule of thumb about what makes a data mining project good. Six criteria are usually listed:

1. A dataset with many rows.
2. A database rich in attributes.
3. Clean data.
4. The data accurately represents the event to be predicted.
5. Well measurable ROI at the organisational level.
6. An actionable domain, where management can change the process.

But there is a seventh one that is rarely mentioned, even though it often decides everything: how quickly we get feedback on whether we made the right decision.

A churn project at a mobile operator is a textbook example because it satisfies all seven: a large customer base, rich data (call patterns, billing, device type), relatively clean data, measurable campaign effectiveness, and multiple channels for intervention.

Why this also decides the model choice

Here is the business edge. In a web shop recommendation system, you can see within hours or days whether it works. In that case, even a black box is acceptable, because if it is wrong, you find out quickly and can correct it.

At a bank or insurer, however, it may take years to find out whether the decision was right. There you cannot wait for the feedback, so only a model that can be understood and defended in advance is acceptable. This is one of the strongest business arguments for explainable artificial intelligence: it is not a theoretical issue, but a consequence of feedback time.

So it is worth asking at the start of the project: how long until I know whether I was right?

What changed with generative AI?

None of the seven reasons disappeared, but two took on a new form.

The wrong question is easier to hide. A prototype built on a language model can look impressive in a few days. That can make it seem as if the project is moving, while nobody has yet asked what this changes in the way the organisation actually works. RAND calls this a bottom-up failure: the technology gets ahead of the problem.

The expectation starts from a higher level. The gap between the quality of a demo and the reliability of a live system is larger than with classical models, because we look at the demo with human eyes, while we judge the live system on the basis of a thousand cases. That is why you need your own test set and evaluation criteria before anyone promises a date.

How to reduce the risk

Three steps at the start of the project are worth more than any later technological choice.

1. Use-case development. Defining the business question, quantifying the success criterion, and clarifying who will decide based on the result. This protects against reasons 1, 4 and 5.
2. ROI calculation and simulation. Before the decision, you can see the expected impact, total cost, feasibility and return. This protects against reason 6, and also provides a basis against reason 3. This is what our ROI calculation and solution simulation is for.
3. Pilot project. A small-scale implementation on real data that shows where things get stuck, before a large sum is on the line.

Pre-launch checklist

Eight questions. If you do not have answers to more than three of them, the project is not ready yet.

1. Is the meaning of the few key data items needed for the project written down, and who is responsible for them?
2. Is there a management mandate to obtain the data?
3. Does the data owner also benefit if the project succeeds?
4. Have we quantified what we consider success?
5. Have we named who will make the decision based on the result?
6. Is there budget and legal room for intervention?
7. Have we estimated the expected return, rather than just hoping for it?
8. Do we know how much time it will take before we find out whether we made the right decision?

Frequently asked questions

What percentage of AI projects fail? According to RAND Corporation’s 2024 research, more than 80 percent, which is twice the failure rate of IT projects that do not use AI. The research names five root causes, and four of them are non-technical.

What is the most common reason? In our experience, it is that there is no shared data definition, and that no decision is made based on the result. Both are organisational, not technological, questions, and both are cheapest to address at the start of the project.

What makes a data project good? Seven conditions: enough data and enough rich data, clean data, data that really describes the event to be predicted, measurable return on investment, a domain where intervention is possible, and fast feedback time.

Why does feedback time matter? Because it determines how much risk we can take with the model. Where the result becomes clear within hours, even a black box is acceptable. Where it takes years, only a model that can be understood in advance.

Can these pitfalls be avoided? Not completely, but most of them can be filtered out in the first few weeks of the project. That is why we recommend that use-case development and ROI calculation come before the development decision.

Does this also apply to smaller companies? Reasons 1, 2 and 7 are usually less common there because there are fewer stakeholders. But 4 and 5 are just as present, and in a smaller organisation a project gone wrong is proportionally a bigger loss.

Sources

• James Ryseff, Brandon F. De Bruhl, Sydne J. Newberry: The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed. RAND Corporation, 2024.