Projects fail for many reasons: optimistic estimates; technology that doesn’t work as advertised; misunderstanding the needs of key stakeholders; plans driven by deadlines rather than feasibility; the belief of the team that given enough pizza they can solve any problem by tomorrow morning.
In my experience, many of these reasons come back to one root cause: we lose touch with reality. A few examples:
– We spend insufficient time exploring and validating assumptions at the start of a project, so we operate from different expectations of what’s required or how they’ll work together. This makes it almost impossible to build meaningful estimates and schedules. The project begins from a position of unrealistic expectations and is playing catch-up forever after. See my article “No validation? No project!”.
– We start with an over-optimistic assessment of our team’s capabilities and skills, or of the technology and methodologies, we are using. Initial slow progress is explained by lack of familiarity. Eventually, the reality of slower than expected progress hits us.
– Sponsors may not realize how much effort and commitment they need to put into sponsorship, or what skills will be required. They may assume that the project manager can handle many of the things that are properly their responsibility. A project with weak leadership will then have trouble connecting to broader organizational objectives.
– We negotiate poorly with key stakeholders. How often have you seen an executive beat down the initial estimate from a technical team? The team thinks they’re estimating, but the executive is conducting a negotiation. At least one person has lost touch with reality in this situation.
– Teams under pressure find it difficult to step back and think about what’s really going on. Minor problems pile up because everyone is so busy that they can’t see the underlying trend.
– Even if we can see the underlying trend, we have trouble reporting it. A culture that says “come to me with solutions, not problems”, may mean that unsolved problems don’t get talked about. A culture where senior managers are expected to have all the answers may mean that sponsors can’t admit it that they don’t understand what their project manager is saying. Often we just don’t know who to go to with their observations. People with problems have trouble connecting to people who may have solutions, and the project’s leadership remains disconnected from the real situation.
– And, of course, many of us don’t even want to think about what the real risks of the project might be.
This doesn’t happen because we are lazy, or because they are trying to sabotage their projects. Skills and experience may provide some insulation, but they don’t eliminate the problems completely. We lose touch with reality for reasons rooted in basic psychology. They are called cognitive biases. There are many of them, and most could impact your project. I will discuss the, in my opinion, most «dangerous» ones.
Overconfidence
«The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts” – Bertrand Russel
We tend to overestimate our own ability. Surveys show, for example, that most of us consider ourselves to be above average intelligence, or to be better software developers than average, or to be better project managers than average. Likewise, most consultancies and system integrator claim a significantly higher success rate than the industry average. This specific effect of overconfidence is known as the Dunning-Kruger effect.
Overconfidence is a direct cause of underestimation on many projects. It helps driving phenomena like scope creep (developers think they can implement a feature with little effort) and the famous ninety-ninety rule.
«The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time.» – Tom Cargill, Bell Labs
Oversimplification
«Everything should be made as simple as possible, but not simpler» – Albert Einstein
We tend to use simplifying assumptions and mental model in order to deal with complexity. This is sometimes essential, for example when rapid decision making is important, but it can lead to unintended outcomes very fast. Most of us have been caught by time, cost and complexity estimates that escalate as we get further into the details. Yet most big projects are started with very broad, “high level” estimates.
Likewise, we often operate from mental models for how projects, teams or organizations should work. These are simplifications of reality; they embed large numbers of assumptions. If we use them unthinkingly, we fail to recognize situations where the assumptions are violated. We begin to operate in model land rather than the real world. Anybody who has done some work with one of the MBB management consultants should have experience with this. Not to mention the software and solution architects that have not done any real work since switching to PowerPoint architecture.
When we fail to recognize the true complexity of our projects, their requirements and the organizations that they are serving, we accept an unrealistic scope, estimates, staffing, and plans. We also fail to recognize and manage key risks.
Avoiding Pain
“Pain in this life is not avoidable, but the pain we create avoiding pain is avoidable” – RD Laing
We tend to avoid pain. I am not talking about physical pain, but emotional pain. Sometimes this means deferring immediate pain at the risk (or even certainty) of needing to deal with greater future pain. For example, rather to face up to a tough conversation now (“it probably can’t be done”), we take high-risk options (“the only way it can be done is if we cut all these corners and hope for the best”). Likewise, rather than going through the pain of admitting that we have been delayed, we find reasons to persuade ourselves that we will catch up by the next milestone. Like that will ever happen…
The pain we are avoiding comes from a variety of sources and is very much depending on your personality.
– We may be avoiding confrontation, anger or disappointment. This is why so many of us have trouble saying “no”. We would rather fail later than deal with anger or disappointment now. Especially with people we love, like, fear and/or respect.
– We may be avoiding the feeling for the loss of control or mastery. We may focus on tasks we are good at, rather than those that are important. Technical people often derive a lot of their identity from their ability to solve problems. Admitting they can’t solve a problem represents a deep threat to their sense of identity. Project managers who like to feel in control of the situation (which they never are), may avoid finding out or understanding things they can’t control (which are almost all important things in a project).
– We might be avoiding cultural taboos. Some teams and cultures have difficulties saying “no” because it is just not something you do. There are string cultural pressures to make “positive” responses. Many organizations find it difficult to discuss risks for similar reasons. There is a strong pressure to “smile, it probably never happens”.
– We might be avoiding loss of “face”. For example, senior managers find it difficult to admit their ignorance to the people reporting to them. Rather than seek clarification to status reports, they may allow things to drift. This is a common reason for weak project sponsors and steering committee members. They simply do not know how to help and do not want to ask. Strong project managers can manage/engage their StC and sponsor, but inexperienced project managers, who are most likely to need support themselves, can find this hard to work with.
– We may be avoiding apparent costs. Spending time to clarify objectives; allocating time to risk mitigation activities; undertaking further analysis of the problem; performing independent reviews and assurance. These all involve immediate costs with uncertain paybacks. There is often a string pressure to avoid these immediate and obvious costs in order to reduce the budget.
Confirmation Bias
“We tend to accept information that confirms our prior beliefs and ignore or discredit information that does not. This confirmation bias settles over our eyes like distorting spectacles for everything we look at.» – Kyle Hill
We tend to look for evidence that confirms our existing hopes and beliefs. We want, for example, to believe that our project is on course so we seek evidence that shows we are hitting milestones and ignore information that shows where we are behind schedule. As a team makes commitments to budgets, estimates, schedules, and deliverables, this bias becomes more embedded. They begin to focus their energy on finding ways to achieve these commitments and to avoid any signs of problems or risks. The opposite can happen as well. As soon as a team believes the project will fail, it most probably will, because they will focus on finding reasons why it will.
Confirmation bias is often built in our project approval process. People say what they think they need to say in order to get the projects they believe in approved. They may not lie, but they focus only cursory attention to the costs, risks and side effects. This then becomes the approved version of the project’s storyline, to become embedded in people’s thinking through repetition.
Availability Cascade
«Vision without execution is hallucination.» – Thomas Edison
The availability cascade is a self-reinforcing process in which a collective belief gains more and more plausibility through its increasing repetition in public discourse. If we hear a message often enough, we begin to believe it. This is the basis of most marketing and propaganda. Eventually, we may end up believing our own propaganda. A team takes initial, rough estimates and works with them, gradually losing sight of the fact that there is little basis behind them; a manager repeats the high-level vision to all who will listen and starts to mistake the vision for delivered reality.
Perceptual Bias
“How does a project get to be a year late? One day at a time” – Fred Brooks
We often find it harder to recognize gradual trends than sudden changes. We get locked into our view of individual details rather than seeing the big picture. When we are under pressure we see those days as individual, disconnected events and lose track of the bigger trend towards disaster.
Anchoring Effect
You’ve seen the anchoring effect in action a million times, usually in the form of a suggested retail price. When you see a purse with a price tag that claims it normally sells for $400, but is marked down to $200, that little «yippee!» you feel inside at the prospect of getting a bargain? The anchoring effect. The seller simply told you that the purse was worth $400, and we tend to accept this line of bull because we are basically trusting people and trained to use price as our gauge of value.
Anchoring is a cognitive bias that describes the common human tendency to rely too heavily on the first piece of information offered (the «anchor») when making decisions. During decision making, anchoring occurs when individuals use an initial piece of information to make subsequent judgments. Once an anchor is set, other judgments are made by adjusting away from that anchor, and there is a bias toward interpreting other information around the anchor.
So you can imagine now why your first rough estimations are maybe not so harmless.
Reinforcement Loops
All the above-described biases often combine and reinforce each other. We want to believe that we are above average at what we do, so confirmation bias reinforces our overconfidence. As we continue to confirm how good we are, repetition bias adds further reinforcement. This growing overconfidence makes us more prone to use pain-avoidance strategies. We assume we will be able to solve any problems, so we do not face up to potential difficulties or invest in analysis and risk mitigation. It is cheap to fly in two independent experts to validate your solution architecture for your multimillion software development project compared to finding out later that your design will not work and you could have known this from the get-go. Those experts might even have some other great tips and tricks as well.
Organizational Accelerators
“The limits of my language means the limits of my world.” – Ludwig Wittgenstein
On top of the biases and their reinforcement loops, we have the organizational culture that can make things worse.
Organizations that focus on looking for blame instead of solutions increase the pain associated with admitting problems increasing the chance that people will hide them till the very end. Strong hierarchical organizations can increase the pain associated with confrontation or loss of face (think Yakuza), making it difficult for people at a different level to admit their uncertainties to each other. Organizations with a strong collegial culture have a higher risk of “groupthink”, the tendency for people in a group to converge on consensus at the expense of critical examination.
Projects that span organizational boundaries face additional challenges. You have to deal with not one, but multiple cultures. This is especially true in joint ventures and outsourcing relationships that involve multiple organizations. But is also effects projects working across group and unit boundaries within a single organization.
People may be uncertain as to what information can be shared or how it should be shared. Different professional backgrounds, languages, and conventions may make it difficult to understand the information that is shared.
Contracts and legal concerns can create further barriers. Penalty clauses, for example, may help align objectives across organizations, but they also create incentives to hide problems and to be very reluctant in supporting change. Procurement regulations may restrict what information can be shared at key stages in the project. All these barriers can make it harder for people to uncover and deal with all the biases described above.
Closing Thoughts
Underlying all this, reality does not always gives us the message we would like to hear. We do not want to hear that we are less intelligent than we would like, or less capable of solving problems, or that we code too slowly. Project failure is one of the prices we pay to avoid (or at least delay) hearing these messages.
Eventually, however, our projects come face-to-face with reality. Something will happen that we cannot ignore anymore. Unfortunately, by the time this happens, it is often too late.
This is why project reviews exist. They are a way to help teams, sponsors, project managers, or other stakeholders manage these human biases and keep in touch with what is really happening on their projects. Of course, they are not the only way. Test automation, project validation, iterative delivery, financial controlling, real estimations, pair programming and many others will help reducing the biases. But every now and then a project review can save you many headaches.