Why Projects Fail | Incorrect Methodology
Among the dozens of reasons causing projects to collapse (Young, 2021), few project practitioners speak of methodology. That is, the triple constraints of scope, time, cost, combined with quality, dominate the popular mainstream discourse concerning why projects fail. Few mention methodology. This is a mistake. This article points out the importance of getting it right when deciding between agile and waterfall as an approach to project management and execution. Moreover, this paper not only compares and contrasts these two most common project management methodologies used globally (in part 1), but goes further to argue that choosing the wrong methodology can lead directly to project failure (in part 2).
Research question: Can selecting the wrong methodology cause a project to fail?
Part 1: Agile Versus Waterfall – A Comparative Analysis
Waterfall is a popular methodology used today to manage projects. Another type of methodology is called agile, growing quickly in popularity in recent decades. The thesis of this article asserts that selecting the wrong methodology (either agile or waterfall) is devastating, and can lead to project failure. Note that a third popular global methodology, called PRINCE2, is beyond the scope of this article, and will not be discussed herein.
What is Waterfall?
The waterfall methodology, often referred to as the “traditional“, “predictive“ or “conventional“ approach, is perhaps the oldest manner in which projects are managed. It is a linear approach to executing a project’s deliverables. It is a known as a sequencial lifecycle model (Hamilton, 2021). In other words, waterfall represents a step-by-step system in which the current project phase is dependent upon successful completion of the previous phase; and similarly, the next phase cannot commence until the current phase is finished. With some variation depending upon the organization and the particlular project in question, there are typically 6 stages within a classic waterfall project approach, outlined below:
- Ideation: This is where the initial project concept or idea is borne. At this stage, the organization’s leadership assesses the feasibiity of the project, deciding whether or not it is worthwhile to consider pursuing to the next step – planning.
- Planning: Once the viability of the project is approved in step one, planning the high level project begins in step 2, such as project objectives, scope, quality, approximate cost, resources required, general timelines, and critical success factors, including business requirements.
- Design: Armed with a plan (in step 2), the project team can begin executing the plan, which is to design the overall high level architecture of what the desired end state will look like. This could be the design of a new building on a construction project, or a new network architecture design for a technology transformation initiative.
- Develop/Build: Once the design is formally signed off in stage 3, the actual building or development of the product can commence in step 4. In software development, for example, this involves the coding and programming of the product.
- Test: Once the product is built or developed, it cannot proceed to the production environment or delivered to the customer until it passes approved quality assurrance standards. In other words, it must be thoroughly tested and accepted.
- Deploy: True to the sequential nature of waterfall, the last phase in the 6-step process is to implement the new solution to the end user consumer or stakeholder, whether it is a technology enhancement or an audit remediation project.
What is clear is that the waterfall methodology is linear. Each step is a pre-requisite to the subsequent step. See diagram #1 below for an example of the waterfall approach.
Diagram #1: The 6-Step Waterfall Project Lifecycle Model (Young, 2022)
Note: Consider the arrows pointing downward to the next step, representing how water flows or travels in a linear, sequential fashion, as if over a waterfall - hence the “waterfall“ terminology.
Pros and Cons of Waterfall
There is a reason waterfall is the oldest, most used project execution model. It has many advantages project teams enjoy. However, despite its numerous benefits, waterfall is not without its shortcomings. Some of its pros and cons are briefly presented below.
Pros of Waterfall
- Most planning & estimation are done up-front: In waterfall, planning is front-loaded. This means the vast majority of all project planning, in terms of cost estimates, requirements gathering, schedule duration, human resources, testing, among others, is performed very early in the project‘s lifecycle – immediately after the initiation phase. The benefit here is to prepare the plan early based on what is known, then simply follow the plan to completion. As the saying goes, “failing to plan is planning to fail“. Waterfall takes this approach very seriously. The illustration below shows where the planning stage fits within the 5 stages of the project management lifecycle.
Diagram #2: The 5 Phases of the Waterfall Project Management Lifecycle (Young, 2022)
Note: Notice that full project planning is completed early in the project lifecycle, in step #2.
- Early scope definition: Instead of defining, updating and refining scope during project execution, in waterfall, scope is defined, undersood and documented at project inception – during initiation and planning. Thus, by knowing early and up-front all the business requirements of the project allows the focus of the project team to be on strictly executing, as well as monitoring and controlling, the project’s scope.
- Easier to monitor & control: In waterfall, since planning, estimating and scope are well known and defined in the project’s beginning stages, it is easier for the project team to concentrate its efforts and energy on monitoring and controlling what has already been defined and planned. Uncertainty and change are reduced, therefore, making it easier to manage the project to successful completion.
- Measurements are well-defined: The only way to know how the project is performing is to measure its progress against the baseline (or starting point) . Because pertinent details of the project (such as scope, time, cost, and quality) are established very early in the waterfall model, so too can the project‘s measurements be documented and communicated to stakeholders early and often. In other words, metrics, key performance indicators, critical success factors, and all other measurement types, are created and signed off by the project sponsor at the beginning of the project, once planning is complete. This enables the project team members and stakeholders to understand precisely what success looks like while the project is in motion, instead of waiting until the end of the project when it is too late. This refers to the project baseline – the defined standards against which the project’s progress is measured. In this way, ambiguity in terms of what constitutes success, is removed. Thus, every step along the journey, the project team can measure its progress against the baseline to determine if they are on track or not, making course corrections as required.
- Clear roles and responsibilities: Insofar as waterfall has existed for a long time in terms of project management methodology, this has enabled clarity in role definition. That is, the waterfall approach provides clear, distinct roles with unambiguous responsibilities that are easy to articulate, understand and define. Since team responsibilities are typically not shared in waterfall, all team members know what their specific role and responsibilities are, bringing clarity to their execution and team development, simultaneously reducing conflict and ambiguity. Below is a simple organization chart of a typical waterfall-driven project.
Diagram #3: Waterfall organization chart (Young, 2022)
Note: The number of resources will vary based on project size and complexity, but the hierarchical nature of the project structure remains the same on all waterfall projects, with the project manager as the de facto project leader. All other team members, like testers, have their defined role and contribution to the project.
Cons of Waterfall
To be sure, the waterfall methodology has numerous advantages worthy of consideration when initiating a project, especially the certainty it creates by planning early. Waterfall, however, is not without its shortcomings, requiring project sponsors to carefully consider it and other options when establishing a project. Some of the best known cons of waterfall are discussed below.
- Inflexible Structure: Due to the sequencial, linear nature that is waterfall, such a structure may not be compatible with certain projects that require flexibiity. That is, waterfall can be rigid and fixed, not easily bending to the ups, downs and changes inherent in all projects. For example, quality testing cannot start until all planning and development have finished. Such a strict, built-in dependency does not allow much room for the flexibility of testing early and frequently.
- Poor Management of Uncertainty: The waterfall model does not capture uncertainty well. The entire premise of waterfall is that scope, time, cost and other critical success factors are well known in advance, requiring minimal change as the project unfolds. This equates to the assumption of a high degree of project certainty and business intelligence (permitting full project planning to occur). Planning for unknowns, therefore, becomes a tricky proposition in the world of waterfall, because waterfall projects do not know what they do not know.
- Low Customer and Stakeholder Engagement: Some executives relish the fact that waterfall does not require their constant intervention, guidance or “hand-holding“ on the project, empowering the project team to execute the approved project plan. For many senior managers, monthly steering committee updates will do just fine in light of their terribly busy schedules. Such infrequent stakeholder engagement, however, has some challenges and risks. Because customer, end user or executive stakeholder meetings are infrequent in waterfall (or scheduled at specific intervals), when ad hoc issues arise (as they inevitably do), the project team is often caught scrambling getting people’s attention to convene a meeting quickly. Daily senior management engagement, for instance, is unheard of in waterfall, unless the project is red.
- Might Not Be Suitable for Large, Complex Projects: The waterfall approach works best in an environment of high predictability and certainty and therefore low ambiguity. This is so because most planning is started and completed up-front in the very early phases of the project. One can only accurately plan for the future when matters like objectives, scope, quality, time and cost are clear. However, the larger the project the more complex it is likely to be; meaning possibly millions of budget dollars to manage and control, numerous stakeholders to engage and appease, dozens of human resources on the project team to lead with specialized skills, plus a prolonged timeframe (possibly years) to meet the project’s goals. Notably, the longer the duration of the project, the more difficult it is to accurately plan, capture and predict outcomes. It is one thing to plan for two to four weeks from today – but quite another undertaking to accurately plan what the project will look like (not to mention foreseeing risks and issues), one or two years into the future. Thus, the longer the project timeframe, the more complex it is; and the larger the project is makes planning, and hence the waterfall approach, extremely problematic to manage.
- Infrequent & Deferred Testing: In waterfall, testing only commences after planning is complete. Even test cases and testers are planned in advance. It is done only once as a “bucket“ or “batch“ of items or products to test, meaning there is a designated timeframe where all testing is executed – from days to months depending on project size and complexity. Once testing is finished, only then can implementation occur. The shortcoming of this “big-bang“ approach is that testing is not fluid, but rigid and fixed. It is planned months in advance to occur once, instead of at regular, smaller intervals along the way. Re-testing in waterfall is considered poor planning, wasteful, and costs more time and money not originally budgeted for, creating all sorts of problems for the project team. Big-batch style of quality testing does not account for potential changes in business requirements, affecting use-cases and other quality control measures.
- Scope Creep: Scope creep (unapproved changes in business requirements) is a major reason waterfall projects fail (Young, 2021). This is due to the fact that scope is well defined, documented, and approved during project initiation, including at business case writing and project charter development stages. Once scope is clearly defined, therefore, only then can the project be planned. Similarly, in waterfall, project execution is based on the project plan. If, however, during execution, changes to scope arise (creep), specific change management processes must be followed to manage the change in scope – but this is time-consuming, costly, and requires unanticipated (unplanned for) project energy to accommodate the scope change, such as attending change control board meetings seeking scope change approval. For instance, if the project was approved, budgeted for, planned for, and executed based upon a scope of fifteen widgets; but half-way through the project the scope increases to twenty-four widgets, this will have a significant unexpected impact to not just scope, but the schedule, cost, human resources and quality. Thus, while change management practices exist within waterfall to try to manage scope creep, it is still nevertheless problematic, as the waterfall methodology does not embrace change, and is considered a deviation.
To be sure, therefore, the waterfall approach has both advantages and disadvantages. It is an approach not meant for every project, given its inherent, inflexible limitations. As a result, other approaches to project management have emerged in recent decades, offering an alternative to the traditional waterfall methodology. The agile approach is perhaps the most prominent new way of thinking about how projects ought to be managed in the 21st century, and is articulated next.
What is Agile?
Seasoned project practitioners have been employing agile techniques for years. Today, due to increased and fierce competition, the rapid pace of technological advancement, along with disruptive startup companies, has put immense pressure on medium to large organizations to get their products to market quicker, adding business and consumer value sooner – just to stay competitive. As the Agile Alliance succinctly states, “More mature organizations are increasingly prone to being highly complex and potentially slow to innovate, and lag behind in delivering new solutions to their customers. These organizations find themselves competing with smaller organizations and startups that are able to rapidly produce products that fit customer needs. This speed of change will continue to drive large organizations to adopt an agile mindset in order to stay competitive and keep their existing market share.“ (Project Management Institute, 2017).
This is where agile was borne – out of the need to respond to a fast-paced, ever changing, uncertain, highly competitive market and business dynamic. Opposite to waterfall, agile is a mindset that values flexibility and speed. Agile approaches were created to explore project feasibility in short cycles and quickly adapt based on evaluation and feedback (Project Management Institute, 2017).
Stated another way, agile is a practice that facilitates continuous iteration of development and testing (Hamilton, December 2021). In agile, the project team works on different phases of the project at the same time, with short-term timeframes. Refer to diagram #4 below for a depiction of agile.
Diagram #4: The Agile Lifecycle (Young, 2022)
Note: In agile, each iteration (known as a sprint and typically lasting 2 – 4 weeks) contains 6 steps, and delivers a specific feature or benefit to the customer quickly. (An example of an immediate, quick feature benefit of a sprint is to develop a longer battery life on a smart phone). It is not until all the features (or requirements) have been successully delivered will an agile project be considered closed. The pros and cons of agile are briefly explored next.
Pros of Agile
As mentioned, the agile project management methodology is a 21st-century response to a rapidly changing business environment, brought about primarily due to major advances in technology – not the least of which is the revolutionary advent of the Internet. Some of the main advantages of the agile approach are expressed below.
- Frequent Customer Interaction, Communication and Stakeholder Engagement: The very nature of agile’s iterative approach to project management (in the form of short 2 – 4 week sprints), means key stakeholder communication and engagement is actually built into the sprint process, without which agile projects will fail. That is, the role of product owner in agile is equivalent to the business representative in waterfall. The difference is that (unlike waterfall projects) agile projects require regular, consistent interaction with the product owner throughout the sprint lifecycle, providing guidance, direction and constant feedback to the project team as it produces the product. Constant communication between the project team and the business is never a bad thing.
- Faster Development Lifecycle and Delivering of Business Benefits: Insofar as agile is iterative, divided into short sprints designed to deliver incremental business value within a matter of weeks (not months or years like waterfall) is a major benefit many organizations today much prefer. Companies are growing increasingly impatient with the notion of having to wait several months or years to realize project benefits. It is possible that corporate strategy, for instance, may have shifted during the project, rendering that project, therefore, irrelevant – wasting time, resources and money. In agile, however, benefits realization are delivered within weeks, not years - a major boost to the organization in its return on investment calculation.
- Embrace Change & Uncertainty: In waterfall project management, change and uncertainty are bad words, often indicating poor planning, requiring project adjustments. In agile, the opposite is true. Because of the much shorter planning and development lifecycle (2 - 4 week sprints), long-term project risk and uncertainty are mitigated. In fact, lessons learned and retrospectives (the ability to learn from and improve upon previous sprints), is a specific feature of the agile methodology; thus baking in and indeed welcoming the idea of change, uncertainty and continuous improvement.
- Reduces Overall Project Risk: As alluded to earlier, the longer the time horizon or duration of a project, the greater the uncertainty. The greater the uncertainty, the greater the risk that something unanticipated and unplanned for goes wrong. This is because the further into the future a project team tries to predict, the less accurate its predictions and planning is likely to be. In agile, this massive risk is reduced by employing short-term planning and short-term execution in the form of the 2 - 4 week sprint cycles. Intuitively, this makes perfect sense in that it is significantly easier to plan and execute a project deliverable with a 4-week timeframe, rather than a 4-month timeframe – as the distant future is unpredictable and unknowable. Agile accommodates and mitigates this risk with short, iterative planning cycles.
- Improved Team-Building and Empowerment: A critical feature of the agile approach is the desire for colocation, where the entire project team collaborates in person. Thus, remote and virtual team composition are frowned upon as being inherently less productive, less efficient and more burdensome on project team members. Though of course not always possible in the era of globalization, nevertheless, agile espouses the view that self-directed, highly independent, highly empowered work teams produce superior project results.
There is little doubt, therefore, that the agile methodology - which values speed, increased customer interaction, and project team collaboration - represents a profound mindset shift from the traditional waterfall approach, which prefers up-front planning of scope, schedule, resources, quality and costs. Unsurprisingly, however, the agile approach is not flawless, and not ideal for all projects in all organizations. As such, the disadvantages of agile are mentioned next.
Cons of Agile
- Requires Consistent Customer Involvement & Interaction: In that the agile project management approach is built on constant feedback and continuous improvement within each iteration or sprint means the customer or business partner must be regularly and actively involved and engaged throughout the entire process (or iteration) for this methodology to work. This represents a large time commitment on behalf of the customer stakeholder, which may not always be possible in a rapidly changing business environment. Conversely, any delay in getting customer feedback can negatively impact the execution and development of the features within the sprint – thus delaying the project itself.
- Colocation Not Always Possible: A highly commendable attribute of agile is the desire to have all team members colocate; that is, share the same work space, ideally face-to-face. This, theoretically, produces a better team atmosphere and improved results given the closeness of the team in terms of proximity. Indeed, colocation can make certain decision-making techniques, such as brainstorming, easier. However, colocation is not always possible; in fact, many medium to large projects a virtual by design, specifically seeking and employing subject matter experts from across the globe. In this way, colocation is seen as neither feasible nor desired.
- Assumes 100% Team Allocation & Dedication: Unlike many waterfall projects where team members are allocated part-time to the project (having full-time jobs in their functional department) agile works best when all project teammates are 100% allocated and dedicated to one project. This means in agile project team members should not be on multiple projects simultaneously, where their focus and attention are divided, meaning they are multi-tasking. However, organizations face the reality of limited and scarce resources of subject matter experts. A senior engineer, for instance, may be required to lend her expertise to multiple projects, not just one agile project. The alternative would be to hire multiple engineers dedicated 100% on numerous agile projects – which might be cost-prohibitive for an organization, instead of these same projects sharing the expertise of one expert resource.
- The Timebox Approach Not Always Suitable: An example of the timebox concept in agile is the 2 – 4 week sprint interation. That is, the premise of agile is based on the notion of small, fixed, work periods of time, known as a timebox, where a feature or product is produced or developed for quicker customer consumption. But what if this sprint of only 2 – 4 weeks is not suitable or adequate, and more time is required? To be forced or limited into a pre-determined and pre-defined timeframe of 2 – 4 weeks is not guaranteed to match the actual time it may take to develop a good product. The consequence might be to rush product development to meet a pre-defined deadline that may not be realistic – leading to poor quality – and possible project failure.
- Assumes a High Performing, Self-Managed Team: Successful agile projects require a high degree of teamwork and collaboration. Recall some key premises of agile are 100% team allocation, 100% team dedication, and preferably colocation. This closeness is intended to create a bond or cohesiveness which, in turn, should improve project results. In other words, the agile concept removes the need for a project manager providing overall direction on key matters such as scope, schedule, cost, quality and risk. (Instead, agile uses a scrum master, who is essentially a coach and facilitator – not a manager per se). The entire premise of this approach, therefore, is that the agile team is self-managed, self-directed and is a high-performing unit. That is asking a lot of the project team. Not all teams are high-performing. Further, not all teams (or people for that matter) are successful at being highly autonomous, largely independent, self-managed and self-directed. The moment there is team conflict, as is inevitably the case with human beings, the self-managed agile construct can break down, presenting a risk to project outcomes.
Diagram 5 below summarizes neatly the pros and cons mentioned above.
Diagram #5: Project Methodology Pros and Cons Matrix (Young, 2022)
To be sure, there is no one perfect way or methodology to manage a project successfully. Both agile and waterfall have benefits and pitfalls. Thus, it is the responsibility of the project team, in conjunction with the project sponsor and other key stakeholders, to design an execution strategy that works for them based on the organization‘s structure and culture. In part 2 next, it is shown how selecting the wrong project methodology can spell disaster for achieving successful project results, up to and including project failure.
Part 2: How Methodology Misdiagnosis Can Lead to Project Failure
Undoubtedly, projects fail for many reasons, from poor communication and bad cost estimating, to weak monitoring, or insufficient testing and quality control during the execution phase of the project (Young, 2021). But among the very first important decisions the project team must make, in collaboration with the project sponsors and key stakeholders, is which project methodology (agile or waterfall) to choose that is expected to drive the project to successful completion and benefits realization. If the project team and its leadership get this crucial early decision wrong, the project is destined for troubled times ahead, often leading to complete failure.
To appreciate how critical the project methodology decision is in respect of setting up the project team for success, consider the emergency room triage nurse analogy. That is, upon entering a hospital‘s emergency room, the patient immediately is referred to a triage nurse. Among other things, a primary responsibility of the triage nurse at this very early stage of patient care is accurate assessment and preliminary diagnosis of the illness or condition of the patient. Accordingly, the triage nurse performs an evaluation of the patient’s condition and vital signs in order to determine, not only the urgency and severity of the condition, but just as important is the direction or next steps towards treatment. If the evaluation suggests, for example, a heart issue, the patient is directed to the cardiology department for care. Similarly, if the problem is a bone bruise or contusion, the patient is referred to the fracture clinic for repair, and so on. Thus, before any remediation can occur, it is vital that the triage nurse accurately assess the ailment and refers the patient for the appropriate care and treatment.
The inverse is also true – but with dire consequences. In other words, failure of the triage nurse to adequately assess the patient’s needs may lead to disastrous implications, including literally life and death. This point cannot be over-emphasized. Imagine a true, real-life scenario whereby a football (soccer) player collapses on the field and is quickly rushed to the hospital emergency room, unable to walk. The triage nurse concludes the injury is minor, only a sprained ankle, and sends the footballer home with an ice pack and Advil medication.
Later, after walking on the sore ankle for two weeks with no relief or improvement and severe pain, the footballer returns to emergency to be re-assessed. This time a different triage nurse concludes that the injury is a torn achilles tendon, requiring emergency surgery to prevent amputation. The emergency room doctor agrees and surgery is performed immediately, saving the leg from amputation.
Thus, the implication of a misdiagnosis in this case is clear. Inadequate triage can lead to terrible future consequences for a patient. Though not literally life and death, the same principle applies to project management. In the very early stages of a project, before planning begins and preferably during the initiation stage, the key project stakeholders and sponsor must decide, or triage, the state (or condition) of the up-coming project and company culture and structure, selecting which methodology is best suited to successfully execute the project.
Here, the leadership must choose between agile and waterfall. Is scope known and unambiguous? Can small, incremental benefits be realized in short bursts? How much time do stakeholders have to support the project team? How technically complex is the project? Is colocation feasible? What is the project’s size? Is there a high degree of uncertainty associated with the requirements and testing? Is this a short-term or long-term initiative? These and other factors must be strongly considered and triaged well in advance of project execution.
Failure to treat a project like an emergency room triage nurse can lead directly to project failure. Consider, for instance, a scenario whereby the project sponsor does not take the necessary time to triage and assess the project in terms of the best execution strategy and simply defaults to a waterfall methodology as it is the most common. As has been demonstrated in this article, agile and waterfall are nearly polar opposite project mindsets. To try to force what ought to be an agile approach into a waterfall project is destined to fail because they are very different. The most obvious example is that agile employs a scrum master while waterfall uses a pure project manager. Hiring a scrum master to execute a complex waterfall project will likely fail.
It is irrefutable that there are numerous reasons projects fail. Poorly defined scope, cost, quality and schedule are commonly seen as project killers and are well documented. What is often neglected, however, as a significant factor contributing to project failure, is incorrect selection of the appropriate project methodology by organizational leaders before the project even starts. But this is just as important as scope or quality control.
This article has shown that agile and waterfall are two leading approaches to project management – both with advantages and shortcomings. But they are very different in their mindset, execution and philosophies and are not suitable for all projects. To select an agile approach for a short-term, low budget, few resources, global project with low complexity and high certainty is inappropriate and counterproductive. Conversely, choosing waterfall for a project where scope, time, cost and quality are constantly changing, along with business requirements, is likely to fail miserably.
Author: Ricardo Young
Agile: A Practical Guide. (Project Management Institute, 2017). Newton Square, PA.
Fair, J. (2012). Agile versus Waterfall: Which Approach is Right? Paper presented at PMI Global Congress 2012, Marsailles, France. Newton Square, PA.
Hamilton, T. (December 11, 2021). Agile vs Waterfall: Know the Difference Between Methodologes. Retrieved Februray 4, 2022 from https://www.guru99.com/waterfall-vs-agile.html
Hoory, L and Bottorff, C. (October 27, 2021). Agile versus Waterfall: Which Project Management Methodology Should I use? Retrieved February 4, 2022 from https://www.forbes.com/advisor/business/agile-vs-waterfall-methodology/
Lees, H. (September 30, 2021). Agile v Waterfall: What is the Difference? Which is Best For You? Retrieved February 4, 2022 from https://www.trustradius.com/buyer-blog/difference-between-agile-vs-waterfall
Lotz, M. (July 5, 2018). Waterfall v Agile: Which is the Right Development Model for Your Project? Retrieved February 4, 2022 from https://www.seguetech.com/waterfall-vs-agile-methodology/
Santos, J. (September 1, 2021). Agile v Waterfall: Differences in Software Development Methodologies. Retrieved February 4, 2022 from https://project-management.com/agile-vs-waterfall/
Terry, J. (n.d.). Agile Methodologies: A Beginner’s Guide. Retrieved February 4, 2022 from https://www.planview.com/resources/guide/agile-methodologies-a-beginners-guide/history-of-agile/#:~:text=It%20all%20started%20in%20the,new%20software%20to%20market%20faster.