Agile means many things to many people. I believe that the simple high-level Agile Manifesto is close to the way most engineers think about software development. Here’s a quick rundown:
> Individuals and interactions over processes and tools
> Working software over comprehensive documentation
> Customer collaboration over contract negotiation
> Responding to change over following a plan
However, once you get past these high-level ideas into the details, the agreement starts to fade. Agile has some good ideas but it also has problematic elements which are too centered around short-term thinking to work on large and complex engineering projects done not only at huge companies like Google, Amazon and Microsoft, but also in non-tech industries. For example, this could include building a core banking system, a high-frequency trading platform, a drug discovery system, or ERP software.
Without getting buried in details, let’s look at some of the principles behind the Agile Manifesto.
> Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
> The best architectures, requirements, and designs emerge from self-organizing teams.
> At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
> Continuous attention to technical excellence and good design enhances agility.
> Simplicity — the art of maximizing the amount of work not done — is essential.
These principles are almost common sense for smart engineers, and have been known and applied for years. NASA of the sixties, I am looking at you now.
However, there are other parts of the principles which are not a part of a healthy large and complex project development culture. These are the parts which have led to the short-term-focused Scrum process. They seem suited to particular types of development, most notably consulting or contract programming, where the customer is external to the organizations, runs the show because they are paying for development, and can change their mind at any time:
> Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
> Business people and developers must work together daily throughout the project. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
> Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
> Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
This style of short-term planning, direct customer contact, and continuous iteration is well suited to software with a simple core and lots of customer-visible features that are incrementally useful.
It is not so well suited to software which has a very simple interface and tons of hidden internal complexity, software which isn’t useful until it’s fairly complete, or leapfrog solutions the customer can’t imagine.
Companies like Google, Microsoft and Amazon write revolutionary software which has never been written before, and which doesn’t work until complex subcomponents are written.
This type of innovation takes significant up-front design time, and requires working on components over longer periods than one-week iterations. Because the projects have such simple external interfaces and so much internal complexity, much of the work is not even visible to “customers,” so there is no way to write customer-visible stories about it. With this type of software it takes 9–24 months to deliver the first working version to the customer.
Projects like this are the anti-Scrum. They represent extremely long-term thinking on the part of the technical leaders. Instead of working on something that would meet a small need this week, they were laying a foundation for a fundamental shift in the way cluster software was developed. That investment has not only reaped incredible rewards at Google, Microsoft and Amazon, but has influenced the entire industry.
Other industries have similar analogs. From tax-accounting software to computer games, some software is not suited to give to end customers when partially finished.
If I was asked to rewrite the above Agile principles to be more in-line with large and complex style software development, they might look something like this:
> Our highest priority is to increase customer (and programmer) productivity and access to information. Work on the biggest, most frequently used problems you can find, and create the largest net impact. Don’t give the customer what they ask for; understand them, and revolutionize their world.
> Developers should create a project charter (a fairly minimal but structured design doc), detailing the project, outlining what goals it hopes to achieve, and explaining why it can’t be done in other ways. This document should be circulated with stakeholders to get early feedback before the project gets underway. The written record is essential, as it assures there is a clear and agreed understanding of when the project is a success and how it aims to get there.
> At all phases of the project, critical design elements for larger components should be concisely explained and captured in a design document.
> Innovate in leapfrogs. It’s more important to finish and deploy a leapfrog than to attempt perfection. There is no perfection. Instead, be flexible and plan to constantly reinvent at every level of the stack.
> Deliver working software as soon as is reasonably possible, and no sooner. “Dogfood” projects internally before they are shipped externally. Make sure products meet high quality standards before shipping. The quality of the product is more important than the time it takes to achieve it.
While the high-level Agile Manifesto is flexible enough to work with these principles, these are very different than the short-iteration, low-documentation Scrum/SAFe/LeSS/Nexus/DAD processes that have become synonymous with the word «Agile.»
Starting with an understanding of what you are actually doing, the people you are working with, the things you are working with to get work done, and why you are doing it, should be the first priority, not starting by choosing a framework or process.
The biggest problem I am seeing is that people have made “Agile” the goal, worrying about “doing Agile right” before actually understanding the problem they are trying to solve and/or the opportunity they are trying to explore. They take it as a given that Agile is the right approach for the project, before the project is even defined.
In a nutshell: We should all start with an understanding of where we want to go before we figure out how we want to get there … not the reverse.
