Few words strike fear into the hearts of software development managers and developers than scope creep. Long the bane of the development industry, scope creep has affected almost everyone who works on software – even those who work for themselves but especially those who report to a client providing software specifications. The perennial question in software development management has always been “How can we manage or eliminate scope creep?”
First, let us define scope creep for the purposes of this discussion. Scope Creep refers to the tendency within software development projects for the scope of a project to slowly grow over time through changing requirements and specifications. Scope Creep is a disaster for software projects because it makes budgeting nearly impossible, predictions of completion nearly always incorrect – often by astounding margins, makes determination of project feasibility questionable as the development team may not be aware of technical challenges that will be faced due to expanded requirements and can even cause a project to “scope thrash” in a state where “feature complete” is never clearly defined and a project never completes because the current state is never accepted as finished even if all existing features work correctly and current scope is far beyond the original requested scope.
While thinking about this article it occurred to me that there are two schools of thought on scope creep: those who believe accept that scope creep is an inherent artifact of the software development process and those who believe that it can be eliminated through project management practices (rather than simply mitigated.) But that is not the whole story. In reality, I realized, these two schools of thought are products of two higher issues: that some people believe that big design up front (aka BDUF) is a necessity and those who do not. Even this, though, is not the whole picture. At a higher level than this is the old argument as to whether software is an engineering discipline (engineering efforts require a small amount of design followed by a large amount of manufacturing or construction) or a design effort (where the design requires almost all of the effort and duplication is a trivial afterthought.)
Disclaimer: I am well aware that certain small segments of the software development community work on project types which absolutely require BDUF and locked requirements such as nuclear power station control systems or control systems for the space shuttle. This represents a niche software market and falls outside of a more general discussion. However, many projects of this type are only believed to be locked into BDUF processes because of client requirements and not because non-BDUF methodologies would necessarily produce less stable or reliable software. Statistics indicate that we may, in many cases, be increasing risk and failure rates by requiring these projects to avoid the best practices of the industry in general.
The logic goes like this: If you believe that software development is an engineering task then you assume that you must complete all design before “construction” begins. This requires that all or almost all design be completed prior to the beginning of construction or manufacturing. This is the origins of big design up front. This, in turn, requires that scope be carefully controller as BDUF has means of managing changes in scope and small changes can have catastrophic effects on an engineering project. Imagine a bridge where, after construction has started, the people who have request the bridge decide that they choose the wrong location or that its load bearing properties need to be changed to carry trains in addition to automotive traffic!
One of the key differences between software development and traditional engineering disciplines in that traditional clients understood that if the products was already made or partially made that the cost of retooling for changes would be staggering. Even a small change to a car (make it 2 inches longer) would send the entire design team back to the drawing boards and all aspects of the engineering portion would begin again and safety certifications, marketing, etc. would all have to retool almost as if they were starting from scratch. In software, though, clients do not see the final product as immutable and do not understand the impact of changes.
Scope creep in software development is deeper than a misunderstanding between clients and software engineers, however. In specifing a traditional engineering project it is generally fairly simple for the person requesting the product to provide meaningful and useful specifications. Here is an example, I can actually order a bridge from the engineering firm by stating: the location at which I need the bridge, the carrying capacity of the bridge (two lane road, four lane road, etc.) and any special features (one walking path on the side, two paths, open sides for a view, etc.) The rest of the bridge design is carried out without my input because what do I know about designing a bridge? In some cases there might be several designs submitted, all which meet the specifications but with wildly different looks, from which I can choose one to build. Once planning for that build has begun there will be cost involved with me changing my mind for obvious reasons.
Software engineering is not like this, though. In bridge building (or car manufacturing or whatever traditional engineering discipline you wish to imagine) the people physically putting the bridge or car together make absolutely no decisions about what parts to use or how to assemble them. They are provided a design that tells them every nail, rivet, girder, component, material type, weld type, etc. In software, the architect provides only a high level design and the software engineers or developers who actually create the software continue the low level design process choosing the design, components, structure, etc. So the design process continues until completion, at least a a low level. So in software everyone is an engineer and no one is a construction worker (or factory worker.)
This means that design is always underway no matter what we do. If software developers were not doing any design and simply assembling code as specified by a higher level designer then that piece of the work would be automated simply and the higher level designer would “become” the developer in question. That may sound strange until you think about high level languages like Java or C# .NET with large support libraries, application platforms like JBoss or Rails, GUI designers, toolkits like JQuery or Dojo, etc. all which come together to eliminate many routine design decisions for most projects allowing developers to switch from writing extremely low level code to focusing on more design issues and domain specific problems. The “moving up” of the designer is a constantly occurring process that year by year makes the average developer able to do so much more than they could in the past and makes bigger and more complex software projects possible.
Given that we know that scope creep can and will happen to most projects we have three choices: due nothing and see what happens, manage and mitigate scope screep (the BDUF approach) or “embrace change” and build scope change into the ongoing design process (the Agile approach.) Obviously, doing nothing, is a recipe for disaster and should be discounted. Only the naive take this approach.
Managing and mitigating scope creep is a challenging and difficult project management endeavor. This generally requires a large amount of client education and heavy contractual agreements limiting the client’s ability to make changes, to accept a reworking of scheduling and budgeting with each significant request and to lock client’s to a design at some point so that no further changes may be made. This approach can work and often does but its failure rate is one of the most significant contributing factors to the high percentage of failed software development projects throughout the industry.
This approach forces clients to make many decisions at the beginning of a project at the point when the least is known about it. It does not address changes external to the project that may force the need for scope change upon the project. It also risks clients becoming disatisfied with the project partway along and finding it easier to pull out and start again than to rework what already exists. Even when a project does complete it risks being outdated, poorly specified and at lower than anticipated relevance.
Moving to an Agile approach, where scope creep is accepted as being inherent to software development, things work very differently. In most Agile practices there is some design up front but the idea is to design only those portions very unlikely to change and only to design enough to provide the basic foundation and to address any hidden problems that could arise but would have been discovered through an up-front design process. Obviously no project will work well without any design at the beginning. How would you even know where to begin?
Agile does not dispute the need for design but, in fact, is designed around the idea that the design is so important that it can’t be determined at the outset and left as-is. Agile takes the approach that the design is far more important than that ands builds design into every aspect of the development project.
A moderate amount of design is done from the beginning. Then design, both low lever and architectural, continues throughout the project. This approach moves design decisions from the time when the least is known about the project to the time when the most is known. It builds in flexibility so that as the client or business requesting the software learns how the software can and will work they can learn more about how they will use it or expand what they need it to do or even reduce features that they later feel are unnecessary (although we all know how unlikely that is to happen.)
With many Agile processes, such as Extreme Programming, there is a concept of keeping the project in a continual “shippable” state in which everything that has been implemented is currently working and functional. In this way, at any point, if the project budget for time or money runs out the project is at least usable in some form even if not in the form originally envisioned. In this manner, the scope starts very small and grows with each project iteration. This is not creep but a built-in function of the development process and very much intended. In this way, the client or business can decide to use the software at any time while allowing for growth and advancement to continue as well. The project can peacefully continue until the client feels that additional features are no longer necessary. There is still a risk that the client will never call an end to the addition of features but since the project is continuously shippable or usable the risk involved in not reaching the “end” of a project does not exist.
An important aspect of Agile methodologies is that features are prioritized and completed discretely on a priority basis. By addressing features purely by their priority we can assure that the system always exists in its most usable state for the amount of time that has been given to it.
Both approaches to scope creep management have their merits. The risk of the traditional BDUF approach is that it is often used to shift responsibility from the software development team to the client. This is handy when you are the developers and have no long-term interest in the viability of the software but are only concerned about meeting contractual obligations (whether to a third part or to an internal business unit.) The BDUF approach is about solidifying responsibility with the client even though they are not software designers.
The Agile approach accepts that the client or business requiring software is not a software engineer and needs to work with the team to produce something that will meet the business needs. This approach is about meeting business needs not to lock the customer into paying for whatever original requirements were specified. Locking in customers may sound great to some outsourcing shops but businesses would be well advised to look elsewhere for companies whose interest lies in making great and working software not in meeting contract requirements alone.
Businesses looking to generate software internally have no reason to support the use of BDUF which operates in the disinterest of the business itself. This ideology pits the software development team against the business when, in reality, they should be aligned to work towards mutual benefit.
When deciding on a scope creep management strategy you must ask yourself, “Are you going to embrace the unknown and use it to your advantage or are you going to ignore it and continue to make software designed before all current information was gathered.” Businesses whose processes are Agile have a distinct advantage in the marketplace as their products are based on current market needs and pressures and not based as a response to pressures that existed when the project was first proposed.