Engineering: Maybe we have it all wrong

[Note: I wrote this in 2009 but never posted it anywhere. Just re-found it and thought I'd post it here.- RG]


When I was in college, the recommended software "engineering" processes were basically waterfall approaches using structured analysis and structured design. Later came the "object" paradigm and OO versions of analysis, design, and even "OO requirements gathering" (which made no sense to me at all.) Then came a myriad of new tools and processes for improving software "engineering": CASE, 4GLs, UML, Spiral, Crystal, Scrum, XP, etc. But these fads rarely seemed to make projects go faster or produce better products. After reading SEI reports, Steve McConnell's books, and countless articles and combining others' views with my own software projects experiences, it seems that most software projects have used – and continue to use – "hero development". Put together a smart group of developers, give them an idea and a keyboard, and let 'em go. Lots of useful software has been developed this way, some of it even great. But lots of software produced this way has been less than satisfactory, and much of it completely crashed and burned.

For years, I thought the problem with software development is that it is rarely conducted as a true engineering discipline. I kept thinking: we should build software like other companies build bridges, jet engines, and lawnmowers. But over the years, I've talked with engineers who work for companies that build these kinds of things, and I've found they have the same complaints we do: innovation stifled, lots of drudgery "patch" work, burdensome bureaucracies, the Peter Principle, etc. I hear stories from Motorola, Xerox, and, more recently, Boeing, which make me believe other engineers have the same challenges and frustrations as software developers despite their more structured engineering practices. Many of their products perform poorly or fail outright. The successful ones succeed mainly because of years of tight design constraints and exhaustive testing, which also make them very expensive.

It seems that, if software teams applied the same structured engineering processes as our lawnmower brethren, there would be no guarantee that we'd be any more successful than we are now.

So this got me to thinking: maybe we've got it all wrong about how to do "engineering". Perhaps because people are social animals that depend heavily on relationships and ad-hoc communication, structured processes are completely against the DNA that allows us to invent. If hero development often works, perhaps we should embrace the best parts of it and add structure and process only for the non-inventive activities. For example:

• Collaboration: This should be as high-velocity and high-fidelity as possible. This is the electricity that runs the inventive process. This is what makes Web 2.0 work. We should have lots of avenues and few rules for communication.
• Design and coding: These are core inventive activities and should be as unstructured (process-wise) as possible. Of course good software design principles should be used: modular design, efficient algorithms, proven data structures, etc. But significant innovation occurs in these efforts, so process constraints should be at a minimum.
• Feature definition: Deciding what features a product should have is the most important activity of software development. Building a 100% perfect product that no one wants is a 100% failure. We've tried "requirements specifications", "user stories", "feature backlogs", etc., but for commercial software products, feature development remains difficult. Because it cries for rapid prototyping, feedback, and refinement, feature definition is an activity ripe for innovation and creativity. Participants should be able to use UI mock-ups, cardboard posters, hand puppets, or anything else that helps distill the success factors of the project at hand.
• Manufacturing: When a really good car or toaster has been designed, we want the manufacturing tolerances to be as tight as possible. In software, the "manufacturing" activities are things like source control, builds, test execution, and deployment. These should be highly structured and measured. Deviations from allowable tolerances should set off alarms.
• Documentation and testing: It seems to me these often under-emphasized activities have both a design part, which lends itself to the same innovation as software design, and an execution part, which should be treated like any other manufacturing activity. Both should also honor proven techniques and patterns, but every product should have the leeway for innovative documentation and testing. (Imagine a product for which you could right-click any UI widget and select learn, and it would demonstrate how and when to use that widget.)

In other words, let the design activities be fluid, unstructured, and collaborative. Let the manufacturing processes be structured, automated, and measured. We would still need plans, milestones, schedules, and the like to allow us to operate in a corporate environment. But maybe we could invent a new way to do engineering that works better for the way people really are.