Community

This is my field. (Companies hire me to teach them how to organize and manage software development teams.)

The biggest issue is that software is intangible. It's like the proverbial iceberg; for the 10% of the software you can see, there's 90% more required to make that work. And software development is a huge amount of work. Polished, complete products typically take 5 or more people working for months (at least) or years. Big products take much more than that.

But nobody, not even programmers, are good at estimating how much work software development requires. Remember, it's an iceberg. In one study, 90% of projects took longer than the programmers estimated, even though they updated their estimates as work progressed*. Half of them took more than twice as long. These results are typical.

*www.toddlittleweb.com/Papers/...ncertain...

So even the best-meaning software team has a good chance of taking more than twice as long as their good-faith estimate.

Now for the second-biggest issue. The people hiring software teams don't understand how costly and risky software development is. When they get a schedule estimate, they think of the 10% they can see, and respond, "How it could possibly take so long?" (Remember, this is a reaction to an estimate that's almost certainly too optimistic already.) And then they put pressure on the team to go faster.

Programmers respond to the pressure by cutting corners. Here's the thing, though. The way you cut corners in software development is to be sloppy in the way you design and write source code. But sloppiness creates bugs and makes the source code harder to understand. This is called "technical debt." The net result is that if you do this for more than a handful of weeks, you actually end up taking longer than you would have if you had just tried to keep things clean.

They also respond to the pressure by focusing on the 10% of the code you can see (such as the 3D VTT demo shown when 4e was released) and not the 90% required to make it work for real. This shows progress to the business folks, but creates unrealistic expectations about how long things will actually take.

So here's how honest, well-meaning people create software debacles. This happens all the frikkin' time.

1. Company asks software team for some software, and asks how long it will take, so they can plan their budget, marketing, and so forth.

2. Software team creates an estimate that's 2-4x too low. They think they need 9 months and they actually need 2 years.

3. Company flips out and says 9 months is too long and too expensive. "My nephew could do this in two weeks in his spare time." Demands that the software be done cheaper and quicker.

4. Software team caves and says they'll be done sooner. After all, their estimate is just an educated guess; there's no proof, and maybe they actually will be done sooner.

5. Team works on the 10% people can see first, to establish good will and show progress. It's also the most fun part to write.

6. Company is overjoyed at rapid progress and happily signs checks. "I knew they were sandbagging us with that nine-month estimate."

7. Team starts working on the remaining 90%. Company starts pressuring for more results. "You've already shown us everything working, what's taking so long?" Checks signed less happily.

8. Team takes shortcuts, introduces bugs. Work slows down. Strings company along with increasingly desperate promises of progress.

9. Company gets more and more impatient and eventually demands to see the actual software, not the demo they got over a year ago.

10. Actual software fails in a big, spectacular way. It's riddled with bugs, doesn't work on the public Internet, and completely fails to protect company's IP. Furthermore, the source code is so badly written it's pretty much impossible to recover.

11. Product is cancelled, team fired, and work starts over.

There are ways of preventing this; the set of approaches I use are called "Agile software development," but it's by no means easy and most software teams don't even know they don't know enough.