The Impact of Social Forces on Software Project Failures

Douglas C. Schmidt
Editor-in-chief
C++ Report
April 1996

I've been deeply involved in object-oriented C++ software development the past decade. Like my editorial predecessor Stan Lippman, however, my background began outside of computing. In my case, I studied sociology, focusing on organizational behavior and group dynamics. This training left a deep impression on me and throughout my technical career I've been acutely aware of the impact that social forces have on software development. Recently, I've been studying the social forces that cause some software development organizations to succeed and others to fail. In this month's Editors Corner I'll discuss some common dysfunctional traits I've observed in unsuccessful projects. In my next Editors Corner, I'll discuss how other projects I've worked on have succeeded, in spite of the potential traps and pitfalls discussed below.

It's no secret that software projects often fail, e.g., by not meeting customer requirements, arriving substantially overbudget and late, or being cancelled entirely. In addition to my academic research, I've worked on a broad range of software projects ranging from online transaction processing to telecommunication software and distributed medical imaging systems. Some projects have been quite successful -- shipping products that met or exceeded customer requirements and expectations. However, I'd be remiss if I didn't admit that too many projects have failed. I suspect that I'm not alone in my experience -- some reports claims that well over half of all new software projects are never completed, or are substantially downsized due to cost overruns, poor software quality, etc.

There are many reasons for the high rate of failed software projects. A convenient explanation places the blame on the programming language (e.g., C++ vs. Smalltalk), operating system (e.g., UNIX vs. Windows NT), or design method (e.g., object-oriented vs. structured design). In my experience, however, the reasons most projects fail are deeply rooted in social forces. The following is a list of five traits that I've observed in the unsuccessful development teams and organizations I've worked with. If left unchecked, these traits can seriously undermine software project success.

Trait 1: Death Through Quality

I've noticed a disturbing trend in the software industry recently. Many organizations have become obsessed with software Process (note the capital "P") as the panacea for their software quality problems. These organizations often institute Process rituals that devalue substantive technical expertise and proven product experience. Too often, advocates of "quality through Process" do not appreciate the skills, techniques, and tools required to build quality products. This is frequently manifested by an organizational reward structure that favors those who draw diagrams and write reams of useless documentation, rather than those who produce working software.

Clearly, a good development process (note the small "p") is essential to produce quality software for large-scale systems. However, software quality begins with quality software developers. Moreover, a software process should *facilitate* productivity, not destroy it! All too often, the quest for "zero defect" software quality leads to zero software. For example, one project I consulted on in the past several years adopted a heavyweight process model, only to have productivity completely evaporate. In the span of 12 months the team spent multiple person-years of effort attending endless meetings and writing extensive documents detailing their standard operating procedures, coding standards, requirement tracking models, etc. Unfortunately, relatively little time was spent discussing the essential software architecture and design issues or actually writing any software to review against the operating procedures and coding standards. At year's end, the technically creative members of the team had quit in frustration and the project was completely gutted once corporate headquarters recognized that significant technical progress had not occurred nor was likely to occur.

I believe the quest for panaceas from software Process is alluring because it appears to absolve us from having to address the tough development issues. Chief among these is capturing and articulating the abstractions, patterns, and components required to build usable and reusable software in complex domains such as avionics, business data processing, telecommunications, on-line transaction processing, and distributed communication systems. My prognosis is that "death through quality" will claim many more victims before this disease is cured.

Trait 2: Infrastructure Churn

Another trait of unsuccessful software development organizations is what I call "infrastructure churn." This is characterized by an inability to commit to particular set of software tools, platforms, and programming languages long enough to be productive. For example, one company I worked with switched their GUI and OS platform four times within the span of nine months. The selection of infrastructure was made by non-technical managers who fundamentally did not understand their business market. In addition, they did not appreciate the implications of their fickleness, where each shift caused significant perturbation in the technical staff. Skill sets became rapidly obsolete and team members were continuously climbing new learning curves and attending new training classes. Not surprisingly, productivity dropped sharply since it was impossible to establish any momentum. Ironically, by time the infrastructure "stablized" the good developers had left to join other organizations. It turned out that the cross-platform skills they'd developed paid off handsomely, but not for the original company! I suspect that the advent of the World Wide Web and JAVA will give rise to a whole new level of infrastructure churn for the next few years.

Trait 3: Disrespect for Quality Developers

Many unsuccessful software organizations have the misconception that developers are "unskilled labor" who are interchangeable and can be replaced easily. I've seen this attitude most frequently at companies where technology and computing is viewed as a "back office" function or as a "necessary evil" required to remain competitive. In these environments, software developers are often referred to as "coders," which implies that programming is a rote, mechanical activity. Another manifestation of this trait occurs in organizations where technical hiring decisions are not made by the technical staff. I've worked with several companies that operated by the general principle of "least expensive employees get the job." Not surprisingly, this resulted in an environment with high turnover rates and an inability to produce high quality and maintainable software architectures, designs, and implementations. As usual, you get what you pay for. The irony of this situation, however, is that companies are often willing to pay a mucher large aggregate amount of money for armies of low-cost, low-productivity software developers, rather than a smaller aggregate amount of money for a few high-high, high-productivity developers. This problem is exaccerbated when compensation to manager is based on the number of developers who report to them, which encourages ``empire building.''

I predict that as software systems become increasingly complex that companies who recognize and reward quality software developers will increasingly outperform those who don't.

Trait 4: Analysis paralysis

Analysis paralysis stems from excessive focus on methods and modeling and not enough focus on architecture and prototyping. This common trait of unsuccessful software organizations occurs when analysis becomes an end in itself, rather than a means to an end. Analysis paralysis is less trendy than "Death through quality," but more pervasive. This trait is easy to detect since it is manifested by endless design meetings, mammoth documents containing intricate class models and object interaction diagrams, and a complete lack of working software or prototypes. Ironically, analysis paralysis can be a defensive reaction to software infrastructure churn. The project I described above that kept switching GUI and OS platforms was also plagued with analysis paralysis. Many developers became passive-aggressive after being burned repeatedly by changes in the infrastructure. They refused to spend time learning new frameworks and tools because they knew the effort would be wasted when the next upheaval hit.

Trait 5: The NIH Syndrome

Despite all the hype about the power of OO reuse, the "Not Invented Here" (NIH) syndrome is a social force that remains a major impediment to software productivity. The NIH syndrome occurs when developers try to build sophisticated software systems from scratch, rather than leveraging existing efforts. This syndrome is often rationalized by the belief that controlling the entire software system is more economical, more productive, and more efficient. Unfortunately, modern software systems have become so complex, and so reliant on advanced GUI, networking, database, and standard library components, that it's nearly impossible to create sufficiently robust infrastructure *and* solve domain problems in a timely manner. Moreover, if the product is never built, arrives late to market, or has low quality, the financial benefits often don't occur at all.

An example of this "penny wise, but pound foolish" trait occurred on a large-scale distributed system project I consulted on in the past several years. The company had tried to save money on licensing costs by developing most of their distributed components in house. They essentially recreate an ORB from scratch. Unfortunately, this was counter-productive since the effort and expense required to develop these components internally was actually much higher than the licensing costs! In addition, although the development team had a great deal of domain expertise, they had little distributed system expertise. Therefore, their eventual solution wasn't robust, which made maintenance and administration prohibitively expensive.

The five traits discussed above certainly don't exhaust the subject of dysfunctional organizations, but they do convey the essence of key problems I've observed in unsuccessful software projects. You may have noticed that I didn't focus on the role of C++ in project success or failure. This was intentional. No language (or platform, or design method, or framework) by itself can rescue projects plagued with the dysfunctional traits I presented above. I've certainly found that C++ can be used to produce high quality software, but it's important to keep things in perspective.

So what can we do to address these organizational problems? Unfortunately, there are no easy answers. It can be hard, if not impossible, to solve these problems on your own. One of the most important lessons I leared from my sociological training is that social forces can be much more powerful than individual forces. I've certainly felt frustrated working on projects that seemed hell-bent on self-destruction. Not surprisingly, another characteristic of unsuccessful projects is that their leaders are not receptive to external input and do not heed early warning signs of trouble.

In spite of these limitations, it's still worthwhile to keep an eye out for signs of these dysfunctional traits in your own organizations. You may be able to help your team and your company avoid some of these traps and pitfalls, or at least help to alleviate the consequences of these social forces for your own development team. Moreover, by keeping yourself informed (both technically and sociologically), you'll greatly increase your ability to navigate successfully through dysfunctional organizations.

Back to C++ Report Editorials home page.