An earlier version of this article appeared in the C++ Report magazine, January 1999.

Introduction

• Portability, to reduce the effort required to support applications across heterogeneous OS platforms, programming languages, and compilers
• Flexibility, to support a growing range of multimedia datatypes, traffic patterns, and end-to-end quality of service (QoS) requirements
• Extensibility, to support successions of quick updates and additions to take advantage of new requirements and emerging markets
• Predictability and efficiency, to provide low latency to delay-sensitive real-time applications and high performance to bandwidth-intensive applications and
• Reliability, to ensure that applications are robust, fault tolerant, and highly available.

During the past decade, my colleagues and I have worked with hundreds of telecommunication, aerospace, and medical companies and written millions of lines of code while developing widely reusable middleware components and frameworks [Schmidt:01,Schmidt:97] for networked applications. We've also had the opportunity to document several dozen patterns [Schmidt:00] and architectures [POSA:96] that guide the design of these components and frameworks. In addition, we've taught hundreds of tutorials and courses on these topics for thousands of developers and students. In spite of formidable non-technical and technical challenges, we've identified a solid body of knowledge, experience, and software artifacts that can significantly enhance the systematic reuse [Griss:96] of networked application software.

In this article, I outline common reasons why reuse has failed in the past. I then discuss proven steps that organizations, projects, and individuals can take to avoid these traps and pitfalls. The key themes pervading this article are:

• The principles, methods, and skills required to develop reusable software cannot be learned effectively by generalities and platitudes. Instead, developers must learn concrete technical skills and gain hands-on experience by developing and applying reusable software components and frameworks in their daily professional practice.

• To succeed in-the-large, reuse efforts must address both technical and non-technical issues. Focusing on non-technical issues alone neglects important skills-building education that must occur to succeed with non-trivial reuse efforts. Likewise, focusing solely on technological issues neglects the deep impact that social and economic forces and processes have on the success or failure of technology adoption.

• It's easier and more cost effective to develop and evolve networked applications by basing them on reusable distributed object computing middleware, which is software that resides between applications and the underlying operating systems, network protocol stacks, and hardware. Middleware's role is to functionally bridge the gap between application programs and the lower-level hardware and software infrastructure in order to (1) coordinate how parts of applications are connected and how they interoperate and (2) enable and simplify the integration of components developed by multiple technology suppliers.

Why Software Reuse has Failed Historically

Like many other promising techniques in the history of software, however, systematic reuse of software has not universally delivered significant improvements in quality and productivity. There have certainly been successes, e.g., sophisticated frameworks of reusable components are now available in OO languages running on many OS platforms. In general, however, these frameworks have focused on a relatively small number of domains, such as graphical user-interfaces or C++ container libraries like STL. Moreover, component reuse is often limited in practice to third-party libraries and tools, rather than being an integral part of an organization's software development processes.

In theory, organizations recognize the value of systematic reuse and reward internal reuse efforts. In practice, many factors conspire to make systematic software reuse hard, particularly in companies with a large installed base of legacy software and developers. In my experience, non-technical impediments to successful reuse commonly include the following:

• Organizational impediments -- e.g., developing, deploying, and supporting systematically reusable software assets requires a deep understanding of application developer needs and business requirements. As the number of developers and projects employing reusable assets increases, it becomes hard to structure an organization to provide effective feedback loops between these constituencies.

• Economic impediments -- e.g., supporting corporate-wide reusable assets requires an economic investment, particularly if reuse groups operate as cost-centers. Many organizations find it hard to institute appropriate taxation or charge-back schemes to fund their reuse groups.

• Administrative impediments -- e.g., it's hard to catalog, archive, and retrieve reusable assests across multiple business units within large organizations. Although it's common to scavenge small classes or functions opportunistically from existing programs, developers often find it hard to locate suitable reusable assets outside of their immediate workgroups.

• Political impediments -- e.g., groups that develop reusable middleware platforms are often viewed with suspicion by application developers, who resent the fact that they may no longer be empowered to make key architectural decisions. Likewise, internecine rivalries among business units may stifle reuse of assests developed by other internal product groups, which are perceived as a threat to job security or corporate influence.

• Psychological impediments -- e.g., application developers may also perceive ``top down'' reuse efforts as an indication that management lacks confidence in their technical abilities. In addition, the ``not invented here'' syndrome is ubiquitous in many organizations, particularly among highly talented programmers.

I've also observed that developers often put too much faith in language features, such as inheritance, polymorphism, templates, and exception handling, as the primary means to foster reuse. Unfortunately, languages alone don't adequately capture the commonality and variability of abstractions and components required to build and systematically apply reusable software in complex application domains.

How to Make Systematic Reuse Work for You

• Component- and framework-based middleware technologies, such as CORBA, J2EE, and .NET, have become mainstream.

• An increasing number of developers of projects over the past decade have successfully adopted OO design techniques, such as UML and patterns, and OO programming languages, such as C++, Java, and C#.

Over the past decade, I've worked with many companies, including Boeing, Cisco, Ericsson, Iridium, Kodak, Lucent, Motorola, SAIC, Siemens, and Sprint, building reusable networked applications using OO design techniques and programming languages [Schmidt:00,Schmidt:01]. These projects have applied a range of reusable middleware tools including CORBA, the ACE framework [Schmidt:01], which is a C++ framework that implements many patterns for concurrent networked applications, and TAO [Schmidt:97], which is a high-performance, real-time implementation of CORBA that leverages the framework components in ACE.

The following discussion summarizes the lessons my colleagues and I have learned making systematic reuse work with ACE and TAO, which have been deployed in a wide range of commercial applications in many domains, including aerospace, data- and tele-communications, medical, process automation, simulation, and financial services. ACE and TAO are freely available, open-source software packages that consist of frameworks, components, examples, and regression tests containing over one million lines of C++ code. More than 100 person-years of effort have been invested to develop, validate, optimize, and document these reusable assets.

Although there is no magic methodology or process that's guaranteed to foster systematic reuse, I have personally seen the recommendations below applied successfully numerous times over the past decade on many projects at many companies around the world.

Strive to Create the Prerequisites for Successful Systematic Reuse

1. The market is competitive -- In a competitive business environment, such as financial services or wireless networking, time-to-market is crucial. It's therefore essential to leverage existing software to reduce development effort and cycle time. When a market is not competitive, however, organizations tend to reinvent, rather than reuse, software.

   The defense industry in the 1980's is a good example of the ``reinvent rather than reuse'' phenomenon. During the defense buildup in the Reagan administration, defense-related R&D funding was abundant. Not surprisingly, little systematic reuse was achieved, even though the Ada programming language was designed explicitly to support software reuse. Starting in the early 1990's, however, the substantial decrease in DoD funding had a powerful impact on the defense industry. Contemporary aerospace and defense contractors now aggressively seek to integrate "commercial-off-the-shelf" (COTS) software and hardware in order to remain competitive in their changing marketplace.

2. The application domain is complex -- Components that are relatively easy to develop, such as generic linked lists, stacks, or queues, are often rewritten from scratch rather than reused. In contrast, developers working in highly complex domains, such as distributed, real-time systems are often willing to reuse components, such as dynamic scheduling frameworks or real-time ORBs, when building equivalent solutions from scratch proves too error-prone, costly, or time-consuming.

3. The corporate culture and development process are supportive -- Not only is it hard to develop high-quality reusable components and frameworks, it's even harder to reap the benefits of reuse immediately. Significant investment must be expended up-front to produce efficient, flexible, and well-documented reusable software assets before they can be leveraged in subsequent generations of a product line. Therefore, organizations must support an appropriate software development process that allows systematic reuse to flourish.

   Ideally, an organization's software process should reward developers who invest the time and effort to build, document, and reuse robust and efficient components. For instance, a reward system could be built into project budgets, with incentives based on the number of software components reused by individuals or groups. I still find companies, however, whose processes measure programmer productivity solely in terms of the number of lines of source code written from scratch, which penalizes developers who attempt to reuse existing software.

4. Attractive ``reuse magnets'' exist -- To attract systematic reuse, it crucial to develop and support ``reuse magnets,'' [Foote:97] i.e., well-documented framework and component repositories. These repositories must be well-maintained so that application developers will have confidence in their quality and assurance that any defects they encounter will be fixed promptly. Likewise, framework and component repositories must be well-supported so that developers can gain experience through hands-on training and mentoring programs.

   In my experience, ``open-source'' development processes are an effective process for creating attractive reuse magnets. Open-source processes have yielded many widely used software tools and frameworks, such as Linux, Apache, GNU, ACE, and TAO. The open-source model allows users and developers to participate together in evolving software assets. One of the key strengths of this model is that it scales well to large user communities, where application developers and end-users can assist with much of the quality assurance, documentation, and support [Schmidt+Porter:01].

   As a general rule, software development does not scale up as the number of developers increases. The main problems stem from the increased human communication and coordination costs associated with large projects. A team of 10 good developers can therefore typically produce much better quality software systems with much less effort and expense than a team of 1,000 developers.

   In contrast, however, software debugging does scale up as the number of developers helping to debug the software increases. The main reason for this, of course, is that all other things being equal, having more people test the code will identify its ``error-legs'' much more quickly than having just a few people testing code. A team of 1000 testers will therefore typically find many more bugs than a team of 10 testers.

   Moreover, open-source development efforts tend to have short feedback loops between the point when a bug is discovered and the bug is fixed. This increases the incentive for the user community to help with the debugging process since they are ``rewarded'' by rapid feedback and fixes once bugs are identified. In addition, because the source code is available for inspection, developers in the user community can often help fix any bugs they find, which further amortizes the overall debugging effort and improves software quality rapidly.

5. Strong leadership and empowerment of skilled architects and developers -- I've observed that the ability of companies and projects to succeed with reuse is highly correlated with the quality and quantity of experienced developers and effective leaders. Conversely, reuse projects that lack a critical mass of developers with the necessarily technical and leadership skills rarely succeed, regardless of the level of managerial and organizational support.

   In general, the level of experience required to succeed with systematic reuse depends largely on whether programmers are trying to develop reusable components or to use them. I've found that developing reusable frameworks and components for complex domains, such as telecom or avionics, requires highly experienced and skilled architects and developers. These individuals must be trained and empowered to create, document, and support horizontal middleware platforms that reduce the effort required to develop vertical applications.

   In general, the more complex the domain, the greater the skills and leadership required to develop effective reusable middleware that can encapsulate complex communication protocols and mechanisms for concurrency, locking, persistence, fault tolerance, connection management, event demuxing, and service configuration. When middleware architects and developers are successful, they create component abstractions that hide these error-prone and tedious mechanisms and protocols. Application developers therefore needn't be as experienced with complex systems-level technologies since they can program to these higher-level component abstractions.

   I've also observed, however, that horizontal middleware platform efforts generally fail when application developers are (1) too inexperienced, (2) the domain is sufficiently challenging, and (3) the middleware team lacks sufficient training, resources, time, or empowerment to create a stable platform. It's therefore important that developers at all levels improve their technical skills and learn how to apply good software principles, patterns, and practices.

Support Iterative Development Processes and Grow Reuse Assets Incrementally

In my experience, many companies have a hard time getting even relatively small systematic reuse efforts to work right the first time. It's therefore essential to resist the urge to institutionalize ambitious corporate-level reuse initiatives before the people and the processes have reached the proper maturity. I've found that a much more effective process in practice is to iteratively grow reuse efforts out of smaller-scale successes that allow developers and managers to build the necessary skills and confidence.

For systematic reuse to succeed therefore, management must have the vision and resolve to support the incremental evolution of reusable software assets. Fred Brook's observation that ``Plan to throw the first one away, you will anyway'' [Brooks:75] applies as much today as it did 25 years ago. Increasingly, an 80% solution that can be deployed and evolved incrementally is preferable to waiting for a 100% solution that never ships. This phenomenon is referred to as the ``piecemeal growth'' model of software deployment, sometimes referred to by the more colorful name ``Worse is better'' [Gabriel:98]. According to this model, ``the best is the enemy of the good'' since striving for perfection makes it hard to hit tight market windows and gain initial market share.

Maintain a Close Feedback Loop Between Middleware Developers and Application Developers

In contrast, reuse efforts that try to work top-down, e.g., from high-level domain analysis, are highly likely to fail. The culprit is often the lack of close feedback loops between developers of reusable middleware and developers of applications. For instance, I've observed that it's usually counter-productive to create reuse teams that build middleware frameworks and components in complete isolation from application teams.

Since reuse efforts rarely have sufficient resources to please all possible customers, it's important to be goal-directed, rather than exhaustive, in determining which assets to develop, enhance, and maintain. Without intimate feedback from application developers, therefore, the software artifacts produced by component teams rarely address core business problems and won't be reused effectively.

It's also important that the relationship between application developers and reuse groups be mutually synergistic. For instance, reuse teams should be responsive to fix problems that inevitably arise in their middleware. Likewise, application developers should actively help to improve reusable artifacts, rather than waiting passively for the reuse team to find and fix all the problems. While it's possible to depend upon developer altruism and good will to achieve these goals, it's usually more effective to institutionalize a reward system with incentives to encourage effective relationships between developers and users.

Buy, Rather than Build, System Infrastructure and Middleware Integration Frameworks

• System infrastructure frameworks -- These frameworks simplify the development of portable and efficient system infrastructure, such as concurrent OO network programming toolkits [Schmidt:01] and frameworks for user interfaces and language processing tools. System infrastructure frameworks are primarily used internally within a software organization and not sold to customers directly.

• Middleware integration frameworks -- These frameworks are commonly used to integrate networked applications and components. Middleware integration frameworks are designed to enhance the ability of software developers to modularize, reuse, and extend their software infrastructure to work seamlessly in a distributed environment. There is a thriving market for middleware integration frameworks, which are rapidly becoming commodities. Common examples include CORBA, J2EE, .NET, and transactional databases.

• Enterprise application frameworks -- These frameworks address vertical application domains, such as avionics mission computing, call processing, and manufacturing, and are the cornerstone of enterprise business activities. Relative to system infrastructure and middleware integration frameworks, enterprise frameworks are expensive to develop and/or purchase. Enterprise frameworks can provide a substantial return on investment, however, since they directly support the development of end-user applications and products.

Develop Software Based on Architectures, Rather Than on Particular Middleware Technologies

For large-scale, long-lived networked applications it's essential to design and use architectures that transcend any specific technology or middleware standard. For instance, programming directly to a proprietary middleware API is risky since these APIs can rapidly become obsolete. I've therefore found it's more fruitful to develop networked applications based on a common architecture that can be instantiated on a range of middleware and OS platforms.

An effective pattern I've seen applied repeatedly to organize a common software architecture is to use (1) frameworks in the horizontal middleware platform layers and (2) components in the vertical application layers. Components are self-contained instances of abstract data types (ADTs) that can be plugged together to form complete applications. Common examples of components include COM+ controls and CORBA Object Services.

The relationship between frameworks and components is highly synergistic, with neither subordinate to the other. Frameworks can be used to develop components, whereby component interfaces provide Facades for internal class structures inside a framework. Moreover, components can be used as ``pluggable strategies'' within a framework.

Compared with frameworks, components are less tightly coupled and can support binary-level reuse more readily. For example, application developers can reuse components without having to subclass from existing base classes. In my experience, application developers are generally more comfortable and successful programming with components than they are customizing frameworks. Conversely, frameworks are useful for middleware teams because they help to simplify the development of horizontal platform software. Naturally, components can also be used to develop infrastructure and middleware.

Avoid One-dimensional ``Solutions'' Complex Software Development Problems

The urge to apply one-dimensional solutions to complex problems isn't limited to technologists, however. For instance, there is a school of thought that claims only the non-technical impediments to reuse are worth addressing since systematic reuse fails solely for economic and organizational reasons, not technological ones. According to this perspective, investing in education or training to improve the technical skills of developers is pointless because it has no impact on success.

Unfortunately, one-dimensional non-technical solutions are no better than one-dimensional technological solutions. Managerial and organizational support is certainly desirable and essential for large-scale adoption of systematic reuse across an enterprise. In my experience, however, this support is not sufficient, nor even always necessary, to succeed with systematic reuse, particularly within smaller parts of organizations.

Moreover, focusing solely on organizational and economic impediments at the expense of technology and skills-building, can yield a corporate culture of ``learned helplessness.'' Developers suffering from this malady often postpone improving their design and reuse skills until the entire organization is ``cured.'' This approach is as futile as waiting for all the customer requirements to solidify before engaging in architecture and design phases.

Failing to invest in technology and education can greatly hamper a company's ability to compete effectively, particularly when time-to-market is crucial to success. It can also cause companies to become dangerously out of touch with contemporary software practice, where an increasing number of companies are in fact succeeding with systematic reuse and COTS technology adoption. Not surprisingly, many of the large companies that suffered the most during the economic downturn in 2001 were also companies that most strongly resisted adopting systematic reuse and COTS.

I therefore believe that we must not wait passively for organizational and economic problems to be resolved completely before building the technical skills and experience level of developers. Instead, we must initiate and support skills-building education now and sustain them over time. These skills are ultimately required to succeed with systematic reuse, in particular, and high-quality software development, in general.

Respect and Reward Top-Notch Developers and Architects

For systematic reuse to succeed, it's crucial that developers who build reusable assets are empowered to own, maintain, and enhance these assets. Ultimately, reusable components and frameworks are only as good as the people who build and use them. In my experience building complex networked applications, there's simply no substitute for hiring high-quality, experienced software developers. The key to success is to hire developers whose skills and knowledge of patterns and design principles have withstood the test of time.

In practice, of course, it's hard to find top-notch software developers. Ironically, many companies---particularly large ones---still treat their developers as interchangeable, ``unskilled labor,'' who can be replaced easily. I'm increasingly noticing, however, that companies who respect and reward their top-notch software developers consistently outperform those who don't.

Invest in Continual Training, Mentoring, and ``Hands-on'' Experience

Systematic reuse is largely a by-product of good designs and experienced developers. Education is crucial to help improve developers' design skills. Fortunately, developing good reusable software requires many of the same set of skills, such as knowledge of architectures, patterns, frameworks, and components, necessary to develop good software in general. The time and effort spent on education will pay off therefore, whether or not developers actually write reusable software artifacts.

Keep the Faith

Ultimately, organizations that attempt systematic reuse without providing an incubation environment will lose their faithful. Many of these faithful will be the most experienced developers or those most capable of coming up to speed quickly. In markets driven by ``Internet cycle times,'' the loss of valuable developers can devastate an organization's long-term competitive viability.

Keeping the faith requires keeping abreast of external R&D developments and global technology trends. In my travels throughout the software industry, I am continually amazed by the rate at which reuse and COTS middleware is being adopted in many businesses and application domains. I suspect that the pundits who dismiss reuse as a myth simply haven't spent enough time ``in the trenches'' lately to recognize the speed at which the software industry is moving away from programming applications from scratch to integrating applications out of reusable frameworks and components.

Concluding Remarks

www.cs.wm.edu/~schmidt/patterns.html
www.cs.wm.edu/~schmidt/ACE.html
www.cs.wm.edu/~schmidt/corba.html
www.cs.wm.edu/~schmidt/TAO.html

I want to stress, however, that these technological solutions alone are not silver bullets [Brooks:87]. I firmly believe the promise of systematic reuse for networked applications will not be fully realized until we address both technical and non-technical impediments effectively. I also strongly believe, however, that there's no virtue in waiting for organizational and managerial maladies to be resolved completely before improving the education and experience of software developers. Fortunately, most software professionals are eager to hone their technical skills, so future impediments to successful reuse will be largely self-imposed.

Acknowledgments

References

[Brooks:87] Frederick P. Brooks, ``No Silver Bullet: Essence and Accidents of Software Engineering,'' IEEE Computer, Volume 20, Number 4, April 1987, 10-19.

[Foote:97] Brian Foote and Joseph Yoder, ``The Selfish Class,'' in Pattern Languages of Program Design 3, eds. Robert C. Martin, Dirk Riehle, and Frank Buschmann, Addison Wesley, 1997.

[Gabriel:98] Richard P. Gabriel, ``Patterns of Software, Tales from the Software Community,'' Oxford University Press, 1998.

[GoF:95] Gamma et al., Design Patterns: Elements of Reusable Object-Oriented Software, Addison-Wesley, Reading, MA, 1995.

[Griss:96] Martin Griss, Systematic Software Reuse: Architecture, Process and Organization are Crucial, Fusion Newsletter, Oct 1996.

[POSA:96] Buschmann et al., Pattern-Oriented Software Architectures, Wiley & Sons, 1996.

[Schmidt:97] Douglas C. Schmidt, David Levine, and Sumedh Mungee, ``The Design and Performance of Real-Time Object Request Brokers,'' Computer Communications, Volume 21, No. 4, April, 1998.

[Schmidt+Fayad:97] Douglas C. Schmidt and Mohamed Fayad, ``Object-Oriented Application Frameworks,'' Communications of the ACM, Special Issue on Object-Oriented Application Frameworks, Vol. 40, No. 10, October 1997.

[Schmidt:00] Douglas C. Schmidt, Michael Stal, Hans Rohnert, and Frank Buschmann, Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Objects, Wiley and Sons, 2000.

[Schmidt:01] Douglas C. Schmidt and Steve Huston, C++ Network Programming: Mastering Complexity with ACE and Patterns, Addison-Wesley, 2001.

[Schmidt+Porter:01] Douglas C. Schmidt and Adam Porter, Leveraging Open-Source Processes to Improve the Quality and Performance of Open-Source Software, 1st Workshop on Open Source Software Engineering, ICSE 23, Toronto, Canada, May 15, 2001.

Back to Douglas C. Schmidt's home page.