II. Preliminary Analysis & Discussion

One of the more significant and consistent results that we found in the course of our two years of interviews with software industry insiders was that software development is still considered a craft; that developers cannot control or manage the process effectively, and too often cannot balance the demands of innovation, time to market, quality and cost. Even leaders in software product development, such as Microsoft, still cannot accurately predict the time and cost to produce a major new product.

As a result of the prevalence of these issues in the minds of software executives and other software industry insiders, we decided to undertake a study of product development practices among software publishers. Over the summer quarter of 1995, software team researchers and a small group of graduate students developed a survey methodology and appropriate survey instruments. Our goal was to establish benchmarks, uncover best practices, and identify the key issues in software product management.

Software product management is harder than the development of many other products expressly because of its extreme flexibility. All software product development is R&D, with no tooling or manufacturing component. Because of this ability to go from development lab to market almost literally overnight, aberrant development practices have arisen and survived tenaciously, regardless of the acknowledgment of most parties about their negative effects on quality, reliability, morale, and on the management of the process generally. These practices include adding or dropping of features in the days before product shipping; allowing requirements and planning documents to become outdated; widespread understaffing of product testing, documentation, and quality control; and severely limited planning and budgeting practices.

Through our discussions with product managers, and subsequently through discussions with a carefully selected focus group composed of people working directly in software development, we created a pair of survey instruments, to be completed with reference to a particular software release: one questionnaire for the engineering manager and one for the marketing product manager involved in that release. The questionnaire investigates the range of practices in areas such as requirements formulation, release management, team communication, bug management, planning and testing, from the point of view of the two key people on the development team. We administered the survey to a pilot group 19 people in 11 local companies in two different segments of the software products industry.

The two market segments surveyed were relational database products and call center products. We chose these two segments for the pilot survey partly because we had greater access to local companies in these segments, and partly because we believed that they had different company and market characteristics that might influence their development practices. In general, the relational database segment is a larger, more mature market segment, with bigger companies dominating the field. We perceived the call center segment market as more rapidly changing, with many smaller players with fewer resources, that might therefore make different tradeoffs in the development process.

                                   RDBMS          Call Center       Total Firms              
Very Small (under $10M)              1              1                 2
                        
Small ($10-50M)                      2              4                 6
                        
Large (over $100M)                   3             --                 3
                        
Total                                6              5                11                       

We are still in the process of analyzing this pilot survey data, as well as deciding on what form a larger survey should take. As of this date, it should be noted that we have not yet developed adequate metrics for quality, project management style, timeliness to market, or project "success," but we did ask a number of questions related to testing, documentation, Quality Assurance department influence or involvement, schedules and time delays, and many other issues that we hope will eventually provide insights on best practices.

Some of our preliminary results are outlined below.

Tradeoffs. Software industry executives often discuss how the extremely rapid rate of change of their technologies and markets force them to make a conscious tradeoff among cost, quality, features, and time-to-market in the course of their software development. Our pilot survey probed this tradeoff process. We found that the marketing and engineering functions often favored different tradeoff strategies, based on their differing perceptions of both the market environment and end user expectations.

Marketing managers, for example, saw features and platform diversity to be the essence of the competitive edge. They were more willing to tradeoff the time it takes to do comprehensive testing in order to include additional features. They believed that their success in the marketplace was tied to market factors such as features, price and company reputation. They expected bug fixes to come out in dot releases or to be replaced by new functional releases; thus they avoided talking about quality as a competitive factor. Engineering managers, on the other hand, although very feature-oriented as a group, were about as likely to decide to drop (postpone) features (or reduce feature functionality) as they were to forego testing, in order to get the product out to market. Both functional areas agreed, however, that time-to-market was the over-riding consideration.

One indicator of the tradeoff between time-to-market and quality might be inferred from the timing of feature additions and deletions from a release. Almost half of all of the respondents were willing to add or delete a feature in the last few days before the creation of the "Golden Master" (in preparation for duplication and shipping), obviously leaving little time for testing of additions or correcting user documentation.

While we are still evaluating questions about the tradeoff process, we can already observe that for the companies included in the pilot survey:

Quality. Inadequacy of documentation and understaffing of the documentation function were the quality factors most frequently cited as a problem. A few teams outsourced the documentation task but still pointed to a lack of resources in that area. The quality assurance (QA) function was also sometimes outsourced. Both the documentation and QA areas were often consigned to a lower status than other members of the development team. For example, in several organizations, respondents said that QA had little influence on team activities, and that the QA manager reported to a lower level person than did the marketing or engineering manager.

Some companies charged their engineers with the primary responsibility for testing; QA was there only as a "sanity check" and to make sure that the product integrated with other products within and outside of the company. Other companies did not provide adequate testing time for the QA function (either by engineers or a separate QA group), in order to get products out to market quickly. A few companies believed that testing was not necessary at all because, for example, they were "only porting code to a new platform," not developing a new product. While the majority of respondents expressed concern for the quality of their products, no companies described practices that could consistently produce quality software. Further investigation is needed to identify best practices in high quality software development.

Planning. Marketing was far more likely to have a long-term plan for a product than was engineering. Their vision, however, was not always communicated to (or accepted by) the Engineering Department. Engineering managers not only said that they did not prepare a plan themselves, but often maintained that no long term plan existed. Marketing was also more likely to have a formal document outlining their plans for the particular product release in question.

Engineering typically had some formal written plans, such as milestone objectives, but few had comprehensive specifications covering all aspects of the development process. Even among the companies that had formal engineering plans, more than 60% had so many changes by mid-way through the project that plans were abandoned rather than updated. We have found some preliminary evidence that the more out-of-date plans became relative to actual activities, the more likely companies were to have noticeable schedule slippage. Without plan documents to refer to, project/process changes were hard to disseminate to all parties and to monitor developers progress toward the changed goals.

Planning horizons varied markedly from three months to 2 years. We found that no one in the pilot had a long-term vision beyond two years (although a revised question might bring this out better.) However, those with a longer-term perspective were more likely to plan releases more carefully and, within a plan specifying multiple releases, were also somewhat more likely to have considered such long-term issues as software reuse. From the engineering side, companies with long-term outlooks were also more likely to have someone they identified as a chief architect who was (partially) responsible for the long-term vision (along with a marketing person).

We looked at what percentage of the total project time was spent in planning, development and testing. For the majority of companies in the pilot, companies that spent more time on planning spent relatively less time on testing. However, their ability to meet their own market introduction schedules did not necessarily correlate with how much planning they did on that release.

We thought that perhaps larger companies with more resources to devote to planning would do more planning, but we found little evidence that they did any more planning than their smaller counterparts. There was also surprisingly little correlation between the amount of planning and the relative complexity of the project.

One aspect of planning is creating a formal project budget. Only 18% of companies had a formal project budget or staffing plan. They also usually did not have any idea of whether their project expenditures had been reasonable or in line with their original plans. A few companies, however, said that each department had a budget within which they worked across all product development projects.

Teams. Decision-making and conflict resolution styles differed substantially among companies. For example, we looked at teams to see who dominated meetings and decision-making. We found that 55% of the companies surveyed were engineering-dominated, 27% were senior management-dominated, and 18% were consensus-driven. Companies complaining of low morale or problems with delays or product quality were more likely to characterize themselves as teams in which senior management resolved conflicts and made the most important decisions, or in which engineering dominated all aspects of the process.

We are just beginning to formulate the characteristics of a team that might be judged to be a better functioning group: "democratic" teams with high morale, who value the contribution of the marketing function, and who are more likely to seek a consensus between marketing and engineering for important decisions. Future studies will investigate this issue further. A "democratic" team, as we are currently defining it, was one in which representatives for all departments participated at most/all stages of the development process (including representatives from senior management, engineering, marketing, QA, release management, technical communications and technical support), and in which influence, decision-making and conflict resolution were handled at the team level rather than by senior management.

Correlations. We have begun to try to correlate survey results with salient characteristics of the company. So far we have not found an obvious correlation between product development practices at firms of any particular size. The complexity of the development process has also not been a meaningful indicator yet of such factors as the team composition or carefulness of planning. All companies surveyed believed that their product was adequately successful, and without more research to verify their true success, we have not yet been able to show that any particular practices lead absolutely to more success in the marketplace.

Eisenhardt and Tabrizi have already shown that companies can correctly identify their market window, and that those that manage to get a product to market within that window, enjoy a 33% greater return over 5 years than those who miss the window that they identified. Thus, our results might end up being construed as a confirmation that companies that trade off everything against time-to-market are likely to be employing the optimal strategy. We still need to investigate further, however, whether companies that can make marginal improvements in their development process might enjoy other benefits, such as higher quality, with no additional delays in getting their products to market.

III. Future Research

We are currently in the process of developing a methodology and design for a larger survey of the product development practices of software publishers. From our analysis of the pilot survey data, we have identified which issues are most promising for a broader investigation. Our plan is to solicit responses to several small questionnaires, with administration and data collection done via an on-line questionnaire on the SCIP Web site. A prototype of this Web page was developed last summer and used for data entry from the pilot.

Our focus will first be to extend the data to many more segments of the software products market. We will then take the same issues, with a revised questionnaire, to the world of in-house software project management, as well as software service providers, and embedded software development. We expect to find major differences in project management practices. Administering the questionnaires on-line may allow us to look across regional boundaries as well.

For more information, contact barr@cs.stanford.edu or tessler@cs.stanford.edu