Open Source Software Development Jim Herbsleb ISRI Wean 1321 +1 412 268 8933 jdh@cs.cmu.edu

Geographically Distributed Development Extremely slow Hampered by communication and coordination problems Needs to make extensive use of collaboration technology, e.g., application sharing, shared calendars, teleconferences, chat and IM Requires extensive use of “coordination mechanisms” such as interface specifications, plans, processes Must carefully design division of labor across sites Very difficult to respond to unanticipated events

Open Source Challenge Fundamentally different model of software development How does it really work? What sort of process results from open source principles? What are the properties of the software developed this way? Case study of Apache and Mozilla (with Audris Mockus and Roy Fielding) Research issues in open source The open source challenge -- a fundamental challenge to the economics and organizational forms of traditional software development. Clearly has had some major successes: Linux, Apache, bind, etc. Properties very different from commercial development Built by large numbers of volunteers work not assigned, rather, individuals choose no explicit design no plan, schedule, processes, deliverables What sort of process results from this radical departure? What is the result? How good is the software, how efficient is the process?

Empirical Research Questions How many people wrote code for new functionality? How many people reported problems? How many people repaired defects? Did large numbers of people participate somewhat equally in these activities, or did a small number of people do most of the work? Where did the code contributors work in the code? Was strict code ownership enforced on a file or module level? What is the productivity of OSS developers? What is the defect density of OSS code? How long did it take to resolve problems?

Empirical Methods - 1 Sources: Output: CVS updates, BUGDB numbers Mail archives for ~3 years CVS/BUGDB/developer discussions Core group (about 12 people at any time) have CVS commit privileges Output: CVS updates, BUGDB numbers CVS update record (MR) date, files touched, lines changed author of the change BUGDB tracking number (if it’s a problem fix) BUGDB tracking number record raiser, dates opened, closed resolution module related CVS updates

Empirical Methods - 2 Research questions required change measures Identified several “comparable” commercial projects number of deltas within order of magnitude developed over comparable period all had high reliability requirements Differences must be interpreted cautiously

Roles in Apache Development Size of the development community How many people wrote code for new Apache functionality? (no reference to problem report) 249 people, 6092 submissions How many people reported problems? 458 people, 591 reports that resulted in code change How many people repaired defects? 182 people, 695 fixes How was work distributed within the development community?

The cumulative distribution of contributions to the code base. All code contributions Fixes only The cumulative distribution of contributions to the code base. One figure, with delta or MR, new + fixes + A + B mention features and fixes are distinct sets of people, little overlap among high contributors Two Commercial projects (telecommunications)

Code Ownership Was strict code ownership enforced on a file or module level? No. Out of 42 “.c” files with more than 30 changes 40 had at least two developers making more than 10% of the changes 20 had at least four developers making more than 10% of the changes Use other means of coordinating changes core group members have mutual trust, contribute code to various modules as needed use discussion board

Productivity Compare sets of developers that produced 80% of the code in each application A-E: similar-sized commercial projects   Apache A B C D E KMR/ developer/ year .11 .03 .03 .09 .02 .06 List projects A code for wireless base station -- may be copying B optical network element C, D, E various OAM things done by internal contract house KLOC/ developer/ year 4.3 38.6 11.7 6.1 5.4 10

Defect Density Measures post release and post-feature test per KLOC added and per thousand Delta Measure Apache A C D E Post-release Defects/KLOCA 2.64 .11 0.1 0.7 Post-release Defects/KDelta 40.8 4.3 14 28 10 Post-feature test Defects/KLOCA * 5.7 6.0 6.9 Post-feature test Defects/KDelta 164 196 256 1 24 26 2.6 3.8 high density for Apache in post-release (between 1.5:1 and 26:1 in favor of commercial, high reliability software) Low density for Apache in post-feature test (between . 6:1 and 2:1 in favor of Apache) 1 9.5 2.9 1.5 5 1 .5 .4 .4 1 .25 .2 .16

Defect Resolution Time Larger fonts for legend one half of problems are resolved within: core 1 day most sites 3 days documentation 4 days major optional 15 days os 10 days

Hypotheses Hypothesis 1: Open source developments will have a core of developers who control the code base. This core will be no larger than 10-15 people, and will create approximately 80% or more of the new functionality. Hypothesis 2: For projects that are so large that 10-15 developers cannot write 80% of the code in a reasonable time frame, informal coordination will not suffice. Hypothesis 3: In successful open source developments, a group larger by an order of magnitude than the core will repair defects, and a yet larger group (by another order of magnitude) will report problems. Hypothesis 4: Open source developments that have a strong core of developers but never achieve large numbers of contributors beyond that core will be able to create new functionality but will fail because of a lack of resources devoted to finding and repairing defects. Hypothesis 5: Defect density in open source releases will generally be lower than commercial code that has only been feature-tested, i.e., received a comparable level of testing. Hypothesis 6: In successful open source developments, the developers will also be users of the software. Hypothesis 7: OSS developments exhibit very rapid responses to customer problems. We base this hypothesis both on our empirical findings in this case, and also on observations and common wisdom about maximum team size. The core developers must work closely together, each with fairly detailed knowledge of what other core members are doing. Without such knowledge they would frequently make incompatible changes to the code. Since they form essentially a single team, they can be overwhelmed by communication and coordination overhead issues that typically limit the size of effective teams to 10-15 people. 2. The fixed maximum core team size obviously limits the output of features per unit time. To cope with this problem, a number of satellite projects, such as Apache-SSL, were started by interested parties. Some of these projects produced as much or more functionality than Apache itself. It seems likely that this pattern of core group and satellite groups that add unique functionality targeted to a particular group of users, will frequently be adopted in such cases. In other OSS projects like Linux, the kernel functionality is also small compared to application and user interface functionalities. The nature of relationships between the core and satellite projects remains to be investigated; yet it might serve as an example how to break large monolithic commercial projects into smaller, more manageable pieces. We can see the examples where the integration of these related OSS products is performed by a commercial organization, e.g., RedHat for Linux, ActivePerl for Perl, and CYGWIN for GNU tools. 3-4 Many defect repairs can be performed with only a limited risk of interacting with other changes. Problem reporting can be done with no risk of harmful interaction at all. Since these types of work typically have fewer dependencies among participants than does the development of new functionality, potentially much larger groups can work on them. In a successful development, these activities will be performed by larger communities, freeing up time for the core developers to develop new functionality. Where an OSS development fails to stimulate wide participation, either the core will become overburdened with finding and repairing defects, or the code simply will never reach an acceptable level of quality. 6 In general, open source developers are experienced users of the software they write. They are intimately familiar with the features they need, and what the correct and desirable behavior is. Since the lack of domain knowledge is one of the chief problems in large software projects [7], one of the main sources of error is eliminated when domain experts write the software. It remains to be seen if this advantage can completely compensate for the absence of system testing. In any event, where the developers are not also experienced users of the software, they are highly unlikely to have the necessary level of domain expertise or the necessary motivation to succeed as an OSS project.

Research Questions Resource Allocation,Decision-Making How do key developers decide where to allocate their resources? User innovation model Personal reputation model Product needs model How do individual motivations sum to give the development its trajectory? Not quite a market, not quite a hierarchy, perhaps a network

Research Questions Understanding Current Limitations of OSS Product structure, architecture – comprehension and collaboration What does not get built? Developers only meeting own needs? Differences between developer/users and general users? Effective ways of incorporating requirements of non-developer users? Effects of scale With larger scale, will coordination needs force adoption of “commercial” development techniques? How to collaborate on “big” features? Possible to increase participation by non-core developers?

Research Questions Adoption and Patronage Commercial organizations need ways to assess risk of adopting open source Patronage creates new forms of virtual organization What effects on OSS culture, individual motivation, economic network? How will competitive pressures, business motivations affect development? Cause branching, fragmentation? Evolve toward joint ventures, away from community ownership?