The Cathedral and the Bazaar | Page 6

due to duplication of work by debuggers
almost never seems to be an issue in the Linux world. One effect of a ``release early and
often'' policy is to minimize such duplication by propagating fed-back fixes quickly [JH].

Brooks (the author of The Mythical Man-Month) even made an off-hand observation
related to this: ``The total cost of maintaining a widely used program is typically 40
percent or more of the cost of developing it. Surprisingly this cost is strongly affected by
the number of users. More users find more bugs.'' [emphasis added].
More users find more bugs because adding more users adds more different ways of
stressing the program. This effect is amplified when the users are co-developers. Each
one approaches the task of bug characterization with a slightly different perceptual set
and analytical toolkit, a different angle on the problem. The ``Delphi effect'' seems to
work precisely because of this variation. In the specific context of debugging, the
variation also tends to reduce duplication of effort.
So adding more beta-testers may not reduce the complexity of the current ``deepest'' bug
from the developer's point of view, but it increases the probability that someone's toolkit
will be matched to the problem in such a way that the bug is shallow to that person.
Linus coppers his bets, too. In case there are serious bugs, Linux kernel version are
numbered in such a way that potential users can make a choice either to run the last
version designated ``stable'' or to ride the cutting edge and risk bugs in order to get new
features. This tactic is not yet systematically imitated by most Linux hackers, but perhaps
it should be; the fact that either choice is available makes both more attractive. [HBS]
How Many Eyeballs Tame Complexity
It's one thing to observe in the large that the bazaar style greatly accelerates debugging
and code evolution. It's another to understand exactly how and why it does so at the
micro-level of day-to-day developer and tester behavior. In this section (written three
years after the original paper, using insights by developers who read it and re-examined
their own behavior) we'll take a hard look at the actual mechanisms. Non-technically
inclined readers can safely skip to the next section.
One key to understanding is to realize exactly why it is that the kind of bug report
non-source-aware users normally turn in tends not to be very useful. Non-source-aware
users tend to report only surface symptoms; they take their environment for granted, so
they (a) omit critical background data, and (b) seldom include a reliable recipe for
reproducing the bug.
The underlying problem here is a mismatch between the tester's and the developer's
mental models of the program; the tester, on the outside looking in, and the developer on
the inside looking out. In closed-source development they're both stuck in these roles, and
tend to talk past each other and find each other deeply frustrating.
Open-source development breaks this bind, making it far easier for tester and developer
to develop a shared representation grounded in the actual source code and to
communicate effectively about it. Practically, there is a huge difference in leverage for
the developer between the kind of bug report that just reports externally-visible
symptoms and the kind that hooks directly to the developer's source-code-based mental
representation of the program.
Most bugs, most of the time, are easily nailed given even an incomplete but suggestive
characterization of their error conditions at source-code level. When someone among
your beta-testers can point out, "there's a boundary problem in line nnn", or even just
"under conditions X, Y, and Z, this variable rolls over", a quick look at the offending
code often suffices to pin down the exact mode of failure and generate a fix.
Thus, source-code awareness by both parties greatly enhances both good communication

and the synergy between what a beta-tester reports and what the core developer(s) know.
In turn, this means that the core developers' time tends to be well conserved, even with
many collaborators.
Another characteristic of the open-source method that conserves developer time is the
communication structure of typical open-source projects. Above I used the term "core
developer"; this reflects a distinction between the project core (typically quite small; a
single core developer is common, and one to three is typical) and the project halo of
beta-testers and available contributors (which often numbers in the hundreds).
The fundamental problem that traditional software-development organization addresses is
Brook's Law: ``Adding more programmers to a late project makes it later.'' More
generally, Brooks's Law predicts that the complexity and communication costs of a
project rise with the square of the number of developers, while work done only rises
linearly.
Brooks's Law is founded on experience that bugs tend strongly to cluster at the interfaces
between code