The Netscape Development Process

Tinderbox was originally written at Netscape to support their software
development model. (Now fthe original Netscape development effort has
split into both the Netscape and Mozilla organization I will refer to
their efforts as if they were a single company called Netscape.  This
split does not effect the discussion of their processes.) Tinderbox
does not require that a company closely follow this model but the full
set of features are easily explained by the Netscape organization.
The implied "tinderbox development process" its a variant of
the "daily build" idea which is common in many companies.

At Netscape developers are geographically dispersed and often
telecommute from foreign countries.  There are hundreds of developers
working on a given product and this can lead to difficult
communication problems.  Developers get feature enhancement requests
and bug reports via a bug ticketing system.  The tickets are
prioritized by a Project Management Group call "Drivers".  The
developers work on the highest priority tickets in their queue and
when the changes are ready they look to the tinderbox web page to see
the current state of the source code.  If Tinderbox shows that some
tests do not work or code does not not compile then the developer must
wait until tinderbox indicates that everything is working. When a
developer checks in code tinderbox must verify that it compiles on all
architectures and all tests pass before a developer can leave that
night.

Should a programmer ever check in code which does not compile or does
not pass all tests then he is responsible for fixing the problem
immediately.  By looking at the tinderbox history it is easy to see
approximately the time the problem was introduced.  This limits the
set of possible changes which caused the problem to a small fraction
of the changes introduced that day and limits the number of possible
programmers who could have made this mistake to a small number.  Thus
the scope of the problem is clearly bounded in time, in source files
and in possible suspects.  If another developer was to check-in
additional changes at this time it would only confuse the issue and
possibly entangle the problem with other changes. No further checkins
are allowed until the problem is fixed.  All this information is
clearly visible to all developers and management on the tinderbox
web-page. When a problem occurs developers must quickly post a message
explaining that they are actively fixing the problem.  Since
development can not proceed until the problem is fixed it is in
everyone interest to solve problems quickly.

When all problems are cleared then the developer looks at the "tree
state" this indicates the management policies in effect.  At Netscape
the convention is that "Open" means checkins are allowed, "Closed"
Means checkins are allowed only with QA approval and "Restricted"
means that changes can only be made with Drivers approval.  The
approving person must be noted in the check-in comments along with the
bug ticket number which this fix was intended to address. Tree states
change according to managements need.  The restricted state is used to
ensure coordination between developers and the drivers group.  The
drivers group monitors the Tinderbox web-page to ensure that proper
check-in procedures are being carried out through out the day.

Usually the tree is "Open", there are some predictable times when the
tree enters another state.  Each morning the QA group closes the tree
and reopens it when they have verified the quality of the previous
days changes.  The Drivers group will restrict the tree for several
days when a major release of the product is imminent. Occasionally the
drivers group may need to restrict the ability of checkins
temporarily, like when a group intends to merge a large development
branch into the main branch.  During a complex merge it is important
that developers not involved in the merge do not check in their
changes and confuse the merging group.  Merging can take a few hours
to complete and it is easiest to do when the source code tree is
frozen except for use by the merging group.  At Netscape large merges
are typically done right after QA opens the tree.

By having the management policy clearly stated at the top of the
Tinderbox page everyone is aware of the current check-in requirements
and these can be quickly changed and the results communicated to all
developers if needed.  If there is a sudden need for an unexpected
policy change, it is still expected that every developer will know the
current policy at the moment they try to check code in.  In Tinderbox2
the tree state is also indicated with every check-in so it is possible
to look back at the history and see if George's last check-in was done
before or after the tree closed.

Each morning QA closes the tree and begins testing all of last nights
changes.  To ensure that QA has the support it needs to quickly finish
this work QA has a list of all the programmers who made changes
yesterday and a list of the changed files and check-in comment.  This
list of programmers is refered to as "The hook" and management
considers it imperative that everyone 'on the hook' be ready each
morning to fix or back out their changes.  When QA runs their daily,
mostly manual, tests no further checkins can occur till QA is
satisfied that the new code is at least as good as the old
code. Checkins are only made at QA's request to fix problems which
arise during testing.  Any problems which QA flags must be fixed or
backed out before the day's development can begin. Once QA approves 
daily build all programmers are expected to synchronize their code
with the latest known good sources. QA maintains a database of tree
opening times, so that it is easy to find older source code which has
been approved by QA. The "hook" is simply the list of all checkins
since the last time the tree was opened.

The focus for daily testing is on whether the set of day's changes is
good enough to be accepted into the main branch.  This is a separate
QA effort than the reporting of "bugs" which may take hours of QA time
to find and weeks of developer time to fix.  When QA is ready the tree
is opened and the day's development can begin. If there is ever a
serious problem with the development code it is always possible to
check out the version of the code at the time when the tree was opened.
This is the build which QA approved and is the last know good.

So there are daily QA milestones which ensure that the code works, and
that all developers can move forward.  The tinderbox tool monitors the
progress of all effort between milestones and reports what is going on
on a short time scale.  There are additional QA milestones which are
project dependent.  The longer term focus of these other milestones is
not an issue for the tinderbox system.