Feedback from the TAP deployments led to
a dramatic improvement in quality as Microsoft
fixed bugs related to reliability and usability.
“We’d get admins telling us that certain UIs
didn’t make sense,” Hinrichs added. “Eventually
we got to the point where the Active Directory
role was so stable that every Monday we’re
updating our Windows development domain
with the Monday build. It’s in such good shape.
But you can only do that with 18 months of
deployment to validate that it’s ready.”
Realistically, Microsoft realizes that even
the most stringent beta test process won’t
uncover all bugs because some issues simply
don’t crop up until you’ve gotten the product
out in the real world. “We can’t predict everything,”
Hinrichs told me. “So the only way to
make sure is to deploy broadly. We built that
religion with Windows 2000/2003, and it’s the
mantra we live by for 2008.”
Microsoft IIS is another good example.
Microsoft has been incredibly aggressive
deploying IIS internally and externally, and
Microsoft.com has been running on the IIS
version in Server 2008 for years now. Microsoft
also pushed the IIS Go Live program to
customers as early as Beta 3. “The message
is simple,” Hinrichs said: “Deploy, deploy,
deploy.”
A Compartmentalized
Build Process That Flows
Internally, Microsoft has restructured the build
process for Server 2008 so that the process, like
the product itself, is more compartmentalized.
A Main OS build is created every day, as with
previous product versions, but the process
of getting revisions into that Main build is far
more granular than before.
Under the server roles group, for example,
you’ll see subgroups such as the AD team,
the Terminal Services team, the IIS team, and
about 20 others. The developmental lead on
the AD team checks in code at the server roles
level while inheriting code from above. After
that code is ready for broader consumption, it’s
checked into a higher branch and consolidated
back into the Main development tree. This process
is ongoing, obviously, and requires people
at each level who can be trusted to monitor the
quality and necessity of new code additions.
“There’s code flowing up and down the
tree nonstop,” Hinrichs said, “but we maintain
high quality at both levels by a set of quality
gates. These gates are looking at BVTs [build
verification tests], a battery of tests against subbuilds
that make sure something the AD guys
do doesn’t break other things or prevent other
teams from testing. Everything has to work.”
Microsoft also runs code quality tools
which look for security bugs, buffer overruns,
and anything else that might cause problems.
There are also code dependency checks-
some 40-odd layers of dependencies between
components, Hinrichs told me. “To maintain
the componentization of the OS, you have to
make sure you aren’t unwittingly breaking
dependencies.” The goal of all these tests is to
catch these issues far down on the tree so that
they affect the smallest possible group of developers.
“All these tools run overnight,” Hinrichs
said. “And we get status reports in the morning.
We can see where different teams are.”
Because of the componentization of the
development process with Server 2008, the
ship room strategy has also changed since
Windows 2003. “It’s more evolved now,” Hinrichs
said. “We don’t just have the main ship
room. Now we also have seven distributed
ship rooms, run by people who meet with the
people checking in code below them. They
all have daily meetings, as does the main ship
room. The main ship room’s agenda is simple:
Who in the seven distributed ship rooms is
ready to bring code up [the tree into the Main
build]?”
While the main ship room is still used for
triaging code bugs, many of these bugs are
now handled lower in the tree, so the main
ship room’s emphasis has changed somewhat.
“We communicate what the focus is, the
testing we’re doing, but we have to rely on
local expertise [lower in the tree],” Hinrichs
said. “It’s much more distributed now, with
more local ownership. The system is just so
big. As you can imagine, the people in the
middle tier have awful jobs, awful. They have
to work up and down the tree and end up
working their butts off. They have over 20
groups below them and me on the top. It’s a
very, very tough job.”
Looking Ahead
In future issues of Windows IT Pro and on our
SuperSite for Windows, I will continue this
behind-the-scenes look at the development of
Server 2008. Stay tuned for more information
about Microsoft’s internal build process for this
very complex product.
Register
