- Ward, Jonathan H.
Countdown to a Moon Launch.
Cham, Switzerland: Springer International, 2015.
ISBN 978-3-319-17791-5.
-
In the companion volume,
Rocket Ranch (December 2015),
the author describes the
gargantuan and extraordinarily complex infrastructure which was built
at the Kennedy Space Center (KSC) in Florida to assemble, check out, and
launch the Apollo missions to the Moon and the Skylab space station.
The present book explores how that hardware was actually used,
following the “processing flow” of the
Apollo 11 launch vehicle and spacecraft from the
arrival of components at KSC to the moment of launch.
As intricate as the hardware was, it wouldn't have worked, nor would
it have been possible to launch flawless mission after flawless mission
on time had it not been for the management tools employed to
coordinate every detail of processing. Central to this
was
PERT
(Program Evaluation and Review Technique), a methodology
developed by the U.S. Navy in the 1950s to manage the Polaris
submarine and missile systems. PERT breaks down the progress of
a project into milestones connected by
activities into a graph of dependencies. Each activity
has an estimated time to completion. A milestone might be, say,
the installation of the guidance system into a launch vehicle.
That milestone would depend upon the assembly of the components
of the guidance system (gyroscopes, sensors, electronics, structure,
etc.), each of which would depend upon their own components.
Downstream, integrated test of the launch vehicle would depend
upon the installation of the guidance system. Many activities
proceed in parallel and only come together when a milestone
has them as its mutual dependencies. For example, the processing and
installation of rocket engines is completely independent of
work on the guidance system until they join at a milestone where
an engine steering test is performed.
As a project progresses, the time estimates for the various
activities will be confronted with reality: some will be
completed ahead of schedule while other will slip due to
unforeseen problems or over-optimistic initial forecasts.
This, in turn, ripples downstream in the dependency graph,
changing the time available for activities if the final
completion milestone is to be met. For any given graph
at a particular time, there will be a
critical
path of activities where a schedule slip of any one
will delay the completion milestone. Each lower level milestone
in the graph has its own critical path leading to it. As
milestones are completed ahead or behind schedule, the
overall critical path will shift. Knowing the critical path
allows program managers to concentrate resources on items
along the critical path to avoid, wherever possible, overall
schedule slips (with the attendant extra costs).
Now all this sounds complicated, and in a project with the scope of
Apollo, it is almost bewildering to contemplate. The Launch Control
Center was built with four firing rooms. Three were outfitted with
all of the consoles to check out and launch a mission, but the fourth
cavernous room ended up being used to display and maintain the PERT
charts for activities in progress. Three levels of charts were
maintained. Level A was used by senior management and contained
hundreds of major milestones and activities. Each of these was
expanded out into a level B chart which, taken together, tracked
in excess of 7000 milestones. These, in turn, were broken down
into detail on level C charts, which tracked more than 40,000
activities. The level B and C charts were displayed on more than
400 square metres of wall space in the back room of firing room four.
As these detailed milestones were completed on the level C charts,
changes would propagate down that chart and those which affected
its completion upward to the level A and B charts.
Now, here's the most breathtaking thing about this: they did
it all by hand! For most of the Apollo program, computer
implementations of PERT were not available (or those that existed
could not handle this level of detail). (Today, the PERT network
for processing of an Apollo mission could be handled on a laptop
computer.) There were dozens of analysts and clerks charged with
updating the networks, with the processing flow displayed on an
enormous board with magnetic strips which could be shifted around
by people climbing up and down rolling staircases. Photographers
would take pictures of the board which were printed and distributed
to managers monitoring project status.
If PERT was essential to coordinating all of the parallel activities
in preparing a spacecraft for launch, configuration control was
critical to ensure than when the countdown reached T0, everything
would work as expected. Just as there was a network of dependencies
in the PERT chart, the individual components were tested, subassemblies
were tested, assemblies of them were tested, all leading up to
an integrated test of the assembled launcher and spacecraft. The
successful completion of a test established a tested configuration
for the item. Anything which changed that configuration in
any way, for example unplugging a cable and plugging it back in,
required re-testing to confirm that the original configuration had
been restored. (One of the pins in the connector might not have
made contact, for instance.) This was all documented by paperwork
signed off by three witnesses. The mountain of paper was
intimidating; there was even a slide rule calculator for estimating
the cost of various kinds of paperwork.
With all of this management superstructure it may seem a miracle
that anything got done at all. But, as the end of the decade
approached, the level of activity at KSC was relentless (and took
a toll upon the workforce, although many recall it as the most
intense and rewarding part of their careers). Several missions
were processed in parallel: Apollo 11 rolled out to
the launch pad while Apollo 10 was still en route
to the Moon, and Apollo 12 was being assembled and
tested.
To illustrate how all of these systems and procedures came
together, the author takes us through the processing of
Apollo 11 in detail, starting around six months
before launch when the Saturn V stages, and
command, service, and lunar modules arrived independently
from the contractors who built them or the NASA facilities
where they had been individually tested. The original
concept for KSC was that it would be an “operational
spaceport” which would assemble pre-tested components
into flight vehicles, run integrated system tests, and then
launch them in an assembly-line fashion. In reality, the
Apollo and Saturn programs never matured to this level, and
were essentially development and test projects throughout.
Components not only arrived at KSC with “some assembly
required”; they often were subject to a blizzard of
engineering change orders which required partially disassembling
equipment to make modifications, then exhaustive re-tests to
verify the previously tested configuration had been restored.
Apollo 11 encountered relatively few problems in
processing, so experiences from other missions where problems
arose are interleaved to illustrate how KSC coped with
contingencies. While Apollo 16 was on the launch
pad, a series of mistakes during the testing process damaged
a propellant tank in the command module. The only way to repair
this was to roll the entire stack back to the Vehicle Assembly
Building, remove the command and service modules, return them to
the spacecraft servicing building then de-mate them, pull the
heat shield from the command module, change out the tank,
then put everything back together, re-stack, and roll back to the
launch pad. Imagine how many forms had to be filled out. The launch
was delayed just one month.
The process of servicing the vehicle on the launch pad is described
in detail. Many of the operations, such as filling tanks with
toxic hypergolic fuel and oxidiser, which burn on contact, required
evacuating the pad of all non-essential personnel and special
precautions for those engaged in these hazardous tasks. As
launch approached, the hurdles became higher: a Launch Readiness
Review and the Countdown Demonstration Test, a full dress rehearsal
of the countdown up to the moment before engine start, including
fuelling all of the stages of the launch vehicle (and then de-fuelling
them after conclusion of the test).
There is a wealth of detail here, including many obscure items I've
never encountered before. Consider “Forward Observers”.
When the Saturn V launched, most personnel and spectators were kept
a safe distance of more than 5 km from the launch pad in case of
calamity. But three teams of two volunteers each were
stationed at sites just 2 km from the pad. They were charged with
observing the first seconds of flight and, if they saw a catastrophic
failure (engine explosion or cut-off, hard-over of an engine gimbal,
or the rocket veering into the umbilical tower), they would signal
the astronauts to fire the launch escape system and abort the
mission. If this happened, the observers would then have to dive into
crude shelters often frequented by rattlesnakes to ride out the
fiery aftermath.
Did you know about the electrical glitch which almost brought the
Skylab 2
mission to flaming catastrophe moments after launch? How lapses in
handling of equipment and paperwork almost spelled doom for the
crew of Apollo 13? The time an oxygen leak while fuelling
a Saturn V booster caused cars parked near the launch pad to
burst into flames? It's all here, and much more. This is an
essential book for those interested in the engineering details of
the Apollo project and the management miracles which made its
achievements possible.
January 2016 