« October 29, 2006 |
Main
| November 4, 2006 »
Friday, November 3, 2006
HotBits: Redundant Servers in Production
The fully redundant
HotBits generator and proxy server configuration is now in production. At the start of October, the initial
third generation HotBits generator entered service and the hosts on the server farm were configured to obtain random data from it, with the second generation HotBits machine acting as a backup. That machine has now been retired and its detector and radiation source transferred to a duplicate of the original third generation machine. The two HotBits generators have no components in common, are powered from different Uninterruptible Power Sources, and are connected to the DMZ segment of the
local network through separate interconnected Gigabit Ethernet switches, which are cross-connected to the redundant network interfaces on the server farm machines. (I haven't yet updated the network architecture drawing to reflect the installation of the second HotBits generator.)
The two HotBits generators (and any others which may be added later) are peers in every sense—there is no “active” or “backup”—were ten generators installed, they would collectively produce data at ten times the rate of one. The latest version of the HotBits server program, which is now available for downloading from the link in the last paragraph of the
How HotBits Works page, allows you to define, for each proxy server, two sets of data sources which are polled, in round-robin order, to obtain inventory when required. The first set is intended to be populated by primary generator machines: those with a radiation source and detector attached. If the data request can be satisfied by any of these machines, it will be. Should none of the first-tier machines respond to the request, the second team will be interrogated, also in round-robin order (i.e., starting with the next machine after that which satisfied the most recent request, considering the machines arranged in a circular list). The second tier usually consists of proxy servers on other hosts in the server farm; if all of the primary HotBits generators are down, a server may,
in extremis, “poach” the inventory of its peers to meet an immediate need. (Proxy servers always exclude themselves from the list even if named as sources. This allows a uniform configuration to be used across a server farm without the need for each server to have a special configuration naming every server save itself.)
Before putting the second HotBits generator into production, I spent a week running diagnostics on it, then building a 16 megabyte random data set and subjecting it to the same tests I used to certify the initial third generation HotBits machine. If you're interested, you can download the data sets and test results for the new server from its
archive directory. The files are in the same format as those documented in the
statistical testing of the first generator machine.
There are two keyboards and mice in this installation because, at least in Switzerland, it isn't possible to order a Dell
Optiplex 210L without them. The machines share a legacy monitor, which is switched as necessary between them through the elegant expedient of plugging its video cable into the desired machine. Fortunately, almost all administration of these machines can be done remotely, so there is rarely a need to use the directly connected peripherals.
Posted at
20:49
