Fourmilog: None Dare Call It Reason

« September 30, 2006 | Main | October 4, 2006 »

Sunday, October 1, 2006

HotBits: Third Generation Server in Production

The third generation HotBits server is now in production. The first generation server used a modified handheld radiation detector and Krypton-85 source, with a Microsoft Windows-based interface program. The second generation server used a commercial, off-the-shelf detector illuminated by a Cæsium-137 check source, while retaining the original Windows server software. The third generation server uses the same commercial detector and radiation source, but replaces the Microsoft-based driver with a Linux application which serves both as the driver for HotBits generator machines and the proxy server for server farm hosts—a proxy server is simply a system which obtains its HotBits inventory from another machine on the local network instead of from a directly connected generator.

I have been planning to upgrade the HotBits generator for some time. The second generation generator ran on the same Compaq 386/20 (subsequently upgraded to a 80486/40 with a CPU swap kit) on which it has run since HotBits was launched more than ten years ago. The Compaq machine has been absolutely rock solid, but as it is now approaching its twentieth birthday, one begins to fret about the tail end of the bathtub curve. More immediately worrisome is that the machine runs Windows 95 which, in the Microsoft tradition, randomly hangs and crashes for no apparent reason, usually right after I depart on a trip, leaving the site unattended.

Until recently, my intention had been to use a Linux-based single board industrial control computer for the HotBits generator driver. A wide variety of these devices are available on the market, many of which have the required RS-232 serial and Ethernet ports integrated on the board and, booting from flash memory with low power consumption permitting convection cooling, no moving parts. One does, of course, have to cobble up a power supply, some kind of enclosure for the gizmo, and learn about the version of Embedded Linux and development tools used with the board.

But, upon further reflection, the option of simply using a Linux-based PC became more and more attractive. First of all, it would eliminate all of the do-it-yourself aspects from the project and provide a 100% commercial, off-the-shelf solution for folks who wished to build their own HotBits generator. All they'd need is a PC with a serial port and Ethernet interface capable of running Linux, the computer-interfaced radiation detector and a commodity radiation check source, and they'd be in business. Further, using a standard Linux PC environment would allow me to eliminate the distinction between HotBits generator machines and proxy servers and use the same software for both applications, differing only in which driver was used to obtain the random data. While a new PC would be substantially more expensive than a single board computer, many people have an old “retired” PC lying around or can obtain one for free from a friend with one cluttering up their basement. The HotBits generation software will run on the most modest of hardware; as I noted above it has been running on a 40 Mhz 80486 for the last ten years, so any machine on which you can manage to install Linux is likely to be suitable.

The HotBits generator works by timing nuclear decay events which the radiation detector signals to the computer by toggling the state of the “ring indicator” (RI/CE/125) pin on the RS-232 port. To obtain precise timing, this signal is monitored by a software timing loop in the HotBits generator software which runs with interrupts prevented; while this may make the generator machine briefly nonresponsive, that's no problem since the machine is dedicated to the application (and besides, interrupts are enabled after each interval measurement, so a queued interrupt will be quickly processed in any case).

On the original Windows 95 generator, the HotBits driver runs in privileged mode and can prevent interrupts and use I/O instructions at will. Linux applications normally run in protected mode and do not have access to these facilities, but there is a little-known system call, iopl, which allows a process owned by the super-user to raise its I/O privilege level, granting access to I/O instructions and allowing interrupts to be locked out. (This system call is Linux and i386 architecture specific but, hey, that's what we're running on!) Using the iopl facility avoids the need to install a special kernel driver for the serial port, which would substantially complicate the process of getting a HotBits generator running; with the driver embedded entirely in the application, you can load a standard Linux distribution “out of the box”, install the HotBits software, and you're random ready.

Because all of the potential candidate machines I considered turning into a dedicated HotBits generator were flaky in some regard or difficult to use with modern Linux distributions, I decided to install a new machine for the purpose. I chose a Dell Optiplex 210L entry-level business desktop PC with an Intel 2.8 GHz Celeron processor. This machine has vastly more CPU power, memory, and disc space than is required for the application, but this is pretty much what constitutes a bottom-feeder these days if you're looking for a professional office configuration with a full complement of built-in ports. Of course, you can't buy a desktop PC these days without paying the “Microsoft tax”, so I ordered the machine with the cheapest option: Windows XP Home Edition, which I immediately blew away when I repartitioned and formatted the hard drive to install Fedora Core 5 Linux. I would have preferred a machine with ECC memory, but wasn't prepared to pay the additional cost for such a system; when the machine arrived at Fourmilab, I booted Memtest86 and let it run for 80 passes (more than 44 hours) with no errors reported.

After gaining more experience with this new HotBits generator (which has run flawlessly so far), I intend to install a second identical machine and retire the Compaq generator, using its radiation detector and source on the new system. This will eliminate the last single-point failure in the Fourmilab server network, with each machine on the server farm able to obtain HotBits inventory from either of the redundant generator machines.

Concurrent with the launch of the new HotBits generator, I have also added a page to the HotBits Web tree which reports results from a variety of statistical tests for randomness on a large data set created with this generator. You can download the 16 megabyte data set used in these tests if you wish to apply your own tests or have an application for such a large random data set.

Posted at 12:54