« August 6, 2006 | Main | August 8, 2006 »

Monday, August 7, 2006

Fourmilab: New In-house Server in Production

The new Fourmilab in-house file and compute server went into production last week-end. With the exception of tape backups (which will be transferred when a tape changer for the new server arrives), all of the functions of the Sun in-house server installed in 1998 have now been assumed by the new machine, a Dell PowerEdge 2800 with two Intel Xeon 2.8 GHz dual-core CPUs, which (since each of the cores is itself “hyper threaded”) gives the equivalent of eight symmetric processors sharing 12 Gb of common memory. There are eight 300 Gb 10,000 RPM SCSI drives on two channels, all hot-swappable from the front panel, configured as a 7 drive RAID-5 array with a total capacity of 1.8 Terabytes with one hot spare which, when mounted, looks like this in a “df -h”: 
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             9.5G  404M  8.6G   5% /
/dev/sda10            1.4T  124G  1.2T  10% /home
/dev/sda5             190G  7.8G  172G   5% /server
/dev/sda7             9.5G  151M  8.9G   2% /tmp
/dev/sda6              24G  3.9G   19G  18% /usr
/dev/sda8             9.5G  309M  8.7G   4% /var
Yep, that's right—1.4 Terabytes on /home! There is a /server partition which is configured identically to that of machines in the server farm. This permits the in-house server to be the definitive copy of the public server content, serve as a testbed, and act as the hub from which updates are published onto the server farm with rdist.

The new server runs Fedora Core 5 Linux, which installed on the first attempt with no problems whatsoever. Although this machine, on the private local network, is protected by the firewall, I decided to install SELinux in “Enforcing mode” to gain experience with that package with the intent of eventually deploying it on the public servers. The presence of a /server partition configured identically to the server farm machines allows this machine, in extremis, to join the public server farm with only the change of its IP address and plugging its network cables into the DMZ hubs.

The machine is remarkably quiet. Even with the eight disc drives and twin dual-core processors, it makes a lot less racket than one of the PowerEdge 1800 “blade” servers in the server farm. This may be because the limited height of the rack servers means their fans need to spin much faster to pull the air through, but still the 2800 makes far less noise than the Sun server, which has the same number of disc drives and a small fraction of the memory and CPU capacity.

The fully qualified name of this server on the private local network is “ceres.lan.fourmilab.ch”. I'm naming in-house servers after asteroids now—tens of thousands have been named, so I'll run out of Class C IP space long before I run out of names! At the time the new firewall was installed, I partitioned the Domain Name Service for the site into a public view which resolves only hosts in the externally accessible host.fourmilab.ch IP address block (193.8.230.0/24), and an internal view which also includes hosts on the LAN (host.lan.fourmilab.ch, 10.1.0.0/16) and servers on the DMZ (host.dmz.fourmilab.ch, 10.2.0.0/16). The Fourmilab Network Architecture chart (to which I haven't yet added ceres) describes how these networks are interconnected.

Although the machine is shown here with a Fourmilab “Bitmobile” crash cart attached, it is usually administered as a “headless” system via ssh and the Dell Remote Access Controller, which provides remote console access across the Web.

Posted at 23:33 Permalink