« Valley of the Dells: RAID Firmware Upgrade | Main | Reading List: Go Directly to Jail »
Saturday, April 9, 2005
Server Farm Status
I've been writing so much about the server farm recently it's high time I showed a picture of its present state of development. (Click the image for an enlargement in a separate window.)
The two boxes at the bottom are the Dell PowerEdge 1850 servers which host the site. Each has dual Intel Xeon 3.6 GHz hyper-threaded processors, which gives each server the equivalent of four CPUs. Each has 8 Gb of ECC RAM, dual 146 Gb 10,000 RPM SCSI drives on an embedded RAID controller, and two Gigabit Ethernet interfaces, which are "bonded" into a single logical interface, with each physical interface connected to one of the two 16 port Dell PowerConnect 2616 Gigabit Ethernet switches at the top of the rack. The interface to switch connections of the two servers are crossed with respect to one another. The two switches are connected together and normally forward packets to one another; each is connected to the DMZ port of one of the two redundant firewalls (which aren't in this rack, but in the communications rack upstairs).
Between the servers and switches are two identical Coyote Point Equalizer 350 load balancers run in primary/backup high availability mode. The top load balancer is connected to the top switch and the bottom load balancer to the bottom switch. Hence, they exchange heartbeats through the interconnected switches, so if one switch goes down, whichever load balancer is connected to the remaining switch will become primary, and since each server has an interface connected to both switches, it will continue to be able to communicate to both servers.
The rack is 24 units high, and all the components are designed to permit dense packing, but I've spaced them out both because it makes them easier to work on and remove if necessary, and also because additional breathing room can't hurt cooling. It isn't obvious from this picture, but the rack is deep--73.5 cm from front to back rails and just a tad less than one metre for the entire cabinet; the Dell servers are 1U high, but they just keep on coming in the depth dimension.
The load balancers are less than 50 cm deep, so I've exploited the unused space by mounting two 15 socket outlet strips on the back rails, one behind each load balancer. These are plugged into independent APC SmartUPS 1500 units which sit on the floor behind the rack, fed from separate dedicated 10 A 230 V circuits with slow-blow thermal fuses. (Never plug a UPS or any other equipment with a big iron-core transformer into a circuit with a fast-trip breaker. The inrush current after even a momentary power blip may pop the breaker and bring your holiday to an unexpected end. This has happened to me.) The servers have dual redundant power supplies and each is plugged into both outlet strips, while the other pairs of components have one of each plugged into each strip. The UPS units are not mounted in the rack due to my earlier surreal adventure with a rack-mounted UPS. The UPS monitoring and control port of each UPS is connected to the serial port of one of the two servers; there is presently no broadcast shutdown, but since each UPS can handle the load of both servers and each server can run on one of its two power supplies, this expedient gets the job done, albeit inelegantly. The servers run the Apcupsd monitoring and control software. The load balancers, which are really FreeBSD machines, also would like to be shut down cleanly before the power goes down, but at the moment that's something which remains on my to-do list.
Posted at April 9, 2005 15:50