Fourmilog: None Dare Call It Reason

« January 2005 | Main | March 2005 »

Monday, February 28, 2005

Reading List: The Adventure of English

Bragg, Melvyn. The Adventure of English. London: Sceptre, 2003. ISBN 0-340-82993-1.

How did a language spoken by 150,000 or so Germanic warriors who invaded the British Isles in the fifth century A.D. become the closest thing so far to a global language, dominating the worlds of science and commerce which so define the modern age? Melvyn Bragg, who earlier produced a television series (which I haven't seen) with the same name for the British ITV network follows the same outline in this history of English. The tremendous contingency in the evolution of a language is much to be seen here: had Shakespeare, Dr. Johnson, or William Tyndale (who first translated the Bible into English and paid with his life for having done so) died in infancy, how would we speak today, and in what culture would we live? The assembly of the enormous vocabulary of English by devouring words from dozens of other languages is well documented, as well as the differentiation of British English into distinct American, Caribbean, Australian, South African, Indian, and other variants which enrich the mother tongue with both vocabulary and grammar. Fair dinkum, innit man?

As English has grown by accretion, it has also cast out a multitude of words into the "Obs." bin of the OED, many in the "Inkhorn Controversy" in the 16th century. What a loss! The more words, the richer the language, and I hereby urge we reinstate "abstergify", last cited in the OED in 1612, defined as the verb "To cleanse". I propose this word to mean "to clean up, æsthetically, without any change in function". For example, "I spent all day abstergifying the configuration files for the Web server".

The mystery of why such an ill-structured language with an almost anti-phonetic spelling should have become so widespread is discussed here only on the margin, often in apologetic terms invoking the guilt of slavery and colonialism. (But speakers of other languages pioneered these institutions, so why didn't they triumph?) Bragg suggests, almost in passing, what I think is very significant. The very irregularity of English permits it to assimilate the vocabulary of every language it encounters. In Greek, Latin, Spanish, or French, there are rules about the form of verbs and the endings of nouns and agreement of adjectives which cannot accommodate words from fundamentally different languages. But in English, there are no rules whatsoever--bring your own vocabulary--there's room for everybody and every word. Come on in, it's great--the more the better!

A U.S edition is now available, but as of this date only in hardcover.

Posted at 01:41

Sunday, February 27, 2005

Iron Jack Racks 'em

Finally living up to my pirate name "Iron Jack Rackham", today I took all of the components of the emerging Fourmilab server farm, jacked 'em up, and racked 'em in a big iron box.

Largely as a matter of pride, but also to validate the design of the server farm, the entire transfer between floors, installation of the boxes in the rack, and recabling to use new network components was accomplished without any outage whatsoever in access to Fourmilab. As is evident from the picture (click the image for an enlargement), things are still a tad tacky--the two bitmobiles parked to the right of the rack allow direct KVM access to the two servers, and the one in front has the laptop which acts as the debug console for the load balancer.

The cable roller in front of the rack is my homebrew 50 metre RJ-45 patch extender; I'm using it to connect the backup firewall to the cluster hub, since the nearest free network jack was farther away than any regular cable I have on hand. In the rack, the two boxes at the top are Dell 16 port Gigabit Ethernet switches which interconnect everything on the server farm--at the moment, only the top switch is used. Below that is the Coyote Point Equalizer E350 load balancer (soon to be joined by an identical hot spare backup), and at the bottom are the two Dell PowerEdge 1850 servers which run the site.

The gap between the switches and the load balancers is due to my having had it with the crap-weasel 3Con 3CR16110-95 firewalls I was foolish enough to install three years ago. I'm unhappy with everything else on the market (either it doesn't support fail-over if something goes down, or it costs as much as the GDP of the Gambia and has a manual so long you can watch your fingernails grow as you read it). So, I am building my own firewall based on a Dell server configured with three Ethernet interfaces (LAN/WAN/DMZ) using Linux iptables. When I get this working, I'll post all of the configuration code so other folks can do likewise.

Posted at 01:27

Saturday, February 26, 2005

Reading List: Persepolis 2: The Story of a Return

Satrapi, Marjane. Persepolis 2: The Story of a Return. New York: Pantheon Books, [2002, 2003] 2004. ISBN 0-375-42288-9.

Having escaped from Iran in the middle of Iran/Iraq war to secular, decadent Austria, Marjane Satrapi picks up her comic book autobiography with the culture shock of encountering the amoral West. It ends badly. She returns to Tehran in search of her culture, and finds she doesn't fit there either, eventually abandoning a failed marriage to escape to the West, where she has since prospered as an author and illustrator. This intensely personal narrative brings home both why the West is hated in much of the world, and why, at the same time, so many people dream of escaping the tyranny of dull conformity for the light of liberty and reason in the West. Like Persepolis: The Story of a Childhood, this is a re-lettered English translation of the original French edition published in two volumes: (3, 4).

Posted at 02:03

Thursday, February 24, 2005

Reading List: Linux iptables Pocket Reference

Purdy, Gregor N. Linux iptables Pocket Reference. Sebastopol, CA: O'Reilly, 2004. ISBN 0-596-00569-5.

Sure, you could just read the manual pages, but when your site is under attack and you're the "first responder", this little book is just what you want in your sweaty fingers. It's also a handy reference to the fields in IP, TCP, UDP, and ICMP packets, which can be useful in interpreting packet dumps. Although intended as a reference, it's well worth taking the time (less than an hour) to read cover to cover. There are a number of very nice facilities in iptables/Netfilter which permit responding to common attacks. For example, the iplimit match allows blocking traffic from the bozone layer (yes, you--I know who you are and I know where you live) which ties up all of your HTTP server processes by connecting to them and then letting them time out or, slightly more sophisticated, feeding characters of a request every 20 seconds or so to keep it alive. The solution is:

    /sbin/iptables -A INPUT -p tcp --syn \
        --dport 80 -m iplimit --iplimit-above 20 \
    	--iplimit-mask 32 -j REJECT

Anybody who tries to open more than 20 connections will get whacked on each additional SYN packet. You can see whether this rule is affecting too many legitimate connections with the status query:

    /sbin/iptables -L -v

Geekly reading, to be sure, but just the thing if you're responsible for defending an Internet server or site from malefactors in the Internet Slum.

Posted at 00:04

Monday, February 21, 2005

It . . . is . . . alive!

On Friday, February 18th 2005 around 15:00 UTC, the prototype of the new www.fourmilab.ch server farm was put into pre-production test. The prototype is more of a server jardin potager than a farm, since it consists of only one main server with an identically configured laptop impersonating the second production server until its hardware arrives. The main server (the deep box with the black front and silver top at the bottom of the stack on the bitmobile at the right of the desk) is a Dell PowerEdge 1850 with dual Intel Xeon 3.6 GHz processors. These are "hyper-threading" CPUs, so there are logically four processors as seen by the operating system. Main memory is 8 Gb of ECC DDR2 RAM, which permits keeping the Earth and Moon Viewer image databases in memory (essential to avoid page thrashing) even if one memory bank fails power on self-test and is excluded from the configuration. Two 146 Gb 10,000 RPM SCSI drives are configured as a RAID1 mirror with hardware RAID support on the motherboard. The server can run "headless", but during the development phase I've attached a flat panel monitor, keyboard, and mouse to connectors on the back panel, which are replicated on the front for use with mobile KVM carts for data centre administration. The server has dual redundant power supplies, each with its own power cord, permitting them to be plugged into separate UPSes; one suffices to run the server, so you can pull one of the plugs at any time without crashing the machine. Both power supplies and hard drives can be "hot swapped" without powering down; when a new hard drive is detected, the RAID firmware will automatically reconstruct it from the mirror on the other drive. The servers are run under the Fedora Core 3 Linux distribution with current kernel (2.6.10-1.760_FC3smp) and utilities releases. The generic, binary distribution SMP (symmetric multi-processor) kernel is used.

Atop the server, elegantly resting on two pieces of salvaged packaging foam, is the Coyote Point Equalizer E350 which front-ends the server farm. When you connect to the IP address of www.fourmilab.ch, the load balancer receives the packet and forwards it to the available server most likely to provide the best response time based on an algorithm which includes the server's own estimate of its load from a Fourmilab custom program. The load balancer is configured to maintain session persistence where required, so if a user generates dynamic content in response to a request (for example, an image from Earth and Moon Viewer), that image will be retrieved from the cache on the same server which generated it. This is the most challenging part of implementing a server farm, and accounted for about 75% of the work to date in migrating the site from a single four processor SPARC/Solaris server to the server farm.

Yes, the present packaging is less than elegant. These Dell servers are 1U high, but you can't just bung them into any old rack because they're so deep. An 80 cm rack is the absolute minimum, but when you take into account the bend radius of power cords and the need to mount power distribution components on the back rails, you really need a one metre deep rack, so that's what I've ordered from Dell along with the second server. Once that comes to hand, I'll install the current components in the rack and proceed to a fully redundant configuration. This will include a second load balancer in hot spare mode, two switches to which the servers are cross-connected by their dual Ethernet interfaces (using "Bond: Ethernet Bond" mode), and two separate UPSes powering the servers, load balancers, and switches to avoid any single point failure mode.

Of course, there remain single point failures which can take the site down; after all, there was only one ascent engine on the lunar module: sometimes things just have to work. Failure of the leased line connection to the ISP, leased line modem, gateway router, or hub which connects the router to the boundary firewalls remain single points of failure, but then none of these have ever failed in the 10 years Fourmilab has been on the Internet. History has no predictive value whatsoever but, knock on biocomposite, maybe we'll go another decade without any of these weak links rendering the site out of sight.

And since this is a production test, if something at this site doesn't seem to be working, please report it with with our feedback form.

Posted at 01:17

Wednesday, February 16, 2005

Reading List: The Habit

Kuhns, Elizabeth. The Habit. New York: Doubleday, 2003. ISBN 0-385-50588-4.

For decades I've been interested in and worried about how well-intentioned "modernisations" might interrupt the chain of transmission of information and experience between generations and damage, potentially mortally, the very institutions modernisers were attempting to adapt to changing circumstances. Perhaps my concern with this somewhat gloomy topic stems from having endured both "new math" in high school and "new chemistry" in college, in both cases having to later re-learn the subject matter in the traditional way which enables one to, you know, actually solve problems.

Now that the radicals left over from the boomer generation are teachers and professors, we're into the second or third generation of a feedback cycle in which students either never learn the history of their own cultures or are taught contempt and hatred for it. The dearth of young people in the United States and U.K. who know how to think and have the factual framework from which to reason (or are aware what they don't know and how to find it out) is such that I worry about a runaway collapse of Western civilisation there. The very fact that it's impolitic to even raise such an issue in most of academia today only highlights how dire the situation is. (In continental Europe the cultural and educational situation is nowhere near as bad, but given that the population is aging and dying out it hardly matters. I read a prediction a couple of weeks ago that, absent immigration or change in fertility, the population of Switzerland, now more than seven million, could fall to about one million before the end of this century, and much the same situation obtains elsewhere in Europe. There is no precedent in human history for this kind of population collapse unprovoked by disaster, disease, or war.)

When pondering "macro, macro" issues like this, it's often useful to identify a micro-model to serve as a canary in the mineshaft for large-scale problems ahead. In 1965, the Second Vatican Council promulgated a top to bottom modernisation of the Roman Catholic Church. In that same year, there were around 180,000 Catholic nuns in the U.S.--an all time historical high--whose lifestyle, strongly steeped in tradition, began to immediately change in many ways far beyond the clothes they wore. Increasingly, orders opted for increasing invisibility--blending into the secular community. The result: an almost immediate collapse in their numbers, which has continued to the present day (graph). Today, there are only about 70,000 left, and with a mean age of 69, their numbers are sure to erode further in the future. Now, it's impossible to separate the consequences of modernisation of tradition from those of social changes in society at large, but it gives one pause to see an institution which, as this book vividly describes, has tenaciously survived two millennia of rising and falling empires, war, plague, persecution, inquisition, famine, migration, reformation and counter-reformation, disappearing like a puff of smoke within the space of one human lifetime. It makes you wonder about how resilient other, far more recent, components of our culture may be in the face of changes which discard the experience and wisdom of the past.

A paperback edition is scheduled for publication in April 2005.

Posted at 23:41

Monday, February 14, 2005

Reading List: Virtual LM

Sullivan, Scott P. Virtual LM. Burlington, Canada: Apogee Books, 2004. ISBN 1-894959-14-0.

I closed my comments about the author's earlier Virtual Apollo expressing my hope he would extend the project to the Lunar Module (LM). Well, here it is! These books are based on intricate computer solid models created by Sullivan from extensive research, then rendered to show how subsystems fit into the tightly-packed and weight-constrained spacecraft. The differences between the initial "H mission" modules (Apollo 9-14) and the extended stay "J mission" landers of Apollo 15-17 are shown in comparison renderings. In addition, the Lunar Roving Vehicle (moon buggy) used on the J missions is dissected in the same manner as the LM, along with the life support backpack worn by astronauts on the lunar surface. Nothing about the Lunar Module was simple, and no gory detail is overlooked in this book--there are eight pages (40-47) devoted to the door of the scientific equipment bay and the Rube Goldberg-like mechanism used to open it.

Sadly, like Virtual Apollo, this modeling and rendering labour of love is marred by numerous typographical errors in text and captions. From the point where I started counting, I noted 25, which is an unenviable accomplishment in a 250 page book which is mostly pictures. A companion CD-ROM includes the Apollo Operations Handbook, Lunar Module flight documents from Apollo 14-16, and photographs of the LM simulator and test article.

Posted at 18:15

Sunday, February 13, 2005

Reading List: The Rough Riders

Roosevelt, Theodore. The Rough Riders. Philadelphia: Pavilion Press, [1899] 2004. ISBN 1-4145-0492-6.

This is probably, by present-day standards, the most politically incorrect book ever written by a United States President. The fact that it was published and became a best-seller before his election as Vice President in 1900 and President in 1904 indicates how different the world was in the age in which Theodore Roosevelt lived and helped define. T.R. was no chicken-hawk. After advocating war with Spain as assistant secretary of the Navy in the McKinley administration, as war approached, he left his desk job in Washington to raise a volunteer regiment from the rough and ready horse- and riflemen of his beloved Wild West, along with number of his fellow Ivy Leaguers hungry for a piece of the action. This book chronicles his adventures in raising, equipping, and training the regiment, and its combat exploits in Cuba in 1898. The prose is pure T.R. passionate purple; it was rumoured that when the book was originally typeset the publisher had to send out for more copies of the the upper-case letter "I". Almost every page contains some remark or other which would end the career of what passes for politicians in today's pale, emasculated world. What an age. What a man! The bloodthirsty warrior who wrote this book would go on to win the Nobel Peace Prize in 1906 for brokering an end to the war between Russia and Japan.

This paperback edition from Pavilion Press is a sorry thing physically. The text reads like something that's been OCR scanned and never spelling checked or proofread--on p. 171, for example, "antagonists" is printed as "antagon1sts", and this is one of many such errors. There's no excuse for this at all, since there's an electronic text edition of The Rough Riders freely available from Project Gutenberg which is free of these errors, and an on-line edition which lacks these flaws. The cover photo of T.R. on his horse is a blow-up of a low-resolution JPEG image with obvious pixels and compression artefacts.

Roosevelt's report to his commanding general (pp. 163-170) detailing the logistical and administrative screwups in the campaign is an excellent illustration of the maxim that the one area in which government far surpasses the capabilities of free enterprise is in the making of messes.

Posted at 22:40

Wednesday, February 9, 2005

Reading List: First Lensman

Smith, Edward E. First Lensman. Baltimore: Old Earth Books, [1950] 1997. ISBN 1-882968-10-7.

There's no better way to escape for a brief respite from the world of session persistence, subnet masks, stateful fallover, gratuitous ARP packets, and the like than some coruscating, actinic space opera, and nobody does it better than the guy who invented it, Doc Smith. About every decade I re-read the Lensman series, of which this is the second of six volumes (seven if you count Masters of the Vortex) and never cease to be amazed at Smith's talent for thinking big--really big. I began this fourth expedition through the Lensman saga with the first installment, Triplanetary, in June 2004. Old Earth Books are to be commended for this reprint, which is a facsimile of the original 1950 Fantasy Press edition including all the illustrations.

Posted at 18:00

Saturday, February 5, 2005

Reading List: Load Balancing Servers, Firewalls, and Caches

Kopparapu, Chandra. Load Balancing Servers, Firewalls, and Caches. New York: John Wiley & Sons, 2002. ISBN 0-471-41550-2.

Don't even think about deploying a server farm or geographically dispersed mirror sites without reading this authoritative book. The Internet has become such a mountain of interconnected kludges that something as conceptually simple as spreading Web and other Internet traffic across a collection of independent servers or sites in the interest of increased performance and fault tolerance becomes a matter of enormous subtlety and hideous complexity. Most of the problems come from the need for "session persistence": when a new user arrives at your site, you can direct them to any available server based on whatever load balancing algorithm you choose, but if the user's interaction with the server involves dynamically generated content produced by the server (for example, images generated by Earth and Moon Viewer, or items the user places in their shopping cart at a commerce site), subsequent requests by the user must be directed to the same server, as only it contains the state of the user's session.

(Some load balancer vendors will try to persuade you that session persistence is a design flaw in your Web applications which you should eliminate by making them stateless or by using a common storage pool shared by all the servers. Don't believe this. I defy you to figure out how an application as simple as Earth and Moon Viewer, which does nothing more complicated than returning a custom Web page which contains a dynamically generated embedded image, can be made stateless. And shared backing store (for example, Network Attached Storage servers) has its own scalability and fault tolerance challenges.)

Almost any simple scheme you can come up with to get around the session persistence problem will be torpedoed by one or more of the kludges and hacks through which a user's packet traverses between client and server: NAT, firewalls, proxy servers, content caches, etc. Consider what at first appears to be a foolproof scheme (albeit sub-optimal for load distribution): simply hash the client's IP address into a set of bins, one for each server, and direct the packets accordingly. Certainly, that would work, right? Wrong: huge ISPs such as AOL and EarthLink have farms of proxy servers between their customers and the sites they contact, and these proxy servers are themselves load balanced in a non-persistent manner. So even two TCP connections from the same browser retrieving, say, the text and an image from a single Web page, may arrive at your site apparently originating from different IP addresses!

This and dozens of other gotchas and ways to work around them are described in detail in this valuable book, which is entirely vendor-neutral, except for occasionally mentioning products to illustrate different kinds of architectures. It's a lot better to slap your forehead every few pages as you discover something else you didn't think of which will sabotage your best-laid plans than pull your hair out later after putting a clever and costly scheme into production and discovering that it doesn't work. When I started reading this book, I had no idea how I was going to solve the load balancing problem for the Fourmilab site, and now I know precisely how I'm going to proceed. This isn't a book you read for entertainment, but if you need to know this stuff, it's a great place to learn it.

Posted at 21:49

Friday, February 4, 2005

Safetyland: Power Cords Are Hazardous!

I recently purchased an Ethernet switch from a vendor in Switzerland. It's the product of a U.S. company, assembled in the Far East. Inside the box was this enlightening sticker. Now we all know that in the U.S. risk is everywhere--unless alerted by prominent warning labels, preferably DayGlo^® orange with 36 point bold black type--people will fold their baby strollers with the baby inside, scrub their backs with toilet brushes, or drive off in their cars having neglected to remove the sun-screen from the windshield. No doubt these folks also frequently plug in electrical appliances while firmly grasping the plug by both prongs, and I'm shocked, shocked that there's no warning label to avert them of the consequences--imagine the lawsuits! Anyway, even if you don't manage to electrocute yourself inserting or pulling the plug, in California anyway, you're still at risk of birth defects simply from handling the cord. (Presumably we're talking about your offspring being at risk although, this being California, perhaps one must take retrocausality into account.) But what I want to know is what safety precautions are in place for the workers in Asia who smear the toxic lead all over the cords! Note the warning on the warning sticker that "consumers" may have to cope with installing this sticker on a cord with a different shape than that illustrated. Also, they didn't note that lead ingestion can cause you to go crazy but, hey, in California, how could you tell?

Posted at 21:15

Thursday, February 3, 2005

Reading List: Salt: A World History

Kurlansky, Mark. Salt: A World History. New York: Penguin Books, 2002. ISBN 0-14-200161-9.

You may think this a dry topic, but the history of salt is a microcosm of the history of human civilisation. Carnivorous animals and human tribes of hunters get all the salt they need from the meat they eat. But as soon as humans adopted a sedentary agricultural lifestyle and domesticated animals, they and their livestock had an urgent need for salt--a cow requires ten times as much salt as a human. The collection and production of salt was a prerequisite for human settlements and, as an essential commodity required by every individual, the first to be taxed and regulated by that chronic affliction of civilisation, government. Salt taxes supported the Chinese empire for almost two millennia, the Viennese and Genoan trading empires and the Hanseatic League, precipitated the French Revolution and India's struggle for independence from the British empire. Salt was a strategic commodity in the Roman Empire: most Roman cities were built near saltworks, and the words "salary" and "soldier" are both derived from the Latin word for salt. This and much more is covered in this fascinating look at human civilisation through the crystals of a tasty and essential inorganic compound composed of two poisonous elements. Recipes for salty specialities of cultures around the world and across the centuries are included, along with recommendations for surviving that "surprisingly pleasant" Swedish speciality surströmming (p. 139): "The only remaining problem is how to get the smell out of the house. . .".

Posted at 23:57