Fourmilog: None Dare Call It Reason

« June 30, 2008 | Main | July 4, 2008 »

Wednesday, July 2, 2008

Digging Up the Street: Fourmilab Fast Fibre

Click image for “digging up the street” gallery.

In the late 1960s, I recall Prof. William C. Lynch explaining why it made business sense for AT&T to fund Bell Laboratories so generously. The basic and applied research performed there had, among other things, allowed the Bell System to squeeze more and more communications traffic onto their existing physical infrastructure. “Technology may be costly,” he said, “but nothing is as expensive as digging up a street.” But, notwithstanding technological miracles which today allow multi-megabit per second broadband Internet service to be delivered over copper telephone lines which, in 1969, typically provided 110 to 300 bits per second service, sometimes you just have to dig up the street.

For many years, Fourmilab's Internet connection has been via a symmetrical 2 Mbit/sec link over a copper pair. (Note that while many DSL links are faster, at least inbound, this service provides the critical outbound bandwidth required for a Web server. In excess of 95% of Fourmilab's traffic is outbound. Also, a leased line connection provides a much higher level of service and guaranteed bandwidth to the Internet backbone.) This was the fastest service available via a copper pair given the length of the run to the central office. While I've received few complaints about the performance of the site, it was clear that the link was approaching saturation at peak periods, and for several years I'd avoided hosting videos and other large files which would further increase the traffic. I could have doubled the bandwidth by installing a second leased line, but that would have almost doubled my monthly connect costs, which are already uncomfortably high.

In discussing options with Fourmilab's Internet Service Provider (ISP), Cablecom, they suggested that installing a fibre optic link might be the most attractive option, and offered to prepare a proposal for a fibre link with 20 Mbit/sec bandwidth—ten times the current speed—with the hardware and civil engineering costs amortised in monthly connectivity fees over a five year service contract.

When the bid came in, I was amazed to see that for an 18% increase in what I was currently paying, I could increase symmetrical bandwidth tenfold. The physical fibre link would permit upgrading to 100 Mbit/sec symmetrical on a single 9 µm monomode fibre (full-duplex is achieved by using different light frequencies for uplink and downlink), and with 12 fibres in the cable, bandwidth could ultimately be expanded to 1 Gbit/sec using 10 fibres with 2 spares, although that's way beyond both my requirements and budget. There was an existing fibre connection to the Maison de Commune in Lignières, which is almost exactly 1 kilometre from Fourmilab as the telephone cables run. Installing the fibre would, then, involve digging up the street in various places between here and there and pulling the fibre cable, then installing the fibre patch panel in the Fourmilab communications rack and bonding the fibre to an outbound fibre at the Maison de Commune. The Digging Up the Street photo gallery shows the various excavations involved in this process. The entire job took about five days, off and on, with another day for the internal cable pull and yet another for the fibre termination at each end.

When it came time to test the cable, no light came out! Subsequent investigation with an optical time-domain reflectometer showed the fault to be 1068 metres from the Fourmilab end, indicating it was somewhere between the Maison de Commune and the hub in La Neuveville, not in the newly installed cable to Fourmilab. The cable guy said that this was most often due to rats gnawing into the fibre cables, which are filled with a gel seemingly made of rat candy. Newer cables, including the one just installed, have a layer of fibreglass armour inside the jacket which fragments when chewed into, irritating the tender little lips and tongues of the vermin and deterring their depredations. The break was fixed the next day.

After a few more rounds of configuration tweaks, the big switch-over occurred yesterday afternoon, with the first Web access arriving over the fibre connection on 2008-07-01 at 12:22:55 UTC. Since then Fourmilab has been running at 20 Mbit/sec over the fibre, so congestion at peak periods should now be a thing of the past. Accessing the Web over the fibre link actually doesn't feel much different than using the 3500/512 Kbit/sec ADSL line I'd been using before to avoid contending with the traffic on the leased line. I suspect that most sites one accesses simply do not have end-to-end bandwidth much greater than 3.5 Mbit/sec for an individual connection or throttle client traffic to limit the impact of bandwidth hogs on other requesters.

Update: There appears to have been a configuration problem in the upstream connection to the backbone which was resolved around 10:00 UTC on 2008-07-03. After the fix I downloaded a 1.2 Gb video file from CERN at a mean transfer rate of 2.01 megabytes per second (as reported by Wget). This is an after-overhead sustained bit rate of about 16 Mbit/sec, so it appears that the connection is now running at 20 Mbit/sec end to end. (2008-07-03 11:27 UTC)

Update: A very nice, free, and extremely stylish bandwidth test is available at Speedtest.net. Using it to test with servers in Geneva and Basel confirms that Fourmilab's Internet connectivity is now running at 20 Mbit/sec bidirectional. Thanks to those who sent bandwidth measurements; there isn't a need for any more. (2008-07-03 17:29 UTC)

Posted at 22:10