Fourmilog: None Dare Call It Reason

Reading List: Losing the Nobel Prize

Sunday, August 5, 2018 10:51

Keating, Brian. Losing the Nobel Prize. New York: W. W. Norton, 2018. ISBN 978-1-324-00091-4.
Ever since the time of Galileo, the history of astronomy has been punctuated by a series of “great debates”—disputes between competing theories of the organisation of the universe which observation and experiment using available technology are not yet able to resolve one way or another. In Galileo's time, the great debate was between the Ptolemaic model, which placed the Earth at the centre of the solar system (and universe) and the competing Copernican model which had the planets all revolving around the Sun. Both models worked about as well in predicting astronomical phenomena such as eclipses and the motion of planets, and no observation made so far had been able to distinguish them.

Then, in 1610, Galileo turned his primitive telescope to the sky and observed the bright planets Venus and Jupiter. He found Venus to exhibit phases, just like the Moon, which changed over time. This would not happen in the Ptolemaic system, but is precisely what would be expected in the Copernican model—where Venus circled the Sun in an orbit inside that of Earth. Turning to Jupiter, he found it to be surrounded by four bright satellites (now called the Galilean moons) which orbited the giant planet. This further falsified Ptolemy's model, in which the Earth was the sole source of attraction around which all celestial bodies revolved. Since anybody could build their own telescope and confirm these observations, this effectively resolved the first great debate in favour of the Copernican heliocentric model, although some hold-outs in positions of authority resisted its dethroning of the Earth as the centre of the universe.

This dethroning came to be called the “Copernican principle”, that Earth occupies no special place in the universe: it is one of a number of planets orbiting an ordinary star in a universe filled with a multitude of other stars. Indeed, when Galileo observed the star cluster we call the Pleiades, he saw myriad stars too dim to be visible to the unaided eye. Further, the bright stars were surrounded by a diffuse bluish glow. Applying the Copernican principle again, he argued that the glow was due to innumerably more stars too remote and dim for his telescope to resolve, and then generalised that the glow of the Milky Way was also composed of uncountably many stars. Not only had the Earth been demoted from the centre of the solar system, so had the Sun been dethroned to being just one of a host of stars possibly stretching to infinity.

But Galileo's inference from observing the Pleiades was wrong. The glow that surrounds the bright stars is due to interstellar dust and gas which reflect light from the stars toward Earth. No matter how large or powerful the telescope you point toward such a reflection nebula, all you'll ever see is a smooth glow. Driven by the desire to confirm his Copernican convictions, Galileo had been fooled by dust. He would not be the last.

William Herschel was an eminent musician and composer, but his passion was astronomy. He pioneered the large reflecting telescope, building more than sixty telescopes. In 1789, funded by a grant from King George III, Herschel completed a reflector with a mirror 1.26 metres in diameter, which remained the largest aperture telescope in existence for the next fifty years. In Herschel's day, the great debate was about the Sun's position among the surrounding stars. At the time, there was no way to determine the distance or absolute brightness of stars, but Herschel decided that he could compile a map of the galaxy (then considered to be the entire universe) by surveying the number of stars in different directions. Only if the Sun was at the centre of the galaxy would the counts be equal in all directions.

Aided by his sister Caroline, a talented astronomer herself, he eventually compiled a map which indicated the galaxy was in the shape of a disc, with the Sun at the centre. This seemed to refute the Copernican view that there was nothing special about the Sun's position. Such was Herschel's reputation that this finding, however puzzling, remained unchallenged until 1847 when Wilhelm Struve discovered that Herschel's results had been rendered invalid by his failing to take into account the absorption and scattering of starlight by interstellar dust. Just as you can only see the same distance in all directions while within a patch of fog, regardless of the shape of the patch, Herschel's survey could only see so far before extinction of light by dust cut off his view of stars. Later it was discovered that the Sun is far from the centre of the galaxy. Herschel had been fooled by dust.

In the 1920s, another great debate consumed astronomy. Was the Milky Way the entire universe, or were the “spiral nebulæ” other “island universes”, galaxies in their own right, peers of the Milky Way? With no way to measure distance or telescopes able to resolve them into stars, many astronomers believed spiral neublæ were nearby objects, perhaps other solar systems in the process of formation. The discovery of a Cepheid variable star in the nearby Andromeda “nebula” by Edwin Hubble in 1923 allowed settling this debate. Andromeda was much farther away than the most distant stars found in the Milky Way. It must, then be a separate galaxy. Once again, demotion: the Milky Way was not the entire universe, but just one galaxy among a multitude.

But how far away were the galaxies? Hubble continued his search and measurements and found that the more distant the galaxy, the more rapidly it was receding from us. This meant the universe was expanding. Hubble was then able to calculate the age of the universe—the time when all of the galaxies must have been squeezed together into a single point. From his observations, he computed this age at two billion years. This was a major embarrassment: astrophysicists and geologists were confident in dating the Sun and Earth at around five billion years. It didn't make any sense for them to be more than twice as old as the universe of which they were a part. Some years later, it was discovered that Hubble's distance estimates were far understated because he failed to account for extinction of light from the stars he measured due to dust. The universe is now known to be seven times the age Hubble estimated. Hubble had been fooled by dust.

By the 1950s, the expanding universe was generally accepted and the great debate was whether it had come into being in some cataclysmic event in the past (the “Big Bang”) or was eternal, with new matter spontaneously appearing to form new galaxies and stars as the existing ones receded from one another (the “Steady State” theory). Once again, there were no observational data to falsify either theory. The Steady State theory was attractive to many astronomers because it was the more “Copernican”—the universe would appear overall the same at any time in an infinite past and future, so our position in time is not privileged in any way, while in the Big Bang the distant past and future are very different than the conditions we observe today. (The rate of matter creation required by the Steady State theory was so low that no plausible laboratory experiment could detect it.)

The discovery of the cosmic background radiation in 1965 definitively settled the debate in favour of the Big Bang. It was precisely what was expected if the early universe were much denser and hotter than conditions today, as predicted by the Big Bang. The Steady State theory made no such prediction and was, despite rear-guard actions by some of its defenders (invoking dust to explain the detected radiation!), was considered falsified by most researchers.

But the Big Bang was not without its own problems. In particular, in order to end up with anything like the universe we observe today, the initial conditions at the time of the Big Bang seemed to have been fantastically fine-tuned (for example, an infinitesimal change in the balance between the density and rate of expansion in the early universe would have caused the universe to quickly collapse into a black hole or disperse into the void without forming stars and galaxies). There was no physical reason to explain these fine-tuned values; you had to assume that's just the way things happened to be, or that a Creator had set the dial with a precision of dozens of decimal places.

In 1979, the theory of inflation was proposed. Inflation held that in an instant after the Big Bang the size of the universe blew up exponentially so that all the observable universe today was, before inflation, the size of an elementary particle today. Thus, it's no surprise that the universe we now observe appears so uniform. Inflation so neatly resolved the tensions between the Big Bang theory and observation that it (and refinements over the years) became widely accepted. But could inflation be observed? That is the ultimate test of a scientific theory.

There have been numerous cases in science where many years elapsed between a theory being proposed and definitive experimental evidence for it being found. After Galileo's observations, the Copernican theory that the Earth orbits the Sun became widely accepted, but there was no direct evidence for the Earth's motion with respect to the distant stars until the discovery of the aberration of light in 1727. Einstein's theory of general relativity predicted gravitational radiation in 1915, but the phenomenon was not directly detected by experiment until a century later. Would inflation have to wait as long or longer?

Things didn't look promising. Almost everything we know about the universe comes from observations of electromagnetic radiation: light, radio waves, X-rays, etc., with a little bit more from particles (cosmic rays and neutrinos). But the cosmic background radiation forms an impenetrable curtain behind which we cannot observe anything via the electromagnetic spectrum, and it dates from around 380,000 years after the Big Bang. The era of inflation was believed to have ended 10−32 seconds after the Bang; considerably earlier. The only “messenger” which could possibly have reached us from that era is gravitational radiation. We've just recently become able to detect gravitational radiation from the most violent events in the universe, but no conceivable experiment would be able to detect this signal from the baby universe.

So is it hopeless? Well, not necessarily…. The cosmic background radiation is a snapshot of the universe as it existed 380,000 years after the Big Bang, and only a few years after it was first detected, it was realised that gravitational waves from the very early universe might have left subtle imprints upon the radiation we observe today. In particular, gravitational radiation creates a form of polarisation called B-modes which most other sources cannot create.

If it were possible to detect B-mode polarisation in the cosmic background radiation, it would be a direct detection of inflation. While the experiment would be demanding and eventually result in literally going to the end of the Earth, it would be strong evidence for the process which shaped the universe we inhabit and, in all likelihood, a ticket to Stockholm for those who made the discovery.

This was the quest on which the author embarked in the year 2000, resulting in the deployment of an instrument called BICEP1 (Background Imaging of Cosmic Extragalactic Polarization) in the Dark Sector Laboratory at the South Pole. Here is my picture of that laboratory in January 2013. The BICEP telescope is located in the foreground inside a conical shield which protects it against thermal radiation from the surrounding ice. In the background is the South Pole Telescope, a millimetre wave antenna which was not involved in this research.

BICEP2 and South Pole Telescope, 2013-01-09

BICEP1 was a prototype, intended to test the technologies to be used in the experiment. These included cooling the entire telescope (which was a modest aperture [26 cm] refractor, not unlike Galileo's, but operating at millimetre wavelengths instead of visible light) to the temperature of interstellar space, with its detector cooled to just ¼ degree above absolute zero. In 2010 its successor, BICEP2, began observation at the South Pole, and continued its run into 2012. When I took the photo above, BICEP2 had recently concluded its observations.

On March 17th, 2014, the BICEP2 collaboration announced, at a press conference, the detection of B-mode polarisation in the region of the southern sky they had monitored. Note the swirling pattern of polarisation which is the signature of B-modes, as opposed to the starburst pattern of other kinds of polarisation.

B-mode polarisation in BICEP2 observations, 2014-03-17

But, not so fast, other researchers cautioned. The risk in doing “science by press release” is that the research is not subjected to peer review—criticism by other researchers in the field—before publication and further criticism in subsequent publications. The BICEP2 results went immediately to the front pages of major newspapers. Here was direct evidence of the birth cry of the universe and confirmation of a theory which some argued implied the existence of a multiverse—the latest Copernican demotion—the idea that our universe was just one of an ensemble, possibly infinite, of parallel universes in which every possibility was instantiated somewhere. Amid the frenzy, a few specialists in the field, including researchers on competing projects, raised the question, “What about the dust?” Dust again! As it happens, while gravitational radiation can induce B-mode polarisation, it isn't the only thing which can do so. Our galaxy is filled with dust and magnetic fields which can cause those dust particles to align with them. Aligned dust particles cause polarised reflections which can mimic the B-mode signature of the gravitational radiation sought by BICEP2.

The BICEP2 team was well aware of this potential contamination problem. Unfortunately, their telescope was sensitive only to one wavelength, chosen to be the most sensitive to B-modes due to primordial gravitational radiation. It could not, however, distinguish a signal from that cause from one due to foreground dust. At the same time, however, the European Space Agency Planck spacecraft was collecting precision data on the cosmic background radiation in a variety of wavelengths, including one sensitive primarily to dust. Those data would have allowed the BICEP2 investigators to quantify the degree their signal was due to dust. But there was a problem: BICEP2 and Planck were direct competitors.

Planck had the data, but had not released them to other researchers. However, the BICEP2 team discovered that a member of the Planck collaboration had shown a slide at a conference of unpublished Planck observations of dust. A member of the BICEP2 team digitised an image of the slide, created a model from it, and concluded that dust contamination of the BICEP2 data would not be significant. This was a highly dubious, if not explicitly unethical move. It confirmed measurements from earlier experiments and provided confidence in the results.

In September 2014, a preprint from the Planck collaboration (eventually published in 2016) showed that B-modes from foreground dust could account for all of the signal detected by BICEP2. In January 2015, the European Space Agency published an analysis of the Planck and BICEP2 observations which showed the entire BICEP2 detection was consistent with dust in the Milky Way. The epochal detection of inflation had been deflated. The BICEP2 researchers had been deceived by dust.

The author, a founder of the original BICEP project, was so close to a Nobel prize he was already trying to read the minds of the Nobel committee to divine who among the many members of the collaboration they would reward with the gold medal. Then it all went away, seemingly overnight, turned to dust. Some said that the entire episode had injured the public's perception of science, but to me it seems an excellent example of science working precisely as intended. A result is placed before the public; others, with access to the same raw data are given an opportunity to critique them, setting forth their own raw data; and eventually researchers in the field decide whether the original results are correct. Yes, it would probably be better if all of this happened in musty library stacks of journals almost nobody reads before bursting out of the chest of mass media, but in an age where scientific research is funded by agencies spending money taken from hairdressers and cab drivers by coercive governments under implicit threat of violence, it is inevitable they will force researchers into the public arena to trumpet their “achievements”.

In parallel with the saga of BICEP2, the author discusses the Nobel Prizes and what he considers to be their dysfunction in today's scientific research environment. I was surprised to learn that many of the curious restrictions on awards of the Nobel Prize were not, as I had heard and many believe, conditions of Alfred Nobel's will. In fact, the conditions that the prize be shared no more than three ways, not be awarded posthumously, and not awarded to a group (with the exception of the Peace prize) appear nowhere in Nobel's will, but were imposed later by the Nobel Foundation. Further, Nobel's will explicitly states that the prizes shall be awarded to “those who, during the preceding year, shall have conferred the greatest benefit to mankind”. This constraint (emphasis mine) has been ignored since the inception of the prizes.

He decries the lack of “diversity” in Nobel laureates (by which he means, almost entirely, how few women have won prizes). While there have certainly been women who deserved prizes and didn't win (Lise Meitner, Jocelyn Bell Burnell, and Vera Rubin are prime examples), there are many more men who didn't make the three laureates cut-off (Freeman Dyson an obvious example for the 1965 Physics Nobel for quantum electrodynamics). The whole Nobel prize concept is capricious, and rewards only those who happen to be in the right place at the right time in the right field that the committee has decided deserves an award this year and are lucky enough not to die before the prize is awarded. To imagine it to be “fair” or representative of scientific merit is, in the estimation of this scribbler, in flying unicorn territory.

In all, this is a candid view of how science is done at the top of the field today, with all of the budget squabbles, maneuvering for recognition, rivalry among competing groups of researchers, balancing the desire to get things right with the compulsion to get there first, and the eye on that prize, given only to a few in a generation, which can change one's life forever.

Personally, I can't imagine being so fixated on winning a prize one has so little chance of gaining. It's like being obsessed with winning the lottery—and about as likely.

In parallel with all of this is an autobiographical account of the career of a scientist with its ups and downs, which is both a cautionary tale and an inspiration to those who choose to pursue that difficult and intensely meritocratic career path.

I recommend this book on all three tracks: a story of scientific discovery, mis-interpretation, and self-correction, the dysfunction of the Nobel Prizes and how they might be remedied, and the candid story of a working scientist in today's deeply corrupt coercively-funded research environment.

Back

Reading List: Une Fantaisie du Docteur Ox

Tuesday, July 24, 2018 21:12

Verne, Jules. Une Fantaisie du Docteur Ox. Seattle: CreateSpace, [1874] 2017. ISBN 978-1-5470-6408-3.
After reading and reviewing Jules Verne's Hector Servadac last year, I stumbled upon a phenomenal bargain: a Kindle edition of the complete works of Jules Verne—160 titles, with 5400 illustrations—for US$ 2.51 at this writing, published by Arvensa. This is not a cheap public domain knock-off, but a thoroughly professional publication with very few errors. For less than the price of a paperback book, you get just about everything Jules Verne ever wrote in Kindle format which, if you download the free Kindle French dictionary, allows you to quickly look up the obscure terms and jargon of which Verne is so fond without flipping through the Little Bob. That's how I read this work, although I have cited a print edition in the header for those who prefer such.

The strange story of Doctor Ox would be considered a novella in modern publishing terms, coming in at 19,240 words. It is divided into 17 chapters and is written in much the same style as the author's Voyages extraordinaires, with his customary huge vocabulary, fondness for lengthy enumerations, and witty parody of the national character of foreigners.

Here, the foreigners in question are the Flemish, speakers of dialects of the Dutch language who live in the northern part of Belgium. The Flemish are known for being phlegmatic, and nowhere is this more in evidence than the small city of Quiquendone. Its 2,393 residents and their ancestors have lived there since the city was founded in 1197, and very little has happened to disturb their placid lives; they like it that way. Its major industries are the manufacture of whipped cream and barley sugar. Its inhabitants are taciturn and, when they speak, do so slowly. For centuries, what little government they require has been provided by generations of the van Tricasse family, son succeeding father as burgomaster. There is little for the burgomaster to do, and one of the few items on his agenda, inherited from his father twenty years ago, is whether the city should dispense with the services of its sole policeman, who hasn't had anything to do for decades.

Burgomaster van Tricasse exemplifies the moderation in all things of the residents of his city. I cannot resist quoting this quintessentially Jules Verne description in full.

Le bourgmestre était un personnage de cinquante ans, ni gras ni maigre, ni petit ni grand, ni vieux ni jeune, ni coloré ni pâle, ni gai ni triste, ni content ni ennuyé, ni énergique ni mou, ni fier ni humble, ni bon ni méchant, ni généreux ni avare, ni brave ni poltron, ni trop ni trop peu, — ne quid nimis, — un homme modéré en tout ; mais à la lenteur invariable de ses mouvements, à sa mâchoire inférieure un peu pendante, à sa paupière supérieure immuablement relevée, à son front uni comme une plaque de cuivre jaune et sans une ride, à ses muscles peu salliants, un physionomiste eût sans peine reconnu que le bourgomestre van Tricasse était le flegme personnifié.

Imagine how startled this paragon of moderation and peace must have been when the city's policeman—he whose job has been at risk for decades—pounds on the door and, when admitted, reports that the city's doctor and lawyer, visiting the house of scientist Doctor Ox, had gotten into an argument. They had been talking politics! Such a thing had not happened in Quiquendone in over a century. Words were exchanged that might lead to a duel!

Who is this Doctor Ox? A recent arrival in Quiquendone, he is a celebrated scientist, considered a leader in the field of physiology. He stands out against the other inhabitants of the city. Of no well-defined nationality, he is a genuine eccentric, self-confident, ambitious, and known even to smile in public. He and his laboratory assistant Gédéon Ygène work on their experiments and never speak of them to others.

Shortly after arriving in Quiquendone, Dr Ox approached the burgomaster and city council with a proposal: to illuminate the city and its buildings, not with the new-fangled electric lights which other cities were adopting, but with a new invention of his own, oxy-hydric gas. Using powerful electric batteries he invented, water would be decomposed into hydrogen and oxygen gas, stored separately, then delivered in parallel pipes to individual taps where they would be combined and burned, producing a light much brighter and pure than electric lights, not to mention conventional gaslights burning natural or manufactured gas. In storage and distribution, hydrogen and oxygen would be strictly segregated, as any mixing prior to the point of use ran the risk of an explosion. Dr Ox offered to pay all of the expenses of building the gas production plant, storage facilities, and installation of the underground pipes and light fixtures in public buildings and private residences. After a demonstration of oxy-hydric lighting, city fathers gave the go-ahead for the installation, presuming Dr Ox was willing to assume all the costs in order to demonstrate his invention to other potential customers.

Over succeeding days and weeks, things before unimagined, indeed, unimaginable begin to occur. On a visit to Dr Ox, the burgomaster himself and his best friend city council president Niklausse find themselves in—dare it be said—a political argument. At the opera house, where musicians and singers usually so moderate the tempo that works are performed over multiple days, one act per night, a performance of Meyerbeer's Les Hugenots becomes frenetic and incites the audience to what can only be described as a riot. A ball at the house of the banker becomes a whirlwind of sound and motion. And yet, each time, after people go home, they return to normal and find it difficult to believe what they did the night before.

Over time, the phenomenon, at first only seen in large public gatherings, begins to spread into individual homes and private lives. You would think the placid Flemish had been transformed into the hotter tempered denizens of countries to the south. Twenty newspapers spring up, each advocating its own radical agenda. Even plants start growing to enormous size, and cats and dogs, previously as reserved as their masters, begin to bare fangs and claws. Finally, a mass movement rises to avenge the honour of Quiquendone for an injury committed in the year 1185 by a cow from the neighbouring town of Virgamen.

What was happening? Whence the madness? What would be the result when the citizens of Quiquendone, armed with everything they could lay their hands on, marched upon their neighbours?

This is a classic “puzzle story”, seasoned with a mad scientist of whom the author allows us occasional candid glimpses as the story unfolds. You'll probably solve the puzzle yourself long before the big reveal at the end. Jules Verne, always anticipating the future, foresaw this: the penultimate chapter is titled (my translation), “Where the intelligent reader sees that he guessed correctly, despite every precaution by the author”. The enjoyment here is not so much the puzzle but rather Verne's language and delicious description of characters and events, which are up to the standard of his better-known works.

This is “minor Verne”, written originally for a public reading and then published in a newspaper in Amiens, his adopted home. Many believed that in Quiquendone he was satirising Amiens and his placid neighbours.

Doctor Ox would reappear in the work of Jules Verne in his 1882 play Voyage à travers l'impossible (Journey Through the Impossible), a work which, after 97 performances in Paris, was believed lost until a single handwritten manuscript was found in 1978. Dr Ox reprises his role as mad scientist, joining other characters from Verne's novels on their own extraordinary voyages. After that work, Doctor Ox disappears from the world. But when I regard the frenzied serial madness loose today, from “bathroom equality”, tearing down Civil War monuments, masked “Antifa” blackshirts beating up people in the streets, the “refugee” racket, and Russians under every bed, I sometimes wonder if he's taken up residence in today's United States.

An English translation is available. Verne's reputation has often suffered due to poor English translations of his work; I have not read this edition and don't know how good it is. Warning: the description of this book at Amazon contains a huge spoiler for the central puzzle of the story.

Back

Reading List: Sanity

Sunday, July 22, 2018 22:10

Neovictorian [pseud.] and Neal Van Wahr. Sanity. Seattle: Amazon Digital Services, [2017] 2018. ISBN 978-1-980820-95-6.
Have you sometimes felt, since an early age, that you were an alien, somehow placed on Earth and observing the antics of humans as if they were a different species? Why do they believe such stupid things? Why do they do such dumb things? Any why do they keep doing them over and over again seemingly incapable of learning from the bad outcomes of all the previous attempts?

That is how Cal Adler felt since childhood and, like most people with such feelings, kept them quiet and bottled up while trying to get ahead in a game whose rules often seemed absurd. In his senior year in high school, he encounters a substitute guidance counsellor who tells him, without any preliminary conversation, precisely how he feels. He's assured he is not alone, and that over time he will meet others. He is given an enigmatic contact in case of emergency. He is advised, as any alien in a strange land, to blend in while observing and developing his own talents. And that's the last he sees of the counsellor.

Cal's subsequent life is punctuated by singular events: a terrorist incident in which he spontaneously rises to the occasion, encountering extraordinary people, and being initiated into skills he never imagined he'd possess. He begins to put together a picture of a shadowy…something…of which he may or may not be a part, whose goals are unclear, but whose people are extraordinary.

Meanwhile, a pop religion called ReHumanism, founded by a science fiction writer, is gaining adherents among prominent figures in business, entertainment, and technology. Its “scriptures” advocate escape from the tragic cycle of progress and collapse which has characterised the human experience by turning away from the artificial environment in which we have immersed ourselves and rediscovering our inherent human nature which may, to many in the modern world, seem alien. Is there a connection between ReHumanism (which seems like a flaky scam to Cal) and the mysterious people he is encountering?

All of these threads begin to come together when Cal, working as a private investigator in Reno, Nevada, is retained by the daughter of a recently-deceased billionaire industrialist to find her mother, who has disappeared during a tourist visit to Alaska. The mother is revealed have become a convert to and supporter of ReHumanism. Are they involved? And how did the daughter find Cal, who, after previous events, has achieved a level of low observability stealth aircraft designers can only dream of?

An adventure begins in which nothing is as it seems and all of Cal's formidable talents are tested to their limits.

This is an engaging and provocative mystery/thriller which will resonate with those who identify with the kind of heroic, independent, and inner-directed characters that populate the fiction of Robert A. Heinlein and other writers of the golden age of science fiction. It speaks directly to those sworn to chart their own course through life regardless of what others may think or say. I'm not sure the shadowy organisation we glimpse here actually exists, but I wish it did…and I wish they'd contacted me. There are many tips of the hat here to works and authors of fiction with similar themes, and I'm sure many more I missed.

This is an example of the efflorescence of independent science fiction which the obsolescence of the traditional gatekeeper publishers has engendered. With the advent of low-cost, high-margin self-publishing and customer reviews and ratings to evaluate quality, an entire new cohort of authors whose work would never before have seen the light of day is now enriching the genre and the lives of their enthusiastic readers. The work is not free of typographical and grammatical errors, but I've read books from major science fiction publishers with more. The Kindle edition is free to Kindle Unlimited subscribers.

Back

Twitterbot is a Bad, Bad Boy

Saturday, July 21, 2018 23:27

After I migrated the WordPress/BuddyPress site I administer, ratburger.org, to the Amazon Web Services (AWS) Linux 2 operating system platform on 2018-07-08, I observed intermittent errors in the system log reporting “php-fpm[21865]: [WARNING] [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 3 idle, and 27 total children” or some such. After correlating these with the HTTPD access_log, I found that they were due to the PHP-fpm mechanism (which is new in Linux 2) running out of worker processes or, even worse, launching so many of them it exhausts system memory and causes worker processes to crash. (And don't tell me to configure a swap file; that will only turn process crashes into system-wide thrashing oblivion.)

And why were all of these PHP processes running around? After all, this is a discussion site with fewer than 120 members and modest traffic. Looking at the log pointed the finger at Twitterbot, a Web crawler operated by the Californian socialist network called Twitter, which claims it's accessing sites to see if they provide “Twitter cards” for URLs posted on its system. Well, it's awfully frenetic in doing so. In the first incident I investigated, it hit my site from four different IP addresses (199.16.157.180-183) a total of 16 times within one second, all requesting the same page. You may call this a Web crawler. To me it looks like a denial of service attack. These requests will all spawn PHP-fpm worker processes and may blow away system memory, and for no reason. We do not support Twitter cards, and there is no conceivable reason for Twitter to make more than one request to determine we don't.

Enough is enough. I decided to tell Twitter to buzz (or flippy-flap) off. I added:

    User-agent: Twitterbot
    Disallow: /
to robots.txt and sat back to to see what would happen. Result? Essentially nothing: it continued to hit the site as before. All right, time to up the ante. I decided to consign Twitterbot to Stalag 403 with the following in .htaccess:
    # Block rogue user agents
    BrowserMatchNoCase 'Twitterbot' evilbots
    Order Allow,Deny
    Allow from ALL
    Deny from env=evilbots
so that any access from Twitterbot will receive a 403 and be informed that its access is forbidden and should not be retried. That ought to fix it, right?

Wrong.

In the last 24 hours there have been three request storms, all for /index.php, with 16 requests the first time and 18 on the second and third. All of these requests were sent within a period of one second, from four different IP addresses: 199.16.157.180-183. The second and third storms were 19 seconds apart, for a total of 36 hits within a period of less than 20 seconds.

For any site running PHP-fpm, this amounts to a denial of service attack: it will blow up the number of worker processes and possibly exhaust memory or start page thrashing and, in any case, delay legitimate user requests. Second, it isn't like the bot is crawling the site: it's making repeated requests for the same page over and over again, from four different IP addresses. Finally, it's violating HTTP protocol. A 403 status means the client has been forbidden access from the server, and the HTTP standard reads, “Authorization will not help and the request SHOULD NOT be repeated.” (capitals in the original). And yet in the third storm a single IP address hammered in 8 requests for the same page after having received a 403 on the first one. This is either exceptionally stupid or malicious, and I'm beginning to suspect the latter. I'm getting closer and closer to firewalling this IP range. This may break our anouncement of posts on Twitter, but at this point I'm not so sure that would be such a bad thing. The IP range is 199.16.157.180/30. Twitter's published outbound IP ranges are much larger: 199.16.156.0/22 and 199.59.148.0/22, but so far I've only seen Twitterbot coming from the four addresses in the first block.

I guess we shouldn't expect too much from a “social network” headquartered in a city now known for human feces and used addict needles on its sidewalks. (Hayek noted that any word in the English language is reduced in value by preceding it with “social”.) But once is happenstance, twice is coincidence, and three times is enemy action (Ian Fleming). Thirty-six times in twenty seconds? Welcome to my firewall.

(And note that these requests came from IPv4 address ranges which Twitter acknowledges are their own and were confirmed by WHOIS. So it's not somebody impersonating Twitterbot.)

By the way, if you're interested in intelligent, civil, and wide-ranging conversation, check out Ratburger,org. It's free; there are no advertisements, and no intrusive tracking. All members can post, comment, create and participate in interest groups, and join our weekly audio meet-up.

Back

Hebrew Bible Updated to Unicode, XHTML Strict

Tuesday, July 17, 2018 13:53

The Web edition of the Hebrew Bible has been available at Fourmilab since 1998. It originally required a browser extension to support downloadable fonts. When this became obsolete, a second edition was released in 2002 which used the ISO 8859-8 character set, which includes the ASCII Latin character set and Hebrew letters (but no vowel signs). Most Web browsers at the time supported this character set, although some required the installation of a "language pack" or font in order to display it.

At the time, I remarked that when Unicode became widely adopted, all of the complexity of special character sets for each language would evaporate, as we'd have a single character encoding which could handle all commonly-used languages (and many obscure ones, as well). Now, in 2018, we have made our landfall on that happy shore. The vast majority of widely-used operating systems and Web browsers support Unicode and provide at least one font with characters for the major languages.

I have just released a third edition of the Fourmilab Hebrew Bible, in which all documents use Unicode for all text, using the UTF-8 representation which now accounts for more than 90% of traffic on the Web. Any browser which supports Unicode and includes a font providing the Hebrew character set will be able to display these documents without any special configuration required—it should just work.

I have also updated all documents to the XHTML 1.0 Strict standard. I prefer this standard to HTML5 for documents which do not require features of the latter standard (such as embedded audio and video or the canvas element) since, being well-formed XML, XHTML documents can easily be parsed by computer programs which wish to process their content.

You can cite a chapter within a book of the Bible with a URL like:

http://www.fourmilab.ch/etexts/www/hebrew/Bible/?Exodus.html#c10
or an individual verse with:
http://www.fourmilab.ch/etexts/www/hebrew/Bible/?Exodus.html#v5:7

Previous editions of the Hebrew Bible did not require the “c” or “v” before the chapter or chapter:verse; this is a requirement of XHTML, in which the “id=” attribute must not start with a digit. For compatibility with existing citations, the “c” or “v” may be omitted, but in direct URLs citing the book document itself, they must be supplied.

This edition of the Hebrew Bible, like its predecessors, does not rely upon the so-called “Unicode Bidirectional Algorithm”. Instead, characters appear in the source HTML documents in the order they are presented in the page, with Hebrew text being explicitly reversed in order to read from right to left. In my experience, getting involved with automatic bidirectional text handling is the royal road to madness, and programmers who wish to keep what little hair that remains after half a century unscrewing the inscrutable trust their instinct about things to avoid. Hebrew text, which would otherwise automatically be rendered right-to-left by the browser, is explicitly surrounded by HTML tags:

<bdo dir="ltr">ת ישארב</bdo>

to override the default direction based upon the characters, in the example, the first word of Genesis. (You can also override the directionality of text by prefixing the Unicode LRO [&#8237;] or RLO [&#8238;] character and appending a PDF [&#8236;] to the string. I chose to use the XHTML override tag since it makes the intent clearer when processing the document with a program.)

To fully appreciate the insanity that Unicode bidirectional mode can induce in the minds of authors of multilingual documents, consider the following simplified HTML code for a sentence from the Hebrew Bible help file.

One writes:
100 as &#1511;,
101 as &#1488;&#1511;,
110 as &#1497;&#1511;, and
111 as &#1488;&#1497;&#1511;.

Want to guess how the browser renders this? Go ahead, guess. What you get is:

One writes: 100 as ק, 101 as אק, 110 as יק, and 111 as איק.

What? Why?? This way leads to the asylum. If you wrap the Hebrew with:

One writes:
100 as <bdo dir="ltr">&#1511;</bdo>,
101 as <bdo dir="ltr">&#1488;&#1511;</bdo>,
110 as <bdo dir="ltr">&#1497;&#1511;</bdo>, and
111 as <bdo dir="ltr">&#1488;&#1497;&#1511;</bdo>.

you get the desired:

One writes: 100 as ק, 101 as אק, 110 as יק, and 111 as איק.

In these examples, I have used HTML text entities (such as “&#1488;”) in the interest of comprehensibility. If you use actual Unicode characters and edit with a text editor such as Geany which infers text direction from the characters adjacent to the cursor, things get even more bewildering. The Hebrew Bible files contain Unicode characters, not text entities, but I only process them with custom Perl programs, never with a text editor.

In case somebody needs it, the ISO 8859-8 edition remains available.

Back