Fourmilog: None Dare Call It Reason

UNUM 3.1: Updated to Unicode 12.1.0, UTF-8 Support Added

Tuesday, January 7, 2020 15:53

Version 3.1 of UNUM is now available for downloading. Version 3.1 incorporates the Unicode 12.1.0 standard, released on May 7th, 2019. Since the Unicode 11.0.0 standard supported by UNUM 3.0, a total of 555 new characters have been added, for a total of 137,929 characters. Unicode 12.0.0 added support for 4 new scripts (for a total of 150) and 61 new emoji characters. Unicode 12.1.0 added the single character U+32FF, the Japanese character for the Reiwa era. (In addition to the standard Unicode characters, UNUM also supports an additional 65 ASCII control characters, which are not assigned graphic code points in the Unicode database.)

This is an incremental update to Unicode. There are no structural changes in how characters are defined in the databases, and other than the presence of the new characters, the operation of UNUM is unchanged. There have been no changes to the HTML named character reference standard since the release of UNUM version 2.2 in September 2017, so UNUM 3.1 is identical in this regard.

UNUM 3.1 adds support for the UTF-8 encoding of Unicode, and allows specification of characters as UTF-8 encoded byte streams expressed as numbers, for example:

    $ unum utf8=0xE298A2
       Octal  Decimal      Hex        HTML    Character   Unicode
      023042     9762   0x2622     ☢    "☢"         RADIOACTIVE SIGN
A new --utf8 option displays the UTF-8 encoding of characters as a hexadecimal byte stream:
  $ unum --utf8 h=sum
     Octal  Decimal      Hex        HTML       UTF-8      Character   Unicode
    021021     8721   0x2211 ∑,∑    0xE28891      "∑"         N-ARY SUMMATION
UNUM Documentation and Download Page


Reading List: The Simulation Hypothesis

Tuesday, January 7, 2020 01:00

Virk, Rizwan. The Simulation Hypothesis. Cambridge, MA: Bayview Books, 2019. ISBN 978-0-9830569-0-4.
Before electronic computers had actually been built, Alan Turing mathematically proved a fundamental and profound property of them which has been exploited in innumerable ways as they developed and became central to many of our technologies and social interactions. A computer of sufficient complexity, which is, in fact, not very complex at all, can simulate any other computer or, in fact, any deterministic physical process whatsoever, as long as it is understood sufficiently to model in computer code and the system being modelled does not exceed the capacity of the computer—or the patience of the person running the simulation. Indeed, some of the first applications of computers were in modelling physical processes such as the flight of ballistic projectiles and the hydrodynamics of explosions. Today, computer modelling and simulation have become integral to the design process for everything from high-performance aircraft to toys, and many commonplace objects in the modern world could not have been designed without the aid of computer modelling. It certainly changed my life.

Almost as soon as there were computers, programmers realised that their ability to simulate, well…anything made them formidable engines for playing games. Computer gaming was originally mostly a furtive and disreputable activity, perpetrated by gnome-like programmers on the graveyard shift while the computer was idle, having finished the “serious” work paid for by unimaginative customers (who actually rose before the crack of noon!). But as the microelectronics revolution slashed the size and price of computers to something individuals could afford for their own use (or, according to the computer Puritans of the previous generations, abuse), computer gaming came into its own. Some modern computer games have production and promotion budgets larger than Hollywood movies, and their characters and story lines have entered the popular culture. As computer power has grown exponentially, games have progressed from tic-tac-toe, through text-based adventures, simple icon character video games, to realistic three dimensional simulated worlds in which the players explore a huge world, interact with other human players and non-player characters (endowed with their own rudimentary artificial intelligence) within the game, and in some games and simulated worlds, have the ability to extend the simulation by building their own objects with which others can interact. If your last experience with computer games was the Colossal Cave Adventure or Pac-Man, try a modern game or virtual world—you may be amazed.

Computer simulations on affordable hardware are already beginning to approach the limits of human visual resolution, perception of smooth motion, and audio bandwidth and localisation, and some procedurally-generated game worlds are larger than a human can explore in a million lifetimes. Computer power is forecast to continue to grow exponentially for the foreseeable future and, in the Roaring Twenties, permit solving a number of problems through “brute force”—simply throwing computing power and massive data storage capacity at them without any deeper fundamental understanding of the problem. Progress in the last decade in areas such as speech recognition, autonomous vehicles, and games such as Go are precursors to what will be possible in the next.

This raises the question of how far it can go—can computer simulations actually approach the complexity of the real world, with characters within the simulation experiencing lives as rich and complex as our own and, perhaps, not even suspect they're living in a simulation? And then, we must inevitably speculate whether we are living in a simulation, created by beings at an outer level (perhaps themselves many levels deep in a tree of simulations which may not even have a top level). There are many reasons to suspect that we are living in a simulation; for many years I have said it's “more likely than not”, and others, ranging from Stephen Hawking to Elon Musk and Scott Adams, have shared my suspicion. The argument is very simple.

First of all, will we eventually build computers sufficiently powerful to provide an authentic simulated world to conscious beings living within it? There is no reason to doubt that we will: no law of physics prevents us from increasing the power of our computers by at least a factor of a trillion from those of today, and the lesson of technological progress has been that technologies usually converge upon their physical limits and new markets emerge as they do, using their capabilities and funding further development. Continued growth in computing power at the rate of the last fifty years should begin to make such simulations possible some time between 2030 and the end of this century.

So, when we have the computing power, will we use it to build these simulations? Of course we will! We have been building simulations to observe their behaviour and interact with them, for ludic and other purposes, ever since the first primitive computers were built. The market for games has only grown as they have become more complex and realistic. Imagine what if will be like when anybody can create a whole society—a whole universe—then let it run to see what happens, or enter it and experience it first-hand. History will become an experimental science. What would have happened if the Roman empire had discovered the electromagnetic telegraph? Let's see!—and while we're at it, run a thousand simulations with slightly different initial conditions and compare them.

Finally, if we can create these simulations which are so realistic the characters within them perceive them as their real world, why should we dare such non-Copernican arrogance as to assume we're at the top level and not ourselves within a simulation? I believe we shouldn't, and to me the argument that clinches it is what I call the “branching factor”. Just as we will eventually, indeed, I'd say, inevitably, create simulations as rich as our own world, so will the beings within them create their own. Certainly, once we can, we'll create many, many simulations: as many or more as there are running copies of present-day video games, and the beings in those simulations will as well. But if each simulation creates its own simulations in a number (the branching factor) even a tiny bit larger than one, there will be exponentially more observers in these layers on layers of simulations than at the top level. And, consequently, as non-privileged observers according to the Copernican Principle, it is not just more likely than not, but overwhelmingly probable that we are living in a simulation.

The author of this book, founder of Play Labs @ MIT, a start-up accelerator which works in conjunction with the MIT Game Lab, and producer of a number of video games, has come to the same conclusion, and presents the case for the simulation hypothesis from three perspectives: computer science, physics, and the unexplained (mysticism, esoteric traditions, and those enduring phenomena and little details which don't make any sense when viewed from the conventional perspective but may seem perfectly reasonable once we accept we're characters in somebody else's simulation).

Computer Science. The development of computer games is sketched from their origins to today's three-dimensional photorealistic multiplayer environments into the future, where virtual reality mediated by goggles, gloves, and crude haptic interfaces will give way to direct neural interfaces to the brain. This may seem icky and implausible, but so were pierced lips, eyebrows, and tongues when I was growing up, and now I see them everywhere, without the benefit of directly jacking in to a world larger, more flexible, and more interesting than the dingy one we inhabit. This is sketched in eleven steps, the last of which is the Simulation Point, where we achieve the ability to create simulations which “are virtually indistinguishable from a base physical reality.” He describes, “The Great Simulation is a video game that is so real because it is based upon incredibly sophisticated models and rendering techniques that are beamed directly into the mind of the players, and the actions of artificially generated consciousness are indistinguishable from real players.” He identifies nine technical hurdles which must be overcome in order to arrive at the Simulation Point. Some, such as simulating a sufficiently large world and number of players, are challenging but straightforward scaling up of things we're already doing, which will become possible as computer power increases. Others, such as rendering completely realistic objects and incorporating physical sensations, exist in crude form today but will require major improvements we don't yet know how to build, while technologies such as interacting directly with the human brain and mind and endowing non-player characters within the simulation with consciousness and human-level intelligence have yet to be invented.

Physics. There are a number of aspects of the physical universe, most revealed as we have observed at very small and very large scales, and at speeds and time intervals far removed from those with which we and our ancestors evolved, that appear counterintuitive if not bizarre to our expectations from everyday life. We can express them precisely in our equations of quantum mechanics, special and general relativity, electrodynamics, and the standard models of particle physics and cosmology, and make predictions which accurately describe our observations, but when we try to understand what is really going on or why it works that way, it often seems puzzling and sometimes downright weird.

But as the author points out, when you view these aspects of the physical universe through the eyes of a computer game designer or builder of computer models of complex physical systems, they look oddly familiar. Here is how I expressed it thirteen years ago in my 2006 review of Leonard Susskind's The Cosmic Landscape:

What would we expect to see if we inhabited a simulation? Well, there would probably be a discrete time step and granularity in position fixed by the time and position resolution of the simulation—check, and check: the Planck time and distance appear to behave this way in our universe. There would probably be an absolute speed limit to constrain the extent we could directly explore and impose a locality constraint on propagating updates throughout the simulation—check: speed of light. There would be a limit on the extent of the universe we could observe—check: the Hubble radius is an absolute horizon we cannot penetrate, and the last scattering surface of the cosmic background radiation limits electromagnetic observation to a still smaller radius. There would be a limit on the accuracy of physical measurements due to the finite precision of the computation in the simulation—check: Heisenberg uncertainty principle—and, as in games, randomness would be used as a fudge when precision limits were hit—check: quantum mechanics.

Indeed, these curious physical phenomena begin to look precisely like the kinds of optimisations game and simulation designers employ to cope with the limited computer power at their disposal. The author notes, “Quantum Indeterminacy, a fundamental principle of the material world, sounds remarkably similar to optimizations made in the world of computer graphics and video games, which are rendered on individual machines (computers or mobile phones) but which have conscious players controlling and observing the action.”

One of the key tricks in complex video games is “conditional rendering”: you don't generate the images or worry about the physics of objects which the player can't see from their current location. This is remarkably like quantum mechanics, where the act of observation reduces the state vector to a discrete measurement and collapses its complex extent in space and time into a known value. In video games, you only need to evaluate when somebody's looking. Quantum mechanics is largely encapsulated in the tweet by Aatish Bhatia, “Don't look: waves. Look: particles.” It seems our universe works the same way. Curious, isn't it?

Similarly, games and simulations exploit discreteness and locality to reduce the amount of computation they must perform. The world is approximated by a grid, and actions in one place can only affect neighbours and propagate at a limited speed. This is precisely what we see in field theories and relativity, where actions are local and no influence can propagate faster than the speed of light.

The unexplained. Many esoteric and mystic traditions, especially those of the East such as Hinduism and Buddhism, describe the world as something like a dream, in which we act and our actions affect our permanent identity in subsequent lives. Western traditions, including the Abrahamic religions, see life in this world as a temporary thing, where our acts will be judged by a God who is outside the world. These beliefs come naturally to humans, and while there is little or no evidence for them in conventional science, it is safe to say that far more people believe and have believed these things and have structured their lives accordingly than those who have adopted the strictly rationalistic viewpoint one might deduce from deterministic, reductionist science.

And yet, once again, in video games we see the emergence of a model which is entirely compatible with these ancient traditions. Characters live multiple lives, and their actions in the game cause changes in a state (“karma”) which is recorded outside the game and affects what they can do. They complete quests, which affect their karma and capabilities, and upon completing a quest, they may graduate (be reincarnated) into a new life (level), in which they retain their karma from previous lives. Just as players who exist outside the game can affect events and characters within it, various traditions describe actors outside the natural universe (hence “supernatural”) such as gods, angels, demons, and spirits of the departed, interacting with people within the universe and occasionally causing physical manifestations (miracles, apparitions, hauntings, UFOs, etc.). And perhaps the simulation hypothesis can even explain absence of evidence: the sky in a video game may contain a multitude of stars and galaxies, but that doesn't mean each is populated by its own video game universe filled with characters playing the same game. No, it's just scenery, there to be admired but with which you can't interact. Maybe that's why we've never detected signals from an alien civilisation: the stars are just procedurally generated scenery to make our telescopic views more interesting.

The author concludes with a summary of the evidence we may be living in a simulation and the objection of sceptics (such that a computer as large and complicated as the universe would be required to simulate a universe). He suggests experiments which might detect the granularity of the simulation and provide concrete evidence the universe is not the continuum most of science has assumed it to be. A final chapter presents speculations as to who might be running the simulation, what their motives might be for doing so, and the nature of beings within the simulation. I'm cautious of delusions of grandeur in making such guesses. I'll bet we're a science fair project, and I'll further bet that within a century we'll be creating a multitude of simulated universes for our own science fair projects.


Reading List: The City of Illusions

Wednesday, January 1, 2020 20:59

Wood, Fenton. The City of Illusions. Seattle: Amazon Digital Services, 2019. ASIN B082692JTX.
This is the fourth short novel/novella (148 pages) in the author's Yankee Republic series. I described the first, Pirates of the Electromagnetic Waves (May 2019), as “utterly charming”, and the second, Five Million Watts (June 2019), “enchanting”. The third, The Tower of the Bear (October 2019), takes Philo from the depths of the ocean to the Great Tree in the exotic West.

Here, the story continues as Philo reaches the Tree, meets its Guardian, “the largest, ugliest, and smelliest bear” he has ever seen, not to mention the most voluble and endowed with the wit of eternity, and explores the Tree, which holds gateways to other times and places, where Philo must confront a test which has defeated many heroes who have come this way before. Exploring the Tree, he learns of the distant past and future, of the Ancient Marauder and Viridios before the dawn of history, and of the War that changed the course of time.

Continuing his hero's quest, he ventures further westward along the Tyrant's Road into the desert of the Valley of Death. There he will learn the fate of the Tyrant and his enthralled followers and, if you haven't figured it out already, you will probably now understand where Philo's timeline diverged from our own. A hero must have a companion, and it is in the desert, after doing a good deed, that he meets his: a teddy bear, Made in Japan—but a very special teddy bear, as he will learn as the journey progresses.

Finally, he arrives at the Valley of the Angels, with pavement stretching to the horizon and cloaked in an acrid yellow mist that obscures visibility and irritates the eyes and throat. There he finds the legendary City of Illusions, where he is confronted by a series of diabolical abusement park attractions where his wit, courage, and Teddy's formidable powers will be tested to the utmost with death the price of failure. Victory can lead to the storied Bullet Train, the prize he needs to save radio station 2XG and possibly the world, and the next step in his quest.

As the fourth installment in what is projected to be one long story spanning five volumes, if you pick this up cold it will probably strike you as a bunch of disconnected adventures and puzzles each of which might as well be a stand-alone short-short story. As they unfold, only occasionally do you see a connection with the origins of the story or Philo's quest, although when they do appear (as in the linkage between the Library of Infinity and the Library of Ouroboros in The Tower of the Bear) they are a delight. It is only toward the end that you begin to see the threads converging toward what promises to be a stirring conclusion to a young adult classic enjoyable by all ages. I haven't read a work of science fiction so closely patterned on the hero's journey as described in Joseph Campbell's The Hero with a Thousand Faces since Rudy Rucker's 2004 novel Frek and the Elixir; this is not a criticism but a compliment—the eternal hero myth has always made for tales which not only entertain but endure.

This book is currently available only in a Kindle edition. The fifth and final volume of the Yankee Republic saga is scheduled to be published in the spring of 2020.


Books of the Year: 2019

Tuesday, December 31, 2019 12:56

Here are my picks for the best books of 2019, fiction and nonfiction. These aren't the best books published this year, but rather the best I've read in the last twelve months. The winner in both categories is barely distinguished from the pack, and the runners up are all worthy of reading. Runners up appear in alphabetical order by their author's surname. Each title is linked to my review of the book.


  • The Powers of the Earth and Causes of Separation by Travis J. I. Corcoran
    I am jointly choosing these two novels as fiction books of the year. They are the first two volumes of the Aristillus series and may be read as one long story spanning two books.
Runners up:


Winner: Runners up:


Reading List: The Sword and the Shield

Sunday, December 29, 2019 16:07

Andrew, Christopher and Vasili Mitrokhin. The Sword and the Shield. New York: Basic Books, 1999. ISBN 978-0-465-00312-9.
Vasili Mitrokhin joined the Soviet intelligence service as a foreign intelligence officer in 1948, at a time when the MGB (later to become the KGB) and the GRU were unified into a single service called the Committee of Information. By the time he was sent to his first posting abroad in 1952, the two services had split and Mitrokhin stayed with the MGB. Mitrokhin's career began in the paranoia of the final days of Stalin's regime, when foreign intelligence officers were sent on wild goose chases hunting down imagined Trotskyist and Zionist conspirators plotting against the regime. He later survived the turbulence after the death of Stalin and the execution of MGB head Lavrenti Beria, and the consolidation of power under his successors.

During the Khrushchev years, Mitrokhin became disenchanted with the regime, considering Khrushchev an uncultured barbarian whose banning of avant garde writers betrayed the tradition of Russian literature. He began to entertain dissident thoughts, not hoping for an overthrow of the Soviet regime but rather its reform by a new generation of leaders untainted by the legacy of Stalin. These thoughts were reinforced by the crushing of the reform-minded regime in Czechoslovakia in 1968 and his own observation of how his service, now called the KGB, manipulated the Soviet justice system to suppress dissent within the Soviet Union. He began to covertly listen to Western broadcasts and read samizdat publications by Soviet dissidents.

In 1972, the First Chief Directorate (FCD: foreign intelligence) moved from the cramped KGB headquarters in the Lubyanka in central Moscow to a new building near the ring road. Mitrokhin had sole responsibility for checking, inventorying, and transferring the entire archives, around 300,000 documents, of the FCD for transfer to the new building. These files documented the operations of the KGB and its predecessors dating back to 1918, and included the most secret records, those of Directorate S, which ran “illegals”: secret agents operating abroad under false identities. Probably no other individual ever read as many of the KGB's most secret archives as Mitrokhin. Appalled by much of the material he reviewed, he covertly began to make his own notes of the details. He started by committing key items to memory and then transcribing them every evening at home, but later made covert notes on scraps of paper which he smuggled out of KGB offices in his shoes. Each week-end he would take the notes to his dacha outside Moscow, type them up, and hide them in a series of locations which became increasingly elaborate as their volume grew.

Mitrokhin would continue to review, make notes, and add them to his hidden archive for the next twelve years until his retirement from the KGB in 1984. After Mikhail Gorbachev became party leader in 1985 and called for more openness (glasnost), Mitrokhin, shaken by what he had seen in the files regarding Soviet actions in Afghanistan, began to think of ways he might spirit his files out of the Soviet Union and publish them in the West.

After the collapse of the Soviet Union, Mitrokhin tested the new freedom of movement by visiting the capital of one of the now-independent Baltic states, carrying a sample of the material from his archive concealed in his luggage. He crossed the border with no problems and walked in to the British embassy to make a deal. After several more trips, interviews with British Secret Intelligence Service (SIS) officers, and providing more sample material, the British agreed to arrange the exfiltration of Mitrokhin, his entire family, and the entire archive—six cases of notes. He was debriefed at a series of safe houses in Britain and began several years of work typing handwritten notes, arranging the documents, and answering questions from the SIS, all in complete secrecy. In 1995, he arranged a meeting with Christopher Andrew, co-author of the present book, to prepare a history of KGB foreign intelligence as documented in the archive.

Mitrokhin's exfiltration (I'm not sure one can call it a “defection”, since the country whose information he disclosed ceased to exist before he contacted the British) and delivery of the archive is one of the most stunning intelligence coups of all time, and the material he delivered will be an essential primary source for historians of the twentieth century. This is not just a whistle-blower disclosing operations of limited scope over a short period of time, but an authoritative summary of the entire history of the foreign intelligence and covert operations of the Soviet Union from its inception until the time it began to unravel in the mid-1980s. Mitrokhin's documents name names; identify agents, both Soviet and recruits in other countries, by codename; describe secret operations, including assassinations, subversion, “influence operations” planting propaganda in adversary media and corrupting journalists and politicians, providing weapons to insurgents, hiding caches of weapons and demolition materials in Western countries to support special forces in case of war; and trace the internal politics and conflicts within the KGB and its predecessors and with the Party and rivals, particularly military intelligence (the GRU).

Any doubts about the degree of penetration of Western governments by Soviet intelligence agents are laid to rest by the exhaustive documentation here. During the 1930s and throughout World War II, the Soviet Union had highly-placed agents throughout the British and American governments, military, diplomatic and intelligence communities, and science and technology projects. At the same time, these supposed allies had essentially zero visibility into the Soviet Union: neither the American OSS nor the British SIS had a single agent in Moscow.

And yet, despite success in infiltrating other countries and recruiting agents within them (particularly prior to the end of World War II, when many agents, such as the “Magnificent Five” [Donald Maclean, Kim Philby, John Cairncross, Guy Burgess, and Anthony Blunt] in Britain, were motivated by idealistic admiration for the Soviet project, as opposed to later, when sources tended to be in it for the money), exploitation of this vast trove of purloined secret information was uneven and often ineffective. Although it reached its apogee during the Stalin years, paranoia and intrigue are as Russian as borscht, and compromised the interpretation and use of intelligence throughout the history of the Soviet Union. Despite having loyal spies in high places in governments around the world, whenever an agent provided information which seemed “too good” or conflicted with the preconceived notions of KGB senior officials or Party leaders, it was likely to be dismissed as disinformation, often suspected to have been planted by British counterintelligence, to which the Soviets attributed almost supernatural powers, or that their agents had been turned and were feeding false information to the Centre. This was particularly evident during the period prior to the Nazi attack on the Soviet Union in 1941. KGB archives record more than a hundred warnings of preparations for the attack having been forwarded to Stalin between January and June 1941, all of which were dismissed as disinformation or erroneous due to Stalin's idée fixe that Germany would not attack because it was too dependent on raw materials supplied by the Soviet Union and would not risk a two front war while Britain remained undefeated.

Further, throughout the entire history of the Soviet Union, the KGB was hesitant to report intelligence which contradicted the beliefs of its masters in the Politburo or documented the failures of their policies and initiatives. In 1985, shortly after coming to power, Gorbachev lectured KGB leaders “on the impermissibility of distortions of the factual state of affairs in messages and informational reports sent to the Central Committee of the CPSU and other ruling bodies.”

Another manifestation of paranoia was deep suspicion of those who had spent time in the West. This meant that often the most effective agents who had worked undercover in the West for many years found their reports ignored due to fears that they had “gone native” or been doubled by Western counterintelligence. Spending too much time on assignment in the West was not conducive to advancement within the KGB, which resulted in the service's senior leadership having little direct experience with the West and being prone to fantastic misconceptions about the institutions and personalities of the adversary. This led to delusional schemes such as the idea of recruiting stalwart anticommunist senior figures such as Zbigniew Brzezinski as KGB agents.

This is a massive compilation of data: 736 pages in the paperback edition, including almost 100 pages of detailed end notes and source citations. I would be less than candid if I gave the impression that this reads like a spy thriller: it is nothing of the sort. Although such information would have been of immense value during the Cold War, long lists of the handlers who worked with undercover agents in the West, recitations of codenames for individuals, and exhaustive descriptions of now largely forgotten episodes such as the KGB's campaign against “Eurocommunism” in the 1970s and 1980s, which it was feared would thwart Moscow's control over communist parties in Western Europe, make for heavy going for the reader.

The KGB's operations in the West were far from flawless. For decades, the Communist Party of the United States (CPUSA) received substantial subsidies from the KGB despite consistently promising great breakthroughs and delivering nothing. Between the 1950s and 1975, KGB money was funneled to the CPUSA through two undercover agents, brothers named Morris and Jack Childs, delivering cash often exceeding a million dollars a year. Both brothers were awarded the Order of the Red Banner in 1975 for their work, with Morris receiving his from Leonid Brezhnev in person. Unbeknownst to the KGB, both of the Childs brothers had been working for, and receiving salaries from, the FBI since the early 1950s, and reporting where the money came from and went—well, not the five percent they embezzled before passing it on. In the 1980s, the KGB increased the CPUSA's subsidy to two million dollars a year, despite the party's never having more than 15,000 members (some of whom, no doubt, were FBI agents).

A second doorstop of a book (736 pages) based upon the Mitrokhin archive, The World Was Going our Way, published in 2005, details the KGB's operations in the Third World during the Cold War. U.S. diplomats who regarded the globe and saw communist subversion almost everywhere were accurately reporting the situation on the ground, as the KGB's own files reveal.

The Kindle edition is free for Kindle Unlimited subscribers.