Disorder and Randomness

Meccano Microcosm: Obsessed with the mystery of randomness, I made this machine blending order and randomness at the end of the 1970s as an agreeable pass time. See here  for the story

I have recently compiled a book-length electronic document containing some of my earlier writings on disorder and randomness. This compilation can be accessed as a PDF here  I worked on this subject as a hobby throughout the 1970s and 1980s and I now publish what was originally a typescript. I have, however, enhanced the content as well as the format.  The current publication is edition 1, but it is likely that I will re-read the document in a few months time and produce edition 2. After a while I get bored with reading the same old text and I have to wait some weeks before the desensitization wears off and I can continue to spot errors, omissions and naiveties. 

Below I reproduce the introduction to the book: 

Introduction

This book deals with the subject of randomness. It presents a personal record of my engagement with a topic which, early in my thinking career, presented itself as something of fundamental importance: The algorithmic nature of deterministic physical laws seemed relatively easy to grasp, but what was this thing called indeterminism? How would we know when it was operating?  Above all, what was this strange discipline of predictive statistics? It was strange because no one expects its predictions to be exactly right. A naive philosophical stance might think that a miss is as good as a mile and therefore anything registering less than 100% truth as nothing short of 100% error; but no, in statistics the concepts of approximation and nearness figure prominently and give the lie to fundamentalist sentiments that anything less than absolute certainty isn’t worthwhile knowledge.

In the short Sherlock Holmes story The Cardboard Box, Holmes solves a particularly tragic crime of passion; both the killer and his victims are ordinary people who against their better judgement are driven by overwhelming human emotions into a conflict that leads to murder. If you got to know these people you would find none of them to be particularly immoral by human standards. In a cool moment they would likely condemn their own hot blooded actions. After bringing the matter to a conclusion in his usual consummate and controlled way, Holmes betrays a rare moment of being completely at a loss as to understanding the meaning of the affair. The details of the crime he guided us through, where all the clues formed a rational pattern which Holmes alone completely understood, stand in stark contrast to his utter mystification at the human significance of the story as a whole:

“What is the meaning of it, Watson?” said Holmes, solemnly, as he laid down the paper.  “What object is served by this circle of misery and violence and fear? It must tend to some end, or else our universe is ruled by chance, which is unthinkable. But to what end? There is the great standing perennial problem to which human reason is as far from an answer as ever.”

In this expression of the problem of suffering and evil, chance and meaninglessness are seen as be related: A world ruled by chance is purposeless, or at least it is difficult to discern purpose. Although the exorable mechanisms of determinism can themselves conspire with chance to generate meaningless scenarios lacking in anthropic significance, the background fear expressed by Holmes is that chance is ultimately sovereign. For Conan Doyle’s Victorian God-fearing character this was unthinkable. Today, of course, it is no longer unthinkable, although for many a very uncomfortable thought. But what is “chance” and just why should it come over as the epitome of meaninglessness and purposelessness? Since the rise of the quantum mechanical description of reality chance figures very strongly in today’s physical science and raises profound questions about the nature and meaning (or lack of meaning) of chance’s manifestation as randomness in the physical world.

Arthur Koestler in his book The Roots of Coincidence spelled out some of the paradoxes of chance events and their treatment using probability theory. In his book Koestler’s interest in the subject arises from his interest in those extrasensory perception experiments which make use of card guessing statistics and where there is, therefore, a need to determine whether the outcome of such experiments is statistically significant. Perhaps, suggests Koestler, the apparently skewed statistics of these experiments are bound up with the very question of just what randomness actually means. After all, in Koestler’s opinion randomness itself seems to have a paranormal aspect to it. To illustrate he gives us a couple of examples where statistics has been used to make predictions: In one case statistics was used by a German mathematician  to predict the distribution of the number of German soldiers per year between 1875 and 1894 kicked to death by their horses. In another case Koestler remarks on the near constancy of the number of dog-bites-man reports received by the authorities in New York. In response to the idea that probability theory leads to an understanding of this kind of “statistical wizardry”, as Koestler calls it, he says this:

But does it really lead to an understanding? How do those German Army horses adjust the frequency of their lethal kicks to the requirement of the Poisson equation? How do the dogs of New York know that their daily ration of biting is exhausted? How does the roulette ball know that in the long run zero must come up once in thirty seven times if the casino is to be kept going? (The Roots of Coincidence, Page 27)

I have to admit that for some time after reading Koestler I too was puzzled by the success of probability theory’s paradox of being able to predict the unpredictable. At the time I knew how to use probability calculus but I had really no idea how and why it worked. Later on in his book Koestler introduces us to mathematician G. Spencer Brown who was researching probability in the 1950s. Of him Koestler says:

...among mathematicians G. Spencer Brown proposed an intriguing theory which attempted to explain the anti-chance results in card guessing experiments by questioning the validity of the concept of chance itself. (The Roots of Coincidence, Page 102)

Koestler tells us that Spencer Brown proposed that ESP statistics “pointed to some anomaly in the very concept of randomness”. Koestler goes on to quote Sir Alister Hardy who sponsored Spencer Brown:

…It remained for Mr. G. Spencer Brown of Trinity College, Cambridge, to suggest the alternative and simpler hypothesis that all this experimental work in so called telepathy, clairvoyance, precognition and psycho-kinesis, which depends upon obtaining results above chance, may be really a demonstration  of some singular and very different principle. He believes that it may be something no less fundamental or interesting – but not telepathy or these other curious things – something implicit in the very nature and meaning of randomness itself… In passing let me say that if most of this apparent card-guessing  and dice influencing  work should turn out to be something very different, it will not have been wasted effort; it will have provided a wonderful mine of material for the study of a very remarkable new principle.  (The Roots of Coincidence, Page 103).

Koestler says that Spencer Brown’s work petered out inconclusively. But whilst we continue to feel that we really don’t understand why it is that probability theory works then Brown’s ideas remain plausible: Perhaps we’ve got probability theory wrong and this accounts for the unfamiliar statistics which apparently show up in paranormal research involving card guessing and dice influencing. Whether or not there was something to Brown’s ideas, at the time I first read Koestler in the early 1970s I couldn’t tell. But for me the “near miss” effectiveness of probability calculus was a nagging problem that remained with me for some time. At university I had taken it for granted that there was no mystery in probability theory, but a social science student whose name I have long since forgotten challenged me with Koestler’s comments. These comments made me realize that I although I could use probability calculus I didn’t really understand why it worked. However, by the time I finished the work in this book, although inevitably there were still questions outstanding, I felt fairly confident that the kind of thing Spencer Brown was looking for did not in fact exist if true randomness was in operation. In the epilogue I briefly take up this matter along with the question raised by Sherlock Holmes.

During the mid-seventies I had dabbled a little with quantum theory with the aim of trying to make anthropomorphic sense of it, but after a while I shelved the problem for later. (The sense I did eventually make of it is an ongoing story I tell elsewhere). However, in the 70s my dabblings with quantum theory gave way to what was in fact, for reasons I have already given, a philosophically more pressing problem; Viz; the nature of randomness and probability. In any case the probability question arose very naturally because probability is at the heart of quantum theory; if I didn’t understand the meaning of the language of probability in which quantum mechanics was formulated how I could understand the nature of quantum theory? The gradual shift in my interests from quantum theory to probability started to produce results immediately. I pursued the idea of taking disorder maxima of binary sequences and rediscovered the statistics of Bernoulli’s formula by way of a very long-winded maximization proof. But as I continued with this line of inquiry it become increasingly apparent that probability and randomness are not the same thing: Probability is to do with our information or knowledge about systems and, in fact, arises in connections where randomness is not present (I have presented more on the subject of probability in my 1988 paper on probability). Randomness, on the other hand, is to do with patterns, specifically, as we shall see, patterns about which practical algorithmic methods of prediction reveals only maximum disorder. This means that a pattern can be disordered and yet not probabilistic; for example, if we have a book of random numbers generated, say, by some quantum process, the pattern is going to be disordered or random, but in the sense that the book has captured the pattern and it can now be thought of as part of human knowledge this pattern is no longer probabilistic.

As I propose in the epilogue of this book, the reason for the close connection between probability and disordered patterns such as generated by quantum processes is that disorder is epistemically intractable until it becomes a recorded happened event; for as this work attempts to demonstrate randomness is not practically predictable by algorithmic means; therefore up until the random pattern is generated and recorded it remains unknown to those who only have available practical methods of making predictions. A random pattern is thereby probabilistic by virtue of it being intractable to practical algorithmic precognition.

The bulk of the mathematics in this book was written in the late seventies and eighties. I eventually produced a typescript document in 1987 and the contents of this typescript provide the core of the book. Like most of my work, all of which is a product of a private passion, this manuscript will be of no interest to the academic community as they have their own very professional way of answering similar questions; the book presents the record of a science hobbyist. As such it is only of autobiographical interest.