(Image: Tim McDonagh)
Data-mining the archives of London’s oldest criminal court has uncovered an unguessed-at story of violence – and history will never be the same again
Fire up the time machine, pop it in reverse and rewind 213 years. Mary Hall is standing in the dock at the Old Bailey, London’s oldest court, as a string of witnesses give evidence against her. A few days before, Hall had picked up a labourer named John Martin stumbling home from the pub. “She asked me what I would give her for a bed that night,” Martin told the court. “I agreed to give her two shillings.”
Advertisement
The next morning, however, Martin caught Hall stealing from his purse. Martin grabbed her to get the money back and Hall cried out. A man wielding a knife burst into the room and threatened to stab Martin in the heart. Another woman hit him with a pair of tongs. As Martin bolted, fearing for his life, yet another man struck him from behind. His attackers tossed his shoes and stockings out of the window into the street, where he dressed and went to fetch the police. Despite the violent assault, only Hall was arrested. For the theft of 5 guineas – about a year’s wages for a servant – she now faced a prison ship to Australia or, worse, the gallows.
Serious crimes have been tried at the Old Bailey for more than 400 years. Hall’s trial is just one of roughly 200,000 recorded in The Proceedings of the Old Bailey, a detailed account of nearly every case heard within its walls between 1674 and 1913. The archive is the largest body of recorded direct speech in the world. “All of human life is there,” says historian at the University of Sussex, UK.
In a 10-year undertaking, the have been digitised, tagged and published online. And the Old Bailey is just the start. From transcripts of US congressional debates since the 1980s to parliamentary records from the French Revolution’s year-long Reign of Terror from 1793 to 1794, historical archives are being digitised across the board. Vast libraries that were once for human eyes only can now be read by computers. And that changes everything – history is becoming a science. “Once you digitise, you can get a birds-eye view that would have been impossible before,” says , a historian at Stanford University in California. For the first time, we can begin to quantify unknowns that historians would otherwise have overlooked, she says.
The sheer scale of many written archives means that historians typically have no choice but to be selective in what they read. The Old Bailey archive, for example, runs to 137 million words. “If you sit down and start reading at the beginning you won’t get to the bit you’re interested in for 10 years,” says Hitchcock. So historians have zoomed in on fragments, looking at pickpockets in the 1800s, say, or piecing together longer trends from a few significant samples. Here digitisation has helped. Tagged and searchable archives make it easy to connect the dots between different records. A quick search in the England & Wales Criminal Register, for example, tells us that Hall was born in 1772, making her 29 at her trial.
Quantative revolution
The potential of digitisation goes far beyond stitching together different documents, though. Historian at Brown University in Providence, Rhode Island, thinks her field is standing on the brink of a quantitative revolution – one that will enable historians to tackle questions previously beyond the scope of human research alone.
Hitchcock is at the forefront of this new way of doing history and played a part in the digitisation of the Old Bailey archives. For Hitchcock, these court records have a lot to reveal about the civilising process in Western society. Each jury trial is a key moment when a place and its citizens come together to define the limits of acceptable behaviour. But he was frustrated by digital history that merely skims the surface. The trick would be to dig deep and shine a light on patterns that nobody had seen before. The information was there in the archives. The question was how to get at it.
Enter of Indiana University in Bloomington. Trained as a cosmologist, DeDeo now studies complexity closer to home. “Simon is interested in finding patterns in large-scale chaotic systems,” says Hitchcock. “Culture is the largest and most chaotic system you could imagine.”
Introduced by a mutual friend, Hitchcock and DeDeo hatched a plan in a London pub to merge history with mathematics. Close reading of selected sources – the traditional method of historians – gives a deep but narrow view. The duo wanted to combine the best of both their worlds: to take the close reading approach but apply it across the whole of the Old Bailey’s digitised archive, hoping to reveal patterns that would otherwise have remained hidden in the wider arc of history. “If you want to understand a social process at that scale and at that level of detail – without reducing it to some aggregate statistic – you have to use some kind of machine-aided reading,” says DeDeo.
Working with Sara Klingenstein at the Santa Fe Institute in New Mexico, they decided to concentrate on the 150-year period between 1760 and 1913 because it produced the most accurate records. Before 1760, says Hitchcock, many trial accounts read like true crime tabloids, heavily edited to make the accused appear as guilty as possible. But by the mid-18th century, transcripts had become full and fair representations of trials, produced by shorthand stenographers hired for the purpose. Still, a lot happened in those 150 years, including the American and French revolutions, the Napoleonic wars, and the industrial revolution. A machine-reading approach would need to tease apart patterns specific to the courtroom from those that were part of a wider trend.
3D space
DeDeo knew he had to come up with something more innovative than yet another data mining algorithm, but he struggled at first to find the right mathematical tool. What was the best way to turn such a vast collection of prose into a usable dataset? Thinking like a physicist, he initially tried modelling the archive as a 3D space. But the results were inconsistent and difficult to interpret. For a start, when it comes to information, relations between points are often asymmetric. You can learn about biology from an encyclopedia, for example, but you would learn little about the contents of an encyclopedia from a book about biology. “The space of information is not the space of things,” says DeDeo.
Instead, he turned to information theory, invented by Claude Shannon in the 1940s. DeDeo’s aim was to reveal gradual changes in the way crimes were spoken about. He split all the trials into two categories – trials for violent crimes like murder or assault and trials for non-violent crimes like pickpocketing or fraud – and then looked at the actual words that people used in the courtroom. Information theory lets you quantify the amount of information given by a word in a specific context. Using a measure known as Jensen-Shannon divergence (see diagram), a word picked at random from the transcript of a trial can be given a score based on how useful it is for predicting the type of the trial.
So, for example, if you walked into the Old Bailey during Hall’s trial and heard the word “murdered” uttered in court, how much information about the type of trial underway would that single word convey? In the early years of the period they looked at, most crimes involved some level of violence. “There might be bloodshed, or an eye gouged out, but the real crime is someone’s wallet got stolen,” DeDeo says. “The casual everyday violence of the past is remarkable.”
Even though Martin testified that he thought he was about to be murdered by his attackers, Hall’s trial was for non-violent theft. “In the late 18th century you could find violence everywhere,” says Hitchcock. If your watch was stolen you’d probably be hit in the face but the thief would only be tried for stealing the watch.
Slowly, however, that changed. By the 1880s, the team found that the vast majority of violent language was reserved for talking about crimes like assault, murder, or rape. So you could walk into the courtroom, hear words like “murdered”, “hit,” “knife” and “struggled” – all words from Martin’s testimony in 1801 – but instead be confident that you were witnessing a trial for a violent crime.
The analysis reveals a story of the gradual criminalisation of violence. This is not necessarily evidence that we have become less violent – as Steven Pinker argues, based on statistics for violent crime, in his book The Better Angels of Our Nature. Rather, it is a story of the state gaining a monopoly on violence and controlling its occurrence among the public. “What is deemed criminal has changed,” says Hitchcock.
DeDeo likens the shift to genetic drift. If you took two herds of goats and left each herd on a separate island for 500 years, those herds would evolve into separate species. Similarly, he sees the Old Bailey cases as populations of violent and non-violent trials. Over time the two types of trials “speciate” and become distinct from one another.
“In 1760, the patterns of language used in both kinds of trial are almost exactly identical,” he says. “Over the next 150 years they diverge.”
This is not apparent in crime rates alone. It requires an analysis at the level of the meaning of words – how people actually spoke. For Guldi, the power of this approach is that it goes beyond counting individual acts of violence and considers the relationship of words to crimes and court rulings. Close reading is combined with exhaustive coverage.
This isn’t about to spell the end for historians, though. Machines can reveal patterns, but these need to be interpreted in the context of an existing historical narrative. “The data is politically loaded, so you have to know the story to correctly interpret it,” says DeDeo. Winterer agrees: “It would be like saying, ‘Now we have lawnmowers, we can use them to cut everybody’s hair.’ We still need the skilled intervention of a barber.”
Winterer has used the digitised letters of polymath Benjamin Franklin to map his “social network”, revealing a picture of his rise to global prominence that was previously hidden. She found that the majority of his correspondence was not with key players but with lowlier, like-minded, individuals. Yet Winterer points out that the network itself is still only a part of the picture. Franklin exchanged many hundreds of letters with his most frequent correspondents but relatively few with a fellow scientist called Peter Collinson, for example. The Collinson letters, however, are lengthy and bursting with details of the electrical theories and experiments that would make Franklin famous.
Digital grinder
Still, the way historians think about the past is changing. “It’s very difficult to take an archive and read it differently from the people who created it,” says Hitchcock. “But digitally, you can grind it down, turn it over and shake it to see what comes out.”
Armed with their digital tools, historians are now turning to other archives. Stanford University recently released the digital records for parliamentary debates during the French revolution. And DeDeo is particularly keen to trawl the digital archives of US congressional debates over the past 30 years. These are full of buzzwords that reveal the political leanings of speakers – “pro-life” versus “pro-choice” and “estate tax” versus “death tax”, for example. He thinks a similar approach to that used with the Old Bailey records could track the development of political parties and the origins of their current polarisation via subtle shifts in rhetoric.
In the case of the French revolution, matters are even more complicated. “I don’t think we can answer the underlying question – how did things go so wrong? – with this sort of analysis alone,” says DeDeo. “But I think we can learn a great deal about the turbulence of ideas and how discussion and debates channel, merge, and fragment over time.” For DeDeo, gaining access to a past mindset is crucial. Too often cultural shifts are glossed in evolutionary terms, he says. “I think these explanations run the risk of minimising the cognitive complexity of the individual.”
Ultimately, DeDeo wants to invent a mathematical framework that captures social behaviour – one that models the theory of mind, the driving force behind much of social complexity. Our capacity to choose how to behave is based on how well we predict or infer what others are thinking. “If you believe X, then I might decide to do Y,” says DeDeo. He sees the same thing at play in the Old Bailey too. “Prosecutors make the case they do based on their beliefs about what juries will respond to,” he says. “Victims and defendants do the same. Everyone, to a greater or lesser extent, is engaged in a competitive prediction game.”
DeDeo compares his foray into social history with the early days of biology, when the only available theoretical framework was the mechanics of Newtonian physics. It wasn’t until Charles Darwin introduced the theory of evolution that biologists finally had the right set of tools to make real progress. “We’re pre-Darwinian when it comes to how we think about social behaviour,” said DeDeo.
In the end, Hall escaped the gallows but wound up on a long voyage to Australia. If DeDeo has his way, we may soon know the mind of the judge who sent her there.
See the evidence for yourself: “A rogue’s gallery from Victorian London’s crime data“
