This Time is Different

18 years ago, a myopic faith in logic, maths, and computer models helped cause a global financial cataclysm. It could never happen again. Could it?

“Ils n’ont rien appris, ni rien oublié.”

“They have learned nothing and forgotten nothing.”

Charles-Maurice de Talleyrand-Périgord

Sacked in the Morning

It was a Saturday night in September.

Like most Saturday nights in that part of my life, I was in a bar somewhere in central London. If memory serves, this particular Saturday was a university friend’s engagement party.

Most of my university friends, like me, worked in and around the banking industry, and at some point conversation turned to the rumours swirling around Lehman Brothers, whose demise was reportedly imminent. The collapse of a major bank like Lehman was unthinkable; conventional wisdom held that institutions of such scale and centrality to the system were “too big to fail” and would be bailed out by the authorities.

That was certainly the consensus view among our little crowd, so much so that at some point in the later stages of the evening, we began serenading the one member of the group who worked for Lehman:

“Sacked in the morning, you’re getting sacked in the morning”. (Sung to the tune of Guantanamera, a Cuban folk song widely adopted by English football fans for all manner of insulting ditties like this one.)

In the logic of twenty-something boys on the piss, this was hilarious precisely because it was never going to happen.

On Monday morning, it happened.

The story of the sub-prime crisis and the global financial meltdown that followed has been told many times, perhaps most insightfully in Andrew Ross Sorkin’s book Too Big To Fail, Gillian Tett’s Fool’s Gold (listen to my conversation with Gillian here), and most memorably in Adam McKay’s film adaptation of Michael Lewis’ The Big Short.

The crisis was caused by a complex web of causes overlapping and interacting with each other, including macroeconomic imbalances between Asia and the rest of the world, US government housing policy, a prolonged period of low interest rates, inadequate regulatory oversight of financial markets, increasingly sophisticated financial engineering, misaligned incentives, groupthink, confirmation bias, and old-fashioned greed.

Underpinning these various causes were misguided assumptions about human nature, financial markets and technology, and a naively optimistic faith in human ability to understand and control the world. Those misplaced assumptions aren’t unique to the banking industry.

The House Always Wins

You’re probably familiar with the tabloid narrative around the subprime crisis. Greedy “casino” bankers, addicted to the high life provided by their vast bonuses, became increasingly reckless, taking bigger and bigger risks with other people’s money, consequences-be-damned, aided and abetted by incompetent regulators who took their eyes off the ball.

US Senator for Illinois and a former retail banker himself, Roland Burris, summed up this popular view in a speech to the Senate in April 2010:

“So Wall Street has basically turned into a casino, and it has done so at our expense. These fat-cat bankers were gambling not just with our money but with our economic future.”

US Senator Roland Burris

Emotive and no doubt popular though this narrative was, it’s only partially true, at best. As Howard Davies (previously chairman of the UK’s Financial Services Authority (FSA) and Deputy Governor of the Bank of England) pointed out, casinos are a terrible analogy, for two reasons.

Firstly, casino operators know very well that they’re gambling, whereas the bankers’ folly was believing that they weren’t. Secondly, the casinos are very good at it: “the house always wins”, as the old adage has it. In 2008 the house most certainly did not win, as a string of catastrophic bankruptcies attests, Lehman chief among them.

In my experience, the trading floor at Credit Suisse (like every other bank) certainly had its share of larger-than-life characters straight out of central casting for an imaginary “Casino Bank” movie: ex-public school rugby players in chinos and pink shirts, street smart cockney wideboys with the gift of the gab, and perma-tanned polyglot Europeans with designer shoes and hyphenated surnames matching their hyphenated nationalities. 

Slightly before my time at the bank, one group of Credit Suisse traders, including the son of wildly successful novelist and disgraced Conservative politician Jeffrey Archer, attained global notoriety for their flamboyant behaviour after styling themselves “the Flaming Ferraris”. Over the years, Hollywood has made plenty of movies about these kinds of antics by these kinds of people: Bonfire of the Vanities, Wall Street, American Psycho, and The Wolf of Wall Street, not to mention TV shows like Billions and Industry.

There are few cinematic portrayals, however, of the other major tribe on the trading floors of banks, brokerages, and hedge funds, because at least superficially, they’re not as interesting as cocaine snorting, champagne quaffing, midget tossing sociopaths. But while the world worried about the Flaming Ferraris propping up the bar, maths nerds with PhDs in physics and spreadsheets that no-one could understand were busy creating weapons of math destruction. Known in the banking world as “quants”, these often unassuming characters are in many ways the most dangerous people in finance.

Six months after Lehman collapsed, the British government published its postmortem. Buried in the Turner Review‘s 122 pages of sober analysis, the section titled “misplaced reliance on sophisticated maths” stands out. In Turner’s words:

“…the very complexity of the mathematics used to measure and manage risk, moreover, made it increasingly difficult for top management and boards to assess and exercise judgement over the risks being taken.

Mathematical sophistication ended up not containing risk, but providing false assurance that other prima facie indicators of increasing risk (e.g. rapid credit extension and balance sheet growth) could be safely ignored.”

Adair Turner

In my previous essay, Mind the Gap, I wrote about the dangers of confusing models with reality, quoting Alfred Korzybski’s classic aphorism, “the map is not the territory”. As the Turner Report suggests, confusing the map for the territory, the model for the reality, was absolutely at the heart of the subprime crisis.

That confusion was, though, the symptom of a deeper, more fundamental malaise. In The Way the World Ends, I explored the wild and terrifying beliefs shared by many Silicon Valley CEOs: the Extropian, Transhumanist fantasies that technology and the brute force application of scientific rationalism will overcome every limit and enable humans to live forever, merge with machines, and colonise the universe.

While the derivatives traders and quantitative structurers of the trading floor didn’t dream of intergalactic immortality, they too belonged to an intellectual culture that systematically favoured formal, mathematical, analytical knowledge over messy, tacit, social and contextual knowledge. It’s a worldview that extends well beyond Wall Street.

Engineer’s Disease

The first half of 2025 was a strange time in America. The second Trump Presidency began with just as much talk as the first, eight years earlier, but a lot more action. Among hordes of controversial figures, the new administration had one undoubted superstar. For a few tumultuous months, Elon Musk led something called “DOGE”: the Department of Government Efficiency, an ostensibly advisory body setup to drive cost cutting, and in Musk’s words, “end the tyranny of the bureaucracy”.

The principle was simple: treat the Federal Government as a software engineering problem, fix glitches in the code, get rid of bad data, delete people or agencies that (in Musk’s view) weren’t necessary. As The Guardian’s excellent retrospective on DOGE put it:

“If the state was just a database, then inefficiency came from bad data: undocumented foreigners, ghost employees, even “vampires” collecting social security.

These were bugs in the codebase, irregularities to be traced, quarantined and purged. Musk had revamped and retrained Twitter into X. To him, the US state was just another system – a glitchy dataset to be scrubbed and optimised.”

Musk assembled a team of his loyal lieutenants, many of them twenty-something software engineers with no experience of government, or anything at all outside Silicon Valley, and set about the task with gusto. Depending on your political perspective, your opinion of DOGE’s intent and results will vary substantially. What’s relevant for our purposes, however, is the underlying assumptions.

Musk’s worldview is a vivid expression of a sentiment common in Silicon Valley (and far beyond). If you’ve been highly successful at solving problems through software development and (traditional) engineering, you tend to assume those methods will successfully translate to every domain. This is both an assumption about the power of those techniques, and about your own intelligence.

It’s also, implicitly, an assumption about the world: that like computer code or an engine it can be broken down into smaller and smaller chunks and understood as a logical, mechanical system. It’s the belief that the world is nothing more or less than a set of engineering problems.

Colloquially, this set of beliefs is known as “engineer’s disease”. The condition is a sickness afflicting ideas, a sickness of worldview, a delusion causing sufferers to imagine they understand the world far better than they really do, and that they have far more control over it than they really do.

More than half a century before the subprime crisis, Friedrich Hayek named the tendency “Scientism”. In the aptly titled The Counter-Revolution of Science: Studies on the Abuse of Reason, Hayek decried the increasing encroachment of methods from the hard sciences onto the altogether messier world of human and social phenomena.

A decade prior to Lehman’s collapse, political scientist James C. Scott’s seminal book Seeing Like A State analysed dozens of historical examples of the failure of this scientistic worldview, which he termed “high modernism”.

Across a dizzying array of case studies, from the artificial precision of Prussian scientific forestry, to the clean, clinical architecture of Le Corbusier, via Lenin’s top-down vision of state socialism, Scott identifies the organising principle of so many ill-fated human schemes as the quest for “legibility”. In order to control the world, we must first make it legible; in order to make it legible, we must simplify and flatten it into a logical model.

Reductio ad Absurdum

That misplaced quest for legibility was at the heart of the subprime crisis. As self-styled financial engineers created ever more complex products, they increasingly relied on reductive, mechanical, mathematical modelling to make the mortgage market legible. So-called “mortgage bonds” bundled hundreds of individual mortgages together, transforming messy human obligations into streams of expected cashflows (ie each borrower’s monthly repayments).

Reality, however, isn’t quite as simple as that. The mortgage market doesn’t consist of millions of independent actors, it’s a highly complex, highly interconnected dynamic human system. A dizzying array of participants including home buyers, home sellers, real estate agents, banks, ratings agencies, central bank rate setters, investors, regulators, and the media, all with their circumstances and motivations, respond to each others’ actions and imagined intentions in a fiendishly complex system of millions of interrelated feedback loops.

The people creating these products weren’t stupid, far from it. They knew the mortgage market was complex and that there was likely to be some correlation between individual borrowers’ ability to repay, and they set out to model those correlations. The assumptions they made in constructing the models became part of a much bigger financial machine whose collective assumptions about human behaviour became increasingly and catastrophically divorced from reality.

Amid the fallout from the Lehman collapse, some journalists pointed the finger at the maths itself, particularly the exotic-sounding “Gaussian Copula”. A notorious article in Wired magazine was headlined Recipe For Disaster: The Formula That Killed Wall Street.

For a financial engineer trying to assess the risk of bundling thousands of loans together, there’s one crucial question: how likely is it that these borrowers are simultaneously unable to repay their loans?

Imagine all the factors impacting one borrower’s likelihood of defaulting on a home loan: relationship, career, health, broader economy, house prices, the list is almost endless. Now imagine scaling that across a large population: the relationship between thousands of borrowers’ likelihoods of default is surely unknowable. Any usable model therefore had to make simplifying assumptions about correlation.

That’s where the “Gaussian Copula” comes in. Using a familiar bell curve framework, it provides an elegant mathematical means of connecting individual default probabilities into a joint probability.

An impossibly complicated question involving the unknowable interactions of millions of human beings could now be converted into something that computers could calculate and traders could price. Messy reality had been made clean, legible, and predictable. Or so it seemed.

That Wired headline inspired the title of a brilliant study of the strange financial engineering culture that led to the subprime crisis. In their 2014 paper ‘The formula that killed Wall Street’: The Gaussian copula and modelling practices in investment banking, University of Edinburgh Sociologists Donald MacKenzie and Taylor Spears investigated the quant world which spawned the Gaussian copula.

The pair reviewed technical documentation and conducted 114 interviews with market participants, more than a quarter of which actually pre-dated the beginning of the crisis in 2007. What they found was a strange subculture of highly educated technical specialists, often with PhDs in mathematical subjects and expertise in computer programming.

Like many professional subcultures, this one was small and unusually interconnected. While their employers were competing, often ferociously, with each other, the quant world extended across organisations with quants from different banks attending the same conferences, reading the same papers, and applying for the same jobs.

Like any professional subculture, the world of the quants had its own shared incentives and constraints. Intellectually, quants prized the mathematical elegance of models. Practically, computational requirements limited the complexity of models and the number of parameters that could be computed. The requirement to revalue trading books every night made calculation speed crucial, encouraging simplification of the models.

MacKenzie and Spears show how a useful simplification can become an industry-wide way of seeing. The Gaussian copula made the messy problem of correlated defaults mathematically manageable; once embedded in trading, pricing and risk management, its abstractions began to acquire the status of reality.

MacKenzie and Spears tell a story that’s more nuanced and more subtle than it might seem. This wasn’t simply a case of credulous quants blindly worshipping equations. Rather, many of the people closest to the models understood their limitations. Indeed, many expressed concerns in interviews with the researchers long before the crisis unfolded.

Yet by then, the models had become part of the machinery operating the market, relied on by traders, product controllers, risk managers, ratings agencies, regulators, and market analysts. Crucially, many of those people lacked the mathematical capability to understand those models, let alone question them. Andrew Ross Sorkin’s magisterial history of the crisis makes clear that very few Wall Street executives understood the finer details of the mortgage-backed securities and exotic credit derivatives lurking on their vast balance sheets.

The models weren’t just measuring reality, but shaping it. The map had become part of the territory.

California Dreamin’

Is it possible that a similar dynamic is unfolding today as organisations and societies grapple with AI?

Much like Wall Street quants, Silicon Valley is a strange subculture, insular and incestuous in equal measure. Just as the investment banks compete ferociously with each other, so too do the AI companies.

Yet despite the competition, the similarities are often greater than the differences. Just as MacKenzie and Spears observed that the quants did the same degrees at the same elite universities, attended the same conferences, and read the same publications, so too the AI researchers share a similar intellectual milieu. Their worldview, or “evaluation culture”, cuts across their organisational boundaries.

What’s more, such a shared evaluation culture isn’t one that necessarily lends itself to outside perspectives. In The Structure of Scientific Revolutions, Thomas Kuhn famously observed that science progresses not only through gradual evolution, but also via occasional dramatic revolutions where paradigms shift dramatically.

In between those periodic revolutions, “normal science” consists of puzzle solving within a shared intellectual paradigm, without ever really questioning that paradigm. Participants disagree ferociously about the answers, while agreeing wholeheartedly about the questions, the intellectual framework, and crucially, what counts as evidence.

There are striking similarities between the intellectual frameworks of AI and quantitative finance. Much like the quants stalking the trading floors of investment banks, many of the leading lights of the AI revolution are gifted mathematicians, physicists, and of course, computer programmers. These academic backgrounds skew heavily male and heavily towards a particular type of intelligence, as a senior engineer at Anthropic recently observed in Vanity Fair:

The whole town [San Francisco] can seem neurodivergent. Daniel Freeman, a senior engineer with a PhD in physics and papers published in all the major AI journals, says, “That is absolutely true. I don’t think it comes up too much, because it’s just seen as normal here.”

Anthropic, Freeman says, is “an extremely welcoming place to neurodivergent people.” As hyper-rationalists, they are “very highly decoupled, very non-emotive, but largely trying to be very rational and very epistemically grounded. These are somewhat appealing norms to certain types of neurodivergence.”

There’s a deep irony in the word “neurodivergence”. Freeman describes Silicon Valley as a place which isn’t so much neurodivergent, but more a monoculture where one cognitive style predominates. Indeed, Freeman seems to be suggesting that rationalism, decoupling, and emotional distance aren’t just personality traits but virtues.

The line, “trying to be very rational and very epistemically grounded”, reminds me very much of a dynamic I used to observe on and around investment bank trading floors.

As so often in corporate subcultures, market participants tended to adopt the strange vernacular of their work in everyday conversation: coffee preferences, lunch orders, or holiday destinations were often described in terms of “bid” and “offer”, “long” and “short”. Back then I cringed inwardly hearing this stuff, just as I cringe writing it now, but it provides an instructive window on a worldview. Those in and around the trading floor overtly aspired to being hyper-rationalist, objective, and data-driven.

Freeman paints a picture of an analogous worldview in Silicon Valley, where technologists and AI researchers similarly value rationalism, logic, and computation above all other kinds of knowledge or thinking.

Stupid is as Stupid Does

As we’ve seen, the fundamental, existential question for the quants working in credit derivatives was how to predict the likelihood that hundreds or thousands of borrowers would default on their mortgages. It’s a stupendously difficult human question, involving thousands, perhaps millions of variables interacting in a complex, dynamic system. Solving it meant simplifying and smoothing complex reality into something cleaner and more legible.

For the big AI labs, the fundamental, existential question is equally human and equally difficult: what is intelligence?

It’s an ancient question with no shortage of answers. Humans have been arguing about the precise nature of intelligence for more than two-and-a-half millennia and still don’t agree. Early twentieth century psychologists sought to make intelligence measurable, arriving at “Intelligence Quotient” (IQ), a widely used but highly controversial concept. More recent cognitive scientists tend to see intelligence as an interacting, overlapping collection of capacities.

In short, “intelligence” is a highly subjective, highly contested, fairly amorphous concept. AI inherited this unresolved argument. We don’t really have any meaningful consensus on what intelligence is, but AI labs want to ask whether machines have it.

While AI labs were well-funded research institutes exploring theoretical questions at the convergence of cognitive science and computing, this was an interesting philosophical debate. When ChatGPT almost entirely accidentally became a runaway overnight success, however, the question attained an entirely different level of urgency and importance. As Anthropic, Google, Microsoft, Meta, X and the rest joined the multi-trillion-dollar arms race, suddenly intelligence ceased to be an abstract philosophical or psychological concept and became a product specification.

The messy reality of human intelligence, including memory, perception, abstraction, reasoning, social relationships, tacit knowledge, emotions, embodiment, creativity, morality, consciousness, won’t fit on a graph. So much like the quants did with borrowers, the AI labs flattened, simplified, and smoothed intelligence until it did fit on a graph. Intelligence had to become legible.

IQ tests measure human capabilities on particular tasks, often verbal, numerical, or spatial reasoning. The labs use the same approach for AI, building benchmark tests in different areas of human activity: maths, computer programming, scientific reasoning and the like. Just like schoolchildren, the models sit the tests. Just like schoolchildren, the models are trained for the tests. And just like schoolchildren, if we’re not careful we conflate the results of the tests with much broader claims.

The logic seems sensible enough on the face of it. We want to measure, say, reasoning so we design tests made up of tasks that require reasoning. A particular model scores well on that test. But that nuanced description tends to morph into the different claim that the model has excellent reasoning capabilities. In turn, that claim morphs into the entirely different claim again that the model can reason. Like a childhood game of “Telephone” (or “Chinese Whispers”, depending on where you’re reading this), the claim morphs further into a bolder claim about the “PhD level” intelligence of the model.

It’s a short road from there to predictions about even more amorphous concepts like “Artificial General Intelligence” and “Artificial Super Intelligence”. Altman, Amodei, Musk and the rest are more than happy to predict when AGI will arrive, without ever addressing the question of what exactly it is.

With benchmark performance routinely conflated with intelligence, and given the commercial incentives for AI labs to make bolder and bolder claims about the intelligence of their models to secure new investment, media attention, increased users, and political clout, it’s not hard to see how this plays out.

Just as teachers whose performance is measured in league tables of exam results might feel pressured to teach to the test, it wouldn’t be a huge shock to learn that AI labs were prioritising benchmark performance as they develop new models. Through a series of small, logical steps, the abstract concept of intelligence becomes a measure, the measure becomes a target, the target becomes a product specification, and the product specification becomes a bold prediction about the future of human society.

As a prominent economist once put it, in the “law” that now bears his name:

“When a measure becomes a target, it ceases to be a good measure.”

Charles Goodhart

There’s an even more fundamental issue here, though. That question again: what is intelligence? Or to reframe it slightly, in light of the use of benchmark tests as a proxy for intelligence: which capabilities are worth testing? What type of benchmarks matter?

The Freeman quote from Vanity Fair hints at one answer. In an intellectual environment predominantly composed of mathematicians, scientists, and computer programmers, the norms around intelligence are “hyper-rationalist”, “highly decoupled”, “non-emotive”. Intelligence is as intelligence does: intelligence is as we are, here in Silicon Valley.

A counterfactual thought experiment is particularly instructive. Imagine, for a moment, a world in which AI had emerged from a radically different intellectual environment. Imagine AI labs composed predominantly of anthropologists, historians, nurses, and social workers. Would we use the same benchmarks for intelligence, measuring scientific reasoning, complex maths, and computer coding? Or might we prioritise the ability to read other people’s emotions, successfully navigate unspoken social norms, or make difficult moral judgements?

AI benchmarks might seem like they objectively measure intelligence, but in reality they implicitly encode assumptions about what intelligence is. That encoding influences what the AI labs build, subtly shaping what artificial intelligence is. Once again, the map becomes part of the territory.

Rhyme Time

“History doesn’t repeat itself but it often rhymes”

Mark Twain (allegedly)

Hyper-intelligent maths geeks with an outsized faith in the complex technology they’d built; a classic collective case of Engineer’s Disease. An internally logical, mathematically elegant model of the world, based on half-hidden, half-forgotten assumptions made with the best of intentions. A flattened, simplified, atomised model of reality, built for legibility at the expense of completeness. A model far too complex to be understood by the uninitiated, too dazzlingly sophisticated to be questioned by outsiders. A model whose apparent success on its own terms encouraged ever greater confidence in the worldview behind it until it seemed inevitable and infallible.

Does that paragraph describe the arcane world of quantitative finance in the 2000s, or the AI boom of the 2020s?

When intelligent people believe their inherent genius has enabled them to understand and shape the world in hitherto impossible ways, we ought to be wary. When a narrow mathematical, logical, computational worldview crowds out all other perspectives, we ought to be wary. When vast amounts of capital pour into one narrow sector of the economy, amid increasingly wild hype and ever more shrill proclamations that “this time is different”, we ought to be wary.

Above all, when the smartest people in the room become convinced their models have made messy, unpredictable reality legible and controllable, we ought to be very wary.

Eighteen years ago, on a Monday morning in September, reality bit back. Who can say when or how it might do so again?