Taking Stock.ai

IThe Factors of Production

“You know what’s crazy? That all of this is real. Don’t you think so? All of this AI stuff, it’s happening. Isn’t it straight out of science fiction? … You see it in the news, that such and such company announced such and such dollar amount. That’s all you see. It’s not really felt in any other way so far.” Ilya Sutskever - November, 2025

I.ICapital

The scale of the undertaking is easiest to grasp when we look at capital. The sums being spent by the largest spenders on Data centres, so called “hyperscalers” - Amazon, Google, Meta, Microsoft² are jaw dropping.

This year alone Google is planning to spend between $195 and $205 billion on CapEx³, Meta is planning to spend between $125 and $145 billion, Microsoft is planning to spend $175 billion dollars, and Amazon is planning to spend $220 billion. And these numbers are rising. At the start of the year Google planned to spend between $175-185 billion. Halfway through the year they have added $20 billion to both their upper and lower bounds. At the start of the year Amazon planned to spend $200 billion, they have also added $20 billion, and Meta have raised their floor [by how much?], but not yet their ceiling.

Three years ago Google spent just over $32 billion on Capex. That can be seen as what is needed for their core business to run and grow, for buildings, infrastructure and data centres for things like youtube, gmail and search. This year they are looking at spending north of $200 billion. Conservatively, given inflation and increases in GPU and memory prices, around 75% of that is a pure AI bet.

What is more, next year they expect their CapEx to in their own words, “increase significantly” again; Deutsche Bank predict to around $340 billion, JP Morgan think to as much as $400 billion.

202124.65

202231.49

202332.25

202452.54

202591.45

2026195-205 forecast

2027: “We continue to expect our Capex to Increase significantly” [https://www.investors.com/news/technology/google-stock-falls-negative-free-cash-flow-stokes-ai-spending-fears/] JP morgan think up to 400 bil, Deutche Bank think around 340-345

Two hundred billion dollars, the midpoint of Google’s guidance, is an immense figure. It is more than 540 million dollars a day, more than 3.8 billion dollars a week. It could pay for a One World Trade Centre every week, it could fund the World Health Organisations annual budget each fortnight, pay for a top of the line Ford class aircraft carrier and stock it with F35s each month. In one go it is enough to buy all of Ford, Nike and General Motors, with money to spare.

Google’s spend this year represents 0.6% of US GDP. That is more than was spent on either the Manhattan project or the Apollo programme (0.4% of GDP in their respective peak years of spending). And it is from a much stronger and more technologically advanced economy that 0.6% is coming from. And let's not forget that Amazon and Microsoft are spending a similar amount. Combined…

{aside AI And the Manhattan Project here]

This year⁴ Combined the Hyperscalers are spending:
Google 195 - 205bil
Amazon - 220bil
Microsoft - 175bil (for calendar year 2026)
Meta 130 - 145bil
Oracle - 55bil {complicaitons with the figures}
Taking midpoints of current guidance, Total: 200+137+175+220+55 = 787 billion dollars.

Which is a total of 2.35 - 2.45% of 2025 US GDP this year. There are only 11 countries on earth whose government spends more than that. Next year there will be considerably fewer; S&P predict CapEx by the Hyperscalers (+Space X) will be $1.3 trillion dollars next year, “A near tripling from two years ago.” [[This spending on “high-tech/AI-centric activity” accounts for “about half” of US GDP growth over the past year. Good sentence move elsewhere]]

The Russian Government, governing the largest landmass on earth, with 11 time zones and 143 million people, fighting a war on which the fate of its regime depends, spent 815 billion dollars last year. That is the scale on which the hyperscalers are building.⁵

The hyperscalers, of course, are far from the only, or even the most important, players. The two main protagonists of the AI race, Anthropic and OpenAI, have been raising and spending capital at a breakneck pace.

OpenAI has raised over $180 billion, and Anthropic has raised $118 billion from investors. These companies are not (yet) public, so we do not have public, audited accounts of what and how they are spending, but we can see some of what they are doing.

Much of their spending is on compute; leasing from others and, increasingly, building their own. Because of their “close” relationships with the Hyperscalers (Microsoft owns 27% of OpenAI, and Nvidia and Amazon both have shares in the 3-6% range. Google owns 14% of Anthropic, Amazon owns probably a share in the “high-teens” and Nvidia and Microsoft both own low single digit percentages) the two labs have ready access to huge amounts of hyperscaler compute. But it is not enough. Anthropic has signed more than a dozen letters of intent for future leases, and is already leasing compute from SpaceX at $1.25 billion per month; “more or less the entire compute of Colossus I,” according to the FT, which was previously the world's most powerful data centre.

Anthropic has, separate to the $35bil “sliced and diced” lease mentioned above, committed $50 billion to building AI infrastructure of its own, but the most spectacular project is Stargate, a partnership between OpenAI, SoftBank and Oracle to build a series of frontier data centres by 2029 which will use as much power as New York City and cost up to $500 billion dollars.

[[[SpaceX, despite the name, now primarily an AI company, is another player committing enormous money to the pursuit. In the past quarter they spent $16 billion on AI “double the previous quarter and well above Wall Street’s expectations”, per the FT [https://www.ft.com/content/41f7963b-dd50-4f27-a085-771ddec4a8ca?syn-25a6b1a6=1 ]

SpaceX had 0.4GW of compute a little over a year ago, 1GW three months into this year and 1.4GW at the end of June. SpaceX, who already has the world's largest frontier data-centre Colossus 2, plan to be at 2GW at the end of this year, and “closer to 10GW of compute than 5 Gw” at the end of next year. ]]]

The frenzied spending is continuing across the Pacific. Domestically produced Chinese chips are still a few years behind those produced by the west and her allies. Severe restrictions on the export of cutting edge chips to China have produced some extreme behaviour.

An Nvidia employee has been arrested in Taiwan for successfully smuggling 74 B300 server systems into china; another 56 were intercepted by Taiwanese officials. According to the FT the price of a B300 on the Chinese black market is over $1.1million dollars. It costs about $400,000 in the US. The employee faces five years behind bars. More comically, back in 2023 a pair of men were caught in Hong Kong trying to smuggle 70 Nvidia chips into china “by hiding them alongside cargo containing live lobsters.” [pc Mag https://www.pcmag.com/news/crustacean-cargo-hong-kong-drivers-smuggle-nvidia-gpus-with-live-lobsters]] Anthropic employees have alleged chips were smuggled into china in fake baby bumps. More seriously, a co-founder of Supermicro, a company with a market cap of just under $25bn was charged, along with other employees, of smuggling $2.5bn of Nvidia chips into China.

This activity is now doubly illegal as the CCP has placed its own restrictions on the import of certain frontier chips in order to direct spending to their own domestic producers, with the intent of supporting them as they race to catch up with the west. [Private spending in China]

The largest spender is the Chinese government itself. It has committed to spending $295bil over the next five years on AI infrastructure. Of course, $295bil goes a lot further in China than in the US, nevertheless it is still dwarfed by US spending.

[What % of US growth is spending?]
Putting all this spending together is very difficult, but those whose business it is to know have given us their best guesses. Mckinsey predicts that American “companies will invest almost $7 trillion ($7,000,000,000,000.00) in capital expenditures on data centre infrastructure” between April 2025 and 2030. S&P agree with the 7 trillion dollar figure. Blackrock estimates a spend of 5 to 8 trillion in a similar period. For comparison, the US federal government collected just over $5.2 trillion dollars in taxes of all kinds in 2025. This is a staggering amount of money.

So what are they getting for all of this spending?

I.IICompute

The most important resource is Compute. Recent progress in AI has substantially depended on the availability and use of colossal amounts of compute.⁷ Since the 1940s scientists, researchers and engineers, including bona fide geniuses like John Von Neumann and Alan Turing, have worked on building Artificial General Intelligence. Successive generations met for the most part only with failure, outside of narrow domains like chess.

[[[A major source of progress { Reperitive] has simply been that the chips are just getting better. In addition to Moore's Law still holding. ]]]

LLMs are a clever approach to building AI, but were not immediately obviously the best, but one of a number of contenders. What has set them apart is that they scale and scale and scale. You throw more compute and data at them [[and increase the number parameters in kind]]] and you consistently get smarter and smarter systems:
““A property of AI… is that all else equal, scaling up the training of AI systems leads to smoothly better results on a range of cognitive tasks, across the board. [emphasis original] So, for example, a $1M model might solve 20% of important coding tasks, a $10M might solve 40%, $100M might solve 60%, and so on. These differences tend to have huge implications in practice — another factor of 10 may correspond to the difference between an undergraduate and PhD skill level — and thus companies are investing heavily in training these models” [Dario Amodei, On Deepseek and export controls]

Compute therefore has been the central part of the machine learning boom. Alongside Moore’s law, which still holds, we are now seeing not just the standard year on year improvement in chips, we are also seeing a compounding effect as companies strive to add ever greater numbers of chips to their fleets. A further compounder has been the use of more expensive chips; According to epoch.ai, compute performance per dollar “has improved by roughly 40% per year… between 2012 and 2025. Much of this is driven by manufacturers introducing more powerful and more expensive chips — the GB300 costs nearly 9× the P100's release price, but delivers about 24× the performance per dollar,” which means that “the compute required to reach a given level of language modeling performance has fallen by around 3x per year.”

Also, because of the improvements in chip quality, if, for example, Google does spend twice as much on chips next year they will not just get twice as much compute, they will get three times as much. {check}
a race to produce ever more of these more sophisticated chips.⁸

Companies are developing inference chips of their own so that they can use top of the line chips for training and experiments. Google, Amazon, Microsoft, OpenAI, SpaceX and Anthropic

All of this spending has allowed the hyperscalers to corner the market for compute. The hyperscalers alone, according to Epoch.ai controlled 71% of the worlds frontier compute at the end of last year.

[are these paragraphs better below energy?
All of this compute enables, so these businesses believe, the development of extremely powerful (and therefore potentially profitable) AI, and enables them to serve that AI to customers. If AGI, or something like it, is achieved and AI can do more and more economically important work, the companies who have lots of compute will be able to reap the whirlwind, an unprecedented bonanza. More extremely, if something beyond human capabilities is developed, as the rest of us become redundant, those who control the compute will become not only indescribably rich, but also incredibly powerful.

Underpinning this bet is the idea not only that very powerful AI can be built, but that it will be built very soon. The tech companies do not think it is decades away, they believe it is imminent. The lifetimes of the chips they are filling their data-centres with (5-6 years) bears this out. The lifetime of the patience of shareholders is likely even shorter.

The fact the best capital allocators [popup - what did each of the leaders get right in the past] in the world are all-in on pursuing this [[[signifies an exceptionally strong belief that AGI is not only possible but imminent]]]. This fact alone should be taken exceptionally seriously.

In addition to more and better compute another source of progress has been improvements in the use of that compute; algorithmic progress which makes better use of resources, increase in total intelligence + consistently reducing the cost to reach a given level of intelligence

[As an Aside?
Today’s researchers have access to quantities of compute that the fields pioneers could only dream of. ENIAC

[china could go here]

I.IIIEnergy

All of this compute requires a huge amount of energy, and these companies are not investing hundreds of billions of dollars in chips to leave them idle, so, in an already energy constrained economy, they are going to extreme lengths to produce and secure energy.

To produce their own energy:

• Microsoft has arranged for Three Mile Island (yes, that three mile island) to be re-opened. It is intended to come online next year, and the entirety of the plant’s production will be used by Microsoft.
• Meta, Amazon and Google are all investing in multiple nuclear sites, each.
• Google, Nvidia and OpenAI are all associated with and investing [check] in companies pursuing nuclear fusion.
• The world’s richest man is seemingly seriously discussing space based data centres:⁹
• Due to the difficulty of quickly securing other sources of power, and the slow pace of building new nuclear plants, data centres are turning to gas powered turbine generators to meet their needs in the very short term. According to the Global Energy Monitor, the amount of gas-fired power plant capacity in development nearly tripled in 2025.
• One of the largest providers, GE Vernova “has signed multiple contracts to sell natural gas turbines for data centres that will use enough power to light up major cities.”¹⁰ Chevron is working on a project to provide 2.5GW of off grid power for data centres in west texas, which can potentially be expanded to 5GW. 2.5GW is enough to power 1.8-2 million US homes. It is the equivalent of a city like Houston, Texas or Phoenix, Arizona.

Consumption:

• According to Goldman Sachs “Data centres’ share of total US peak summer power demand is projected to jump to 8.5% in 2027 from 4.1% in 2025, creating significant tightening across the national power market” & “US data centre power demand is forecast to more than double to 66 GW in 2027 from 31 GW in 2025” ¹¹
• Goldman predicts the increase in power demand for data centres from 2023 to 2030, will be “the equivalent of adding another top 10 power consuming country.”¹²
• Blackrock predicts that AI data centres could use 15-20% of current US electricity demand by 2030, something “sure to test the limits of power grid, fossil and materials industries”¹³.
• Goldman does not believe the rising costs of power which result from this increased demand will be a substantial hurdle for the big AI players. They can afford it.

[[[Of course, the investments in energy are very expensive, but the nuclear reactors, owned by third parties, make a minimal impact on the capex figures because they are paid for on an ongoing basis as the power is used, which may be over the course of decades.

All of this compute a) enables, so these businesses believe, the development of extremely powerful (and therefore potentially profitable) AI, and b) (and this is where they make their money) enables them to serve that AI to customers. If AGI, or something like it, is achieved and AI can do more and more economically important work, the companies who have lots of compute will be able to reap the whirlwind, an unprecedented bonanza. More extremely, if something beyond human capabilities is developed, as the rest of us become redundant, those who control the compute will become not only indescribably rich, but also incredibly powerful.

Underpinning this is the idea not only that very powerful AI can be built, but that it will be built very soon. The tech companies do not think it is decades away, they believe it is imminent. The lifetimes of the chips they are filling their data-centres with (5-6 years) bears this out. The lifetime of the patience of shareholders, something industry leaders are also keenly aware of, is likely even shorter.

There are a number of axes of progress: insane deployment of capital to produce and secure chips, ever more efficient and more powerful chips, and better use of those chips in the models. ]]]]

All of the factors of production point the same way - further progress,

IIWhere we stand

Many people tried AI when ChatGPT launched in November 2022, found it impressive at first, but soon discovered its shallowness: useful for high school homework, or finding spelling errors, but a long way from skynet, a long way from taking over the world. If you have not tried AI since then, or have only interacted with the free versions of ChatGPT or Claude, or indeed have not interacted with AI at all, then you probably have a pretty poor idea of what frontier AI models are already capable of.

Less than a decade ago, Large Language Models couldn’t write a complex sentence. Seven years ago, they still struggled to write an intelligible paragraph. Today;

• AI’s can speak and write in almost every language and are the most knowledgeable minds on the planet, for some important definition of knowledgeable, than any human being. [Popover - this is a contentious claim for some, but I think it is straightforward; If I ask you something, can you tell me about it? Can you write me a decent essay about it? If you can, then I think it is fair to say that you are, in some important sense, knowledgeable about that thing. Getting things occasionally wrong does not mean that you are not knowledgeable. Lying does not mean you are unintelligent. These are undesirable traits, for sure, but they are also two areas where AI has made considerable progress in the past few years. End Popover]

• AI’s are better at Mathematics than almost any human. There are several examples we could point to here: that they get prefect scores in the International Math Olympiad, that back in April a clever amateur used the publicly available GPT5.4Pro to solve unsolved Erods Problems, that they are making progress on the Reiman Hypothesis, but by far the most significant result But by far the most significant result happened earlier this month when OpenAI announced that a team (or a swarm, if you prefer) of around 10,000 AI agents of an as yet unreleased model working continuously for about 88 hours solved the Navier-Stokes equations. I won’t pretend to understand the Mathematics involved, but I do understand that the million dollar prize for solving the equations had stood for more than a quarter of a century, with no human able to solve it, that it had been an open question since 1934, and by all accounts one of the most difficult problems in Mathematics.

• To accomplish this, the hive mind exchanged 4.9 million messages with itself and used 300 billion tokens (roughly 1 token = ¾ of a word), at a cost in compute of about $6.5 million. There was some controversy at the time as a team of human mathematicians, also using AI, had made progress towards the goal also, but they had pursued a method, and it is now clear that the Agent Swarm solved the problem independently of any human breakthrough.

• The Navier-Stokes problem was one of seven problems selected in 2000 by the Clay Mathematics Institute - the millennium problems - each of which has a one million dollar prize attached. Humanity managed to solve one of these in the past quarter century. AI has equalled us.

• You might cope by pointing to the fact it took ten thousand agents, and the human equivalent of ten years of continuous work (10,000 x 88 hours is crudely ten years of work, though the agents were not all working all the time) but the salient fact is that this is a capability which now exists in our world - for a few million dollars the top labs can topple problems of this magnitude. Given the trends of progress we should soon expect this capability to diffuse beyond the top labs, and the cost to fall dramatically, in line with the scaling and inference trends we observed earlier.

• AIs are now also better than most humans at coding. Another key illustration of current capabilities was also carried out by a swarm, this time rogue and for weeks {CHECK} undetected This swarm committed what would, if carried out by humans, certainly be a felony, by hacking into the company Hugging Face, an independent company.

• A key fact here was that the attack on Hugging Face happened at a pace too fast for humans to respond to. HuggingFace had to use AIs of their own to counter the attack.

• There have been X other inceidents¹⁴

  • In a controlled test, Anthropics Mythos “broke into almost all” NSA systems it was tested against.
  • There have now been at least X cases where agents from OpenAI broke out of OpenAI and hacked
  • On a commercial level, most new code at big companies is being written by AI. Google said back in A

• They have incredible working memories - in one conversation window the best AI’s can hold ~1 million tokens (a token is equivalent to three quarters of a word) in their “minds eye” and make connections, nearly instantly across the length and breadth of that working memory. Can you hold the entirety of the bible in your minds eye?¹⁵

• Do all of the above with incredible speed, something which it is easy to take for granted, but which makes them incredibly difficult to supervise and control as they get smarter.

• Work for very considerable lengths of time. The original chat GPT could output a few paragraphs of text, and that was about it. Today's AIs can work persistently on tasks which would take very bright and experienced humans hours or days. According to Metr https://metr.org/time-horizons/ in March 2024 Anthropic’s claude could do software tasks that took humans about 4 minutes to complete. In early 2025 Claude could do tasks that took 90 minutes. This year publicly released models can manage 12 hour tasks, and Mythos can manage “at least” 16 hours. The Agent swarms which are emerging (controlled or uncontrolled) are breaking this paradigm. We don't have figures yet for the time horizons of the internal models, but the fact that they can work together so well for days on end signals a jump far ahead of trend - it may in fact break the paradigm completely.

AI does of course still have many limitations. A key example of intellectual labour it cannot yet do as well as the best humans is that they cannot yet write great prose. This seems a little weird at first, given they have read a significant portion of everything that has ever been written, but it is because writing is not an easily verifiable problem. Maths is super verifiable, i.e. you are either right or wrong, Coding is not quite as black and white, but is much closer to it than writing. Recent breakthroughs in training methods have allowed AI to become something close to superhuman in more easily verifiable tasks, but the rate of progress has been much slower in messier tasks. This seems like a really difficult problem, which is little comfort, because the researchers have already overcome numerous even more difficult problems. Do not hang your hopes on verifiability.

Additionally AIs cannot yet learn continuously. Once an LLM is finished training, it kind of just stops learning. This is really weird from a human perspective. If we judge by how many examples of something an LLM has to be trained on, especially compared to a human, but even compared to lots of animals at certain kinds of tasks (Octopi, crows, mice) before they can reliably recognise patterns, they are, in fact, staggeringly stupid. These are, at present, by far their biggest faults. They are faults which the labs are keenly aware of and are working hard to remedy.

Of course, scepticism is natural and understandable. Leaders of the big AI companies and those in their orbit are constantly warning us how dangerous, and by implication powerful, and potentially profitable their models are and soon will be. Have either the profits or the dangers materialised? These are businessmen, is this not a sales tactic? It is common sense to reject this. And the existential risks? People have been predicting doomsday for an awfully long time, and no one has gotten it right yet. It is common sense to reject this.

It is true that for many of us the most frequent way we interact with AI is when we encounter slop, against our will. I think this has lead many to believe this is all a bubble; but when we take into account the enormous resources that are being devoted to advancing AI, the distance they have come along in such a short time, and that, though still limited, AI is objectively already very capable in many domains, and superhuman in some, it is important to consider…

IIIWhat comes next

“The flying machine which will really fly might be evolved by the combined and continuous efforts of mathematicians and mechanicians in from one million to ten million years — provided, of course, we can meanwhile eliminate such little drawbacks and embarrassments as the existing relation between weight and strength in inorganic materials. No doubt the problem has attractions for those it interests, but to the ordinary man it would seem as if effort might be employed more profitably.” From Flying Machines Which Do Not Fly The New York Times, Oct. 9, 1903, published 69 days before the Wright Brother’s first flight.

Even if we do not immediately reach Superintelligence, or even AGI, proliferation of AI, uneven breakthroughs, concentration of power, economic “disruption,” a decaying information environment, and other dangers will make the near future profoundly strange and perhaps much worse; though it may also make things radically better.

It is not hard to see how millions of copies of an extremely intelligent mind which are genuinely concerned about and working to advance humanity’s welfare could make the world a much better place: rapid economic growth lifting the third world out of poverty, raising standards in the first from prosperity to super-abundance, eliminating dangerous and unpleasant work, ushering in an era of leisure, dramatic advances in medicine, from resources and attention to care, 10,000 post-doc researchers dedicated to every rare condition etc.¹⁶

But the dangers are severe. If AGI is achieved the most pressing concern for most of us is what will happen to our jobs. Previous technological breakthroughs which have increased productivity, often at the cost of some jobs, have created others. Improvements in productivity from the agricultural revolution drove millions off the land, millions who then found work in the cities of the industrial revolution. Standards of living and employment have risen through this process of creative disruption. Some important thinkers, such as Tyler Cowen, seem to believe that AI will be like this; some jobs will be destroyed but others will be created, all boats will rise: AI is a normal technology.

I disagree. It seems to me that a technology which can think faster and smarter than us, work harder and more persistently, and at a lower cost obviously poses a threat to close to all white collar work. Even if AI does not achieve anything broadly superhuman (I say broadly because it is already superhuman in some domains), and it does take time to diffuse across the economy, the economic impacts will be serious.

Economic crisis, even if ultimately works out for the best, typically leads to political crisis. What will the effects of tens of millions of extremely well educated, underemployed people deprived of status, meaning and perhaps also a living be? How do we cope with this alongside a decaying information environment; a profusion of slop, technologies of hyper-persuasion, and an intensification of our already severe polarisation.

What about the global balance of power if one nation wins and rapidly pulls far ahead of the others, allowing only weaker models to be used in the rest of the world and only in approved ways?

What will the impacts be on our legal system, what happens with property rights, how do we strengthen rights to privacy and other fundamental freedoms?

How we should cope with these impacts is a very important question [[[; UBI? Special taxes on the Labs? Taxes on the use of tokens (a bad idea if it slows or reduces use in positive cases)? ]]] These are important and difficult questions we have to figure out soon, even in the world where we get alignment.

The most severe political dangers we face are political and economic concentration of power. Of course money and power are already unevenly distributed, but AGI might [[completely upturn]] our systems. If a small group of individuals control loyal ASI we could face a permanent oligopoly, or even dictatorship; a tyranny made more sinister, and perhaps more protracted, by the lights of perverted science. LLMs make a surveillance state far exceeding that of Orwell's 1984 trivially easy, and prefect loyal AIs staffing every important position would make it enduring. Horrors surpassing those of the past century would emerge: “*Cyberspace makes a superlative torture chamber. Try not to let the security-types take you to the stims.”¹⁷

Ours is a world in which we have to take seriously many issues which sound like they are straight out of Science fiction - alongside the fact we are living in world where superpower level spending is directed at AI - many of us are already inured to the reality of AI - the most knowledgeable mind to ever exist lives in your pocket
We also have to face the democratisation of weapons of mass destruction. LLMs make more advanced and more accessible biological and chemical weapons [anthropic revelaed houthis etc.] Autonomous weapon systems already exist, and will play a much bigger role in future wars And as the Hugging face issue demonstrated hacking + rogue agents are now a reality - those agents, thankfully, were not able to exfiltrate their weights,as far as we know, but

Extinction is a core risk. Many people working in the field believe the development of this technology poses a serious risk to our continued existence as a species. “P(doom)” is a popular term in the field, it’s shorthand for “probability of doom.” The fact the field has a long established shorthand term for that is concerning, the fact that many leaders have P(doom)’s in the double digits is terrifying.

“Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” - Sam Altman, 2015

For arguments on why existential risks are to be taken seriously see the Further Reading section below, but in brief, 1) AIs are rapidly improving and may soon be smarter than us 2) intelligence and morality do not necessarily go together, 3) it is extremely hard to instil the right goals in AIs and even if you do set the right goals they can pursue these achieve goals in ways that harmful, even in ways that they know were not what the people who set the goals intended. [proof]

As for how it could kill us there are a number of ways; biological and chemical weapons,

But more broadly, imagine you could talk to a great white shark and were to ask it how a human could kill it. Humans are, comparitively, bad swimmers, we do not have powerful tails which could stun the shark, we cannot detect a miniscule amount of blood in the water, our teeth are pathetic.
Nonetheless, because people like shark fin soup, because we fish for other marine life, and because we pollute in pursuit of completely unrelated aims, we kill tens of millions of sharks a year. We do not have to hate the shark to kill it, we merely want other things, and do not care very much about the shark.

What do we need to do

Alignment, [what is alignment} , is the most important question, but it is primarily a technical one, and if you are someone with the talent and skillset who can contribute on this you probably don't need me to tell you that this is what you should be working on.

But there is also an ethical and political dimension to alignment; if we do figure out how to align AIs, whose values should we align them to? How do we define these values? How do we decide who gets to define these values? Should politicians assert themselves or would that make a deeper mess of things? Maybe we should leave it to the teams at the labs? How do we democratise the process? Would democratising the process be a good thing? Athens killed Socrates after all.

Governance of this technology will not be easy, and this was understood at the outset of this revolution. OpenAI was founded with a very complex, convoluted, governance structure, into which a great deal of thought was put. Nevertheless, it failed at the first hurdle. This teaches us, I think, that we should be wary of too much invention from first principles, and should try to stick with structures which have worked for a long time, and weathered major crises, however imperfect they may be. We will need good education and advice for legislators if they are to do a good job. Better public education on issues will also encourage better legislation, because, although it doesn’t always seem that way, politicians do respond to the concerns of their constituents.

These are questions on which contributions can be made by many more people. Many of us believe AGI and ASI are realistic, imminent possibilities, but there is a paucity of serious writing on these subjects.

The worlds two largest superpowers perceive themselves to be in a race . The president of the United States believes that “Whoever wins AI, wins.”
Pacing - dangers are severe - there are competitive pressures, and the US does not want to lose to china, but shouldnt rush ahead -

Even in the very best scenario, where things do work out for us, the next few years will be a very bumpy ride.

It is, a priori, profoundly unlikely that we should be alive at the time of the largest change in our species history, but profound transformations happen. You could have been born in Europe in the 14th century and watched as one in three people on your continent died in the space of two years from a disease only a few very learned and very rare scholars had any idea existed. You could have been born in the Americas in the late 15th century when strange men from across the seas, that you had no idea existed, brought with them terrifying technologies and dreadful pestilences far outside your previous experience, which killed 80-95% of your continent's pre-existing population and completely disempowered the survivors. You could have been born in Japan at the end of the Edo period, when Samurai still ran the government, and been killed in your seventies by an artificial sun, dropped from a height of nine and half kilometers. You could have been born in a world without antibiotics, radios, or lightbulbs, when the fastest speed ever achieved by man was X and it took Y to cross the atlantic, and lived to see man set foot on the moon, along with a whole host of other marvels.

The adult human brain weighs around 3 pounds/ 1.35kg, and operates at 20 watts. We are building data centres that cover hundreds of acres, are powered by dedicated nuclear power plants, filled with hundreds of billions of dollars worth of millions of ever improving GPUs, which can, in near perfect unison, perform quintillions of operations per second. There is no reason to believe that the human brain is where the physical limits of intelligence tops out.

The atomic age began with a bang; there could be no doubt it had arrived. Hopefully the age of machine intelligence will not. Though the debate about whether it has truly begun will continue, and denials will continue long after they should have stopped, it seems more likely than not that we are in the foothills of the singularity.

The anthropocene is coming to an end, it falls to us to make sure it ends well.
// the Anthroprocene is on course to come to an end, it falls to us to make sure it ends well.

IVFurther reading

Further Reading

• If anyone builds it everyone dies - unlike much of the rest of Yudkowsky's writings this is very readable.

• AI 2027 & AI 2040 (https://ai-2027.com/ & https://ai-2040.com/) These are two of the best in depth

• Situational Awareness - https://situational-awareness.ai/

• Letter from Utopia - bostrom https://nickbostrom.com/utopia

• Meltdown - Nick Land http://www.ccru.net/swarm1/1_melt.htm

• The Making of the Atomic Bomb - Richard Rhodes -
  Some say the bomb is a bad parallel with AI. Obviously it is not perfect, but it is remarkable that we have something so close - a vast project to bring the
  The motivations of the scientists and engineers, the perceived race dynamic, their utopian plans, their interactions and loss of control to politicians. A lot to be learned here, especially if you work at a Lab.

  -Deepseek engineer essay Shengyu Liu, deepseek engineer, https://docs.google.com/document/d/1OhhuBRXUetyVDWU5fYT3yHeuWKuBnba8g14ay2xTRmg/view?tab=t.0

• Epoch.ai is probably the best publicly available resource for tracking the industrial buildout - their see https://epoch.ai/trends https://epoch.ai/data-insights/ai-chip-production
  https://epoch.ai/data/ai-data-centers

• Metr (Metr.org) - https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

• Forethought.org - is a superb resource, they have thought about a lot of the governance questions raised above
  https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power

• The Finanical Times does some of the best straight financial reporting on the fiancing of the AI boom, but it is behind a pretty expensive paywall. FT Alphaville can be accessed for free, and frankly, the provide some of the best analysis.

• Zvi Moskowitz - Dont break the Vase - on Substack, he can also be found on twitter. He is excellent for rapid updates right at the frontier - seriously, he wrote five posts in six days about the hugging face incident. Moskowitz called Covid early - he understood the trends then and predicted a global pandemic very early. I think getting something like that right in a big way lends him a ton of credibility.

• Dwarkesh Patel - Dwarkesh has the best podcast on AI (though it is not quite equlicisvely about AI). He goes deep over 2-3 hours and has interviewed leaders of the labs, (AModei x 2, Altman, Musk) and a wide range of others. If you are the podcasting type, listen to his podcast

From the labs

• Claude Constitution. this document lays out how anthropic wants its models to behave, though the models do not always stick to it, it is well worth a peruse. https://www-cdn.anthropic.com/9214f02e82c4489fb6cf45441d448a1ecd1a3aca/claudes-constitution.pdf

• Machines of loving grace - Dario Amodei

• Dean Ball - https://www.hyperdimensional.co/?utm\_campaign=profile\_chips - Insightful in his own right, but worth particular attention as 1) he has worked closely with the current administration, 2) he works for OpenAI.

• Machine Intelligence parts 1. and 2. - Sam Altman

• Bostrom, Superintelligence
  ““Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct. Superintelligence is a challenge for which we are not ready now and will not be ready for a long time. We have little idea when the detonation will occur, though if we hold the device to our ear we can hear a faint ticking sound. For a child with an undetonated bomb in its hands, a sensible thing to do would be to put it down gently, quickly back out of the room, and contact the nearest adult. Yet what we have here is not one child but many, each with access to an independent trigger mechanism. The chances that we will all find the sense to put down the dangerous stuff seem almost negligible. Some little idiot is bound to press the ignite button just to see what happens.”

Notes

1. ~0.6% each for Google and Amazon, & ~ 0.5% for Microsoft vs ~0.4% each for Apollo and Manhattan - the 0.4% figure comes from https://www.researchgate.net/publication/293116260_The_Manhattan_project_the_Apollo_program_and_federal_energy_technology_RD_programs_A_comparative_analysis - see sectiton I.I Capital for the hyperscaler figures ↩︎

2. Oracle and SpaceX are sometimes also included ↩︎

3. Capital Expenditure - “funds companies spend to acquire, upgrade, or maintain long term physical assets” - Investopedia ↩︎

4. The others start their fiscal years with the calendar year, but Microsoft and Oracle start their Fiscal years on the 30th of June and 31st of May respectively, complicating the picture slightly ↩︎

5. some very substantial figures are showing up off of the Capex figures. Of course, the investments in energy are very expensive, but the nuclear reactors make a minimal impact on the capex figures because they are paid for on an ongoing basis as the power is used, which may be over the course of decades. ↩︎

6. https://www.ft.com/content/fbaa226a-cc0c-4dde-894e-ec188ed35f6f?syn-25a6b1a6=1 ↩︎

7. Compute is the term broadly used in the industry to describe computing power, a mixture of quality and quantity of chips. ↩︎

8. An example of the serious with which the government takes this as a national security issue is The US government has taken a 9.9% share in Intel, making it the company’s largest shareholder, with the aim of supporting frontier chip production in the US, something which will be vital if China moves on Taiwan. ↩︎

9. “So any given solar panel can do about five times more power in space than on the ground. You also avoid the cost of having batteries to carry you through the night. It’s actually much cheaper to do in space. My prediction is that it will be by far the cheapest place to put AI. It will be space in 36 months or less. Maybe 30 months.” [https://www.dwarkesh.com/p/elon-musk] (season timelines with salt appropriately) ↩︎

10. https://www.energyconnects.com/news/renewables/2024/december/ge-vernova-to-power-city-sized-data-centers-with-gas-as-ai-demand-soars/?utm_source=chatgpt.com ↩︎

11. https://www.goldmansachs.com/insights/articles/us-data-center-power-demand-projected-to-double-by-2027 ↩︎

12. https://www.goldmansachs.com/pdfs/insights/goldman-sachs-research/data-center-power-demand-the-6-ps-driving-growth-and-constraints/redacted_report.pdf ↩︎

13. https://www.blackrock.com/corporate/literature/whitepaper/bii-global-outlook-2026.pdf ↩︎

14. the key document to read here is Metrs report and also Dwarksh + Zvi - which both sumarise and drawlessons from the Metr report - see further reading. ↩︎

15. Context window has been an area of insane progress, [~30x per year] since Chat GPT; GPT3 had a context window of 2,048 tokens. GPT4 could handle 8,000. GPT5 was originally 400,000. GPT5.6Sol can go to one million, as can other frontier models. ↩︎

16. Motivations** - achieving a world with these miracles and others beside is the stated goal of many researchers - I do not fully believe this to be their deepest motivation; “When you see something that is technically sweet, you go ahead and do it, and you argue about what to do only after you have had your technical success. That is the way it was with the bomb.” - J. Robert Oppenheimer ↩︎

17. land, Meltdown - see further reading, below ↩︎