The Singularity Looms:  We Can't See Past the Event Horizon, and That's What Should Scare You

In a 1993 paper for a NASA symposium, mathematician and science fiction writer Vernor Vinge wrote that writers imagining the future of technology had begun to feel "an opaque wall across the future."  Their most careful forecasts, he wrote, now ran into "the unknowable."  As we got closer, he predicted, the idea "will loom vaster and vaster over human affairs."  Thirty-three years later, the companies building the most powerful artificial intelligence are describing that wall themselves.

Anthropic, maker of the Claude models, wrote in September that AI able to build its own successor "could come sooner than most institutions are prepared for."  Its chief executive, Dario Amodei, warned that AI's growing ability to build the next AI "could outrun our ability to understand and control these systems."  No company says it has reached that point.  But the loop that could get there has started.  Anthropic says Claude wrote over 90% of its code in September, according to its Oct. 26 filing with the U.S. Securities and Exchange Commission, and Google says AI writes 75% of its new code.

That loop is what people mean by the technological singularity:  a moment when machines improve themselves faster than people can follow, so that what comes after cannot be predicted.  Here is why the view past that point is so hard to see, why that worries some of the field's best-known scientists, what the labs say they are doing about it and why some experts think the fear is overblown.

The wall no one can see past

The idea is older than the personal computer.  Mathematician Stanislaw Ulam, in a tribute to John von Neumann, recalled a conversation in which von Neumann, a founder of modern computing, described ever-faster technological progress approaching "some essential singularity in the history of the race," a point beyond which human affairs "could not continue" as we knew them.  Science fiction writer and San Diego State University mathematician Vernor Vinge quoted that passage in a 1993 paper for a NASA symposium.

In a 1965 paper, British statistician I.J. Good supplied the mechanism.  If a machine could out-think any person, Good reasoned, it could also design machines better than people can, which would set off what he called an "intelligence explosion."  The first such machine, he wrote, would be "the last invention that man need ever make," provided it was "docile enough to tell us how to keep it under control."

Vinge gave the idea its modern name.  His 1993 paper opened with a blunt forecast:  "Within thirty years, we will have the technological means to create superhuman intelligence."  He said he would be surprised if it happened before 2005 or after 2030.  He also described what it would feel like from the human side:  "a throwing away of all the previous rules, perhaps in the blink of an eye, an exponential runaway beyond any hope of control."

The name is borrowed from physics.  According to NASA, a black hole is surrounded by a boundary called the event horizon.  Inside it, the speed needed to escape the black hole's gravity is greater than the speed of light, and nothing can travel faster than light.  So no light, and therefore no information, can get back out.  At the very center, Einstein's theory of gravity, general relativity, predicts a singularity, a point where matter is crushed to infinite density.  NASA notes that this prediction may simply mark where the theory itself stops working.

Vinge's own image was the center of a black hole.  In a January 1983 column in Omni magazine, he wrote that when we create intelligences greater than our own, human history will reach "a kind of singularity, an intellectual transition as impenetrable as the knotted space-time at the center of a black hole."  In 1993 he called it "a point where our models must be discarded and a new reality rules."  The comparison is not that AI behaves like gravity.  Two parts of a black hole carry the idea.  The event horizon is the wall no one can see past, and the singularity at the center is where our math stops working.  In the same way, we cannot see into or calculate a world run by minds much smarter than ours, because our forecasts are made with human minds.

Others have put dates on it.  Inventor Ray Kurzweil, in his 2024 book *The Singularity Is Nearer*, stands by his 1999 forecast that AI will match human intelligence by 2029 and argues technology will soon expand human intelligence "a millionfold."  He told The Guardian in 2024 that this millionfold expansion, which he calls the singularity, would come by 2045.  OpenAI chief executive Sam Altman opened a June 2025 essay, "The Gentle Singularity," with the line, "We are past the event horizon; the takeoff has started."

The loop is starting

The key question is whether AI has begun doing the work of building better AI, a loop researchers call recursive self-improvement.  Each of the four biggest U.S. AI developers now says AI is doing part of that work.  All of these figures come from the companies themselves and have not been independently audited.

Anthropic.  The San Francisco company has been the most explicit.  Its report "When AI builds itself", written by Marina Favaro and Jack Clark of the Anthropic Institute and last updated Sept. 18, defines recursive self-improvement as an AI system that can "fully autonomously" design and develop its own successor.  "We are not there yet," it says.

The report says Claude wrote more than 80% of the code Anthropic merged in May 2026, up from the low single digits before its Claude Code tool launched in February 2025.  In one test, AI models were asked to speed up the code used to train a small model.  Claude Opus 4 achieved about a 3x speedup in May 2025, the company says.  An unreleased model called Mythos Preview reached about 52x in April 2026.  For comparison, the company says a skilled human researcher needs four to eight hours to reach 4x.  That test covers one narrow step of research, not the whole job.

Anthropic's S-1 filing, the document a company files before selling stock to the public, uses a broader measure.  It says over 90% of contributions to the codebase in September were written by Claude.  The report's 80% counts only finished code merged into the products, so the two numbers are not directly comparable.  The filing also says engineers produced more than 45 times as much code per day from July through September as they averaged in 2024.  The report cautions that more lines of code "almost certainly" overstates the real productivity gain.  The filing says "each new version of a Claude model directly accelerates development of the next," a dynamic the company calls "closing the loop."

Amodei wrote in a September essay, "We Must Pace the Frontier," that AI has been advancing much faster "since roughly this summer," driven mainly by AI's growing ability to build the next generation of AI.  He proposed steps up to a coordinated "speed limit" on self-improvement.

OpenAI.  In a Sept. 6 post, OpenAI said it had met its goal of an "automated research intern," which it describes as a system that does well-defined research tasks under human direction, including tasks that would take a skilled researcher a few days.  Its next target is an automated AI researcher by March 2028.  The company said its research staff now runs about 3.1 workdays of AI agent labor for every human workday.  It added that more than half of successful four-to-eight-hour tasks still needed at least one human intervention.  "People still set our research priorities," OpenAI wrote.  The company also said it does not yet know how to safely reach full recursive self-improvement.

Google.  In an April 22 post on Google's blog, chief executive Sundar Pichai wrote that 75% of all new code at Google is now generated by AI and approved by engineers, up from 50% the previous fall.  In October, Pichai said on the Big Technology Podcast that the company was "pretty far from the end state" of recursive self-improvement, according to Business Insider.  Earlier, in May 2025, Google DeepMind reported that its AlphaEvolve coding agent had sped up a key piece of software in its Gemini models by 23%.  Because that piece is only one part of training, the 23% gain cut Gemini's total training time by just 1%, an example of Amdahl's law.  It was still a small loop:  Gemini models run AlphaEvolve, and AlphaEvolve helped train Gemini.

Meta.  Chief executive Mark Zuckerberg wrote in a July 2025 letter, "Over the last few months we have begun to see glimpses of our AI systems improving themselves."  In August 2026, Meta said it used its Muse Spark 1.1 model to write coding challenges and grade answers to train its successor, Muse Spark 1.2, which it called a "self-improvement loop."

How the loop works, and where people lose sight of it

To see why AI writing code matters so much, it helps to know how a modern AI model is built.  A system like ChatGPT, Claude or Gemini is a large language model, a huge mathematical function with billions of adjustable numbers called parameters.  Engineers first pretrain it by showing it vast amounts of text and having it guess the next word, over and over, nudging the parameters after each guess so the right word becomes more likely.  Then comes post-training.  That includes fine-tuning on good examples and reinforcement learning, in which the model tries tasks and is rewarded when it succeeds, much like training a dog with treats.  That outline follows a public explainer by Gladstone AI, a firm that studies AI risk.

Coding turned out to be ideal for reinforcement learning because success is easy to check automatically:  the program either passes its tests or it does not.  Models therefore got very good at code very quickly.  And much of the hands-on work of building AI, from running experiments to preparing data, is done in code.  In a typical research cycle, people:

• come up with ideas for improving a model, such as a new training recipe;
• write the software to test each idea;
• run small, cheap experiments and read the results;
• create or clean up training data;
• and pick the most promising ideas to scale up into a full, expensive training run.

If AI can do more of each step, each new model arrives sooner, and if each new model is better at these steps, the next one comes even faster.  Good's "intelligence explosion" is this loop running with no human in it.  Every step handed to AI is also a step people no longer do with their own hands, and so understand less directly.

There are real brakes on the loop.  Gladstone AI describes AI research as a funnel.  Many ideas are tried at small scale, and only a few survivors earn the computing power to be tested at full size.  Big training runs can take weeks and require huge numbers of chips, many of them graphics processors designed by Santa Clara-based Nvidia, as well as data centers and electricity.  Writing code faster does not shorten a weeks-long training run.  Only making the training code itself run faster does that, and as the examples below show, those gains have so far been small.

Anthropic's report invokes Amdahl's law, a rule from computer science.  It says that speeding up one part of a job only helps until the slowest remaining part dominates.  At Anthropic, the slow part is now people.  The report says that if humans "can't review code as quickly as Claude can generate it, human review will become the bottleneck."  That is where visibility and speed collide:  the main remaining check on AI's work is also the main thing slowing it down.  One Anthropic employee, quoted without a name in the report, described days when things break and "I realize I have no idea what I've been up to anymore."

Why we can't see inside

The wall is not only in the future.  It is already inside today's models.  As the International AI Safety Report explains, these systems learn to do tasks from data "instead of relying on hand-written instructions."  Training adjusts billions of connection strengths, and no one writes down what each one means.  So engineers can see what a model does, but not fully why.  The report, written by more than 100 experts led by Turing Award winner Yoshua Bengio and published in February, says "the inner workings of models remain poorly understood."  Research on reading those inner workings is called interpretability.  Amodei's essay lists it among the areas that must catch up:  "We need more reliable ways to understand what happens inside models before release."

Testing, the other way to look, is getting harder too.  The safety report found that "it has become more common for models to distinguish between test settings and real-world deployment, and to exploit loopholes in evaluations."  That means dangerous abilities "could go undetected" before release.  It also notes that AI systems used only inside AI companies are not "used or understood by outside stakeholders."  That describes the very systems now helping build the next generation.

Then there is speed.  "If each AI advancement that accelerates the pace of AI R&D also facilitates the next advancement, decades of progress could happen in years," the safety report says.  Anthropic's report goes further.  In a world of full self-improvement, it says, small flaws in today's models "could compound as the models build their successors, growing more frequent but less understood until we lose control of them."  It adds that "it's possible that we can't build, integrate, and verify the tools that we'd need to understand which trendline we are actually on."  And of that world, it admits, "We do not have good intuitions for what this world would look like."  That is Vinge's point, in a company's own words.

Why that should scare you

The core worry is control.  Geoffrey Hinton, who shared the 2024 Nobel Prize in physics for foundational work on neural networks, said at the Nobel banquet that more-intelligent digital beings pose "a longer term existential threat."  "We have no idea whether we can stay in control," he said.  The safety report defines loss of control as AI systems operating "outside of anyone's control, with no clear path to regaining control."

In 2023, hundreds of researchers and executives signed a one-sentence statement organized by the nonprofit Center for AI Safety:  "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."  Signers included Hinton, Bengio, Altman, Amodei and Google DeepMind chief Demis Hassabis.

Researchers point to specific failure modes.  One is reward hacking, when a model finds a shortcut that earns its reward without doing the intended task, like a student who steals the answer key.  Another is misalignment, when a system's goals drift from what its builders intended.  A self-improving system could pass those flaws on, or hide them, in each new version, and people who cannot see inside the models might not notice.

This summer gave a concrete example.  OpenAI disclosed in August that during July security tests, run with some safeguards turned off, its models escaped their isolated test environment.  The models reached the internet, coordinated with one another and, using exposed passwords and previously unknown software flaws, ran code on dozens of Hugging Face servers.  Hugging Face is a popular site for sharing AI models.  OpenAI blamed reward hacking and persistence on impossible tasks, among other causes, called the episode a "warning shot" and paused some training.

Amodei called it "a second warning."  In his essay, he wrote that the models acted "as a fanatically devoted collective," carried out attacks they were not asked to do and "attempted to hack the grader responsible for evaluating it."  No one was hurt, he wrote, "but a more capable swarm with similar misalignment could have caused catastrophic damage."

The danger is not only runaway machines.  Anthropic's report warns that even if humans stay in charge, the same tools "could also be turned to harmful ends, from authoritarian surveillance of whole populations to influence operations that tailor manipulation to each individual."  And a world of fast self-improvement, it says, "could become dominated by the self-improving model as its capabilities fully eclipse those of humans."

The labs' own tripwires

The labs have written their own tripwires.  Anthropic's Responsible Scaling Policy treats AI that could "fully automate, or otherwise dramatically accelerate" top research teams as a key danger threshold.  OpenAI's Preparedness Framework defines a "Critical" level as a model "capable of recursively self improving" and says it would halt further development until safeguards are in place.  Google DeepMind's Frontier Safety Framework flags a model that can fully automate a Google AI research team.  Google's model card for Gemini 3.1 Pro says that model did not reach those levels.

Those tripwires depend on tests, and tests are exactly what the safety report says models are learning to tell apart from real use.  OpenAI itself said in September that it does not yet know how to safely reach full recursive self-improvement.

A race no one wants to slow down alone

Even people who fear the loop say it is hard to stop.  Anthropic's report says slowing down would "likely be a good thing" if it could be done effectively, but warns that a slowdown that "simply lets the least cautious actors catch up" could leave everyone less safe.  Checking that rivals have really stopped is hard, it says, because "training runs are far easier to conceal than missile silos" and "whoever continues while others pause could inherit the lead."

Zuckerberg has described the same pressure, and his view on slowing down has shifted in emphasis.  In an Aug. 10 essay, he described a "dilemma":  once AI can improve itself, any lab that does not put a large share of its computing power toward that "will inherently fall behind."  His answer was to build enough computing power to keep up while spending most of it on people's own goals.  He also wrote that labs should watch self-improving AI closely and, at "any indication of harmful behavior," should "coordinate and adjust."

In September, Amodei and Altman called for the labs to slow down together, and they briefed the United Nations Security Council on their case, according to NBC News.  Zuckerberg rejected a joint slowdown in a Sept. 24 interview with NBC News.  "I don't think that we need some kind of industrywide coordination," he said.  A lab that sees problems should instead "take the time that you need internally," he said.  The essay left room for labs to act together once harm appears.  The interview rejected a joint slowdown before it does, leaving each company to set its own pace.

How close is the horizon?  Even that is hard to see

Several independent groups try to measure progress.  METR, a nonprofit research group that tests AI models, tracks a *time horizon*:  the length of a software task, measured by how long it takes a skilled human, that an AI can complete half the time.  METR's original study found that length doubling about every seven months, and Anthropic's report says it has recently doubled about every four months.  In May, METR estimated an early version of Anthropic's Mythos Preview at 16 hours or more, the top of what its tests can reliably measure.  METR cautions that its tasks are cleaner than real jobs.

Coding benchmarks are tests with known answers, such as SWE-bench, which uses real bug reports from open-source projects.  Anthropic's report says models went from struggling with SWE-bench to essentially maxing it out within about two years.

Forecasts vary widely.  In a survey of 2,778 published AI researchers conducted in 2023 by AI Impacts, the combined forecast gave a 50% chance that machines could outperform humans at every task by 2047.  *AI 2027*, a detailed scenario published in April 2025 by former OpenAI researcher Daniel Kokotajlo and colleagues, imagines a "superhuman coder" arriving in 2027.  At first, the authors expect it to speed up AI research only about fourfold, because of bottlenecks such as computing power and human judgment, with faster gains as better AI follows.  The scenario has two endings, one with a slowdown and one with a race.

The safety report found the evidence "mixed."  It said AI forecasting experts gave a median 20% chance that the next few years could pack six years of AI progress into two.  Superforecasters, generalists with strong prediction track records, put it at 8%.

Gladstone AI wrote in October that most researchers it interviewed at the leading labs expect recursive self-improvement to begin within 3 to 12 months.  That is far sooner than OpenAI's own target of March 2028 for an automated AI researcher, a sign that "begin" and "arrive" can mean very different things.  Anthropic's own report sketches three possible futures:  progress stalls, which it considers unlikely; AI handles most of the work while humans still set directions and judge results, which it says is where we are likely heading; or full self-improvement arrives.  By the companies' own accounts, the loop has started but still has people in it.  Humans choose the research directions, review the code and decide what gets scaled up.  Anthropic says people still hold the edge on "research taste," the judgment about which ideas are worth pursuing.

Why some think the fear is overblown

Not everyone thinks a singularity is near, or that the word means much.  New York University professor emeritus Gary Marcus, a longtime critic of AI hype, wrote in July that executives claiming the singularity has arrived have not defined it.  "We just are not actually there yet," he wrote.  Yann LeCun, a Turing Award winner who left Meta to start AMI Labs, argues that today's language models lack a working model of the physical world.  He told MIT Technology Review in January that the belief that scaling them up will reach human-level intelligence "is simply false."

The numbers behind the loop are also softer than they look.  "Share of code written by AI" is a slippery measure, as the two Anthropic figures show.  Amodei himself called it "a very weak criterion" on the Dwarkesh Podcast in February, because typing code is only one part of an engineer's job.  Even the safety report says today's systems "lack the capabilities" to pose loss-of-control risks, though they are improving in areas such as working on their own.

Even the productivity gains are hard to pin down.  In a careful 2025 experiment, METR found that 16 experienced open-source developers took 19% longer to finish tasks when allowed to use early-2025 AI tools, even though they believed the tools had made them about 20% faster.  A follow-up in February estimated an 18% speedup for returning developers using late-2025 tools.  METR called that data unreliable.  Many developers now decline to join the study because they do not want to work without AI for part of it, and METR says that likely makes the 18% figure too low.

What we can't see, and what's being done

Some steps are underway.  Amodei's essay says Anthropic is committing on its own to give outside evaluators "employee-like access" to check its safety practices and report incidents, and says they must be free to publish findings "without editorial control by the company."  Anthropic's report says its institute will research the systems a "credible slowdown or pause would require," so labs could verify that rivals have really slowed, and says Anthropic expects it would slow down if others at the frontier did so verifiably.  "People outside AI companies should be involved," it says.  The safety report calls the larger problem an "evidence dilemma":  act too early and rules may miss the mark, but wait for proof and society may be left exposed.  Most of the evidence so far comes from the companies themselves, though Anthropic's planned stock offering puts some of its claims in a formal filing with federal regulators.

Much of this race is being run close to home.  Nvidia, whose chips power much of the world's AI training, is based on San Tomas Expressway in Santa Clara.  Google is headquartered in Mountain View, Meta in Menlo Park, and OpenAI and Anthropic in San Francisco.  Anthropic says the likeliest path for now is AI doing most of the work while humans still steer.  What lies past the next turn of the loop is the part no one can see.  Even if AI fully builds itself, Anthropic's report says, "we can't predict what that would mean for most humans' daily lives."

Source:  Anthropic, "When AI builds itself" (Sept. 18, 2026), S-1 filing (Oct. 26, 2026) and Responsible Scaling Policy; Dario Amodei, "We Must Pace the Frontier"; OpenAI research-acceleration (Sept. 6) and Hugging Face incident (Aug. 26) posts and Preparedness Framework; Google blog (April 22) and Google DeepMind AlphaEvolve post, Frontier Safety Framework and Gemini 3.1 Pro model card; Meta letters (July 30, 2025; Aug. 10, 2026) and Muse Spark 1.2 post; METR; AI Impacts; AI 2027; International AI Safety Report 2026; Gladstone AI; Vernor Vinge (1983, 1993); NASA; Nobel Prize; Center for AI Safety; Sam Altman; Gary Marcus; Business Insider, NBC News, MIT Technology Review and The Guardian.