The idea that advanced AI will acquire “godlike” power and wipe out humanity has shaped years of public debate, but it rests on speculation. Misuse in cyber operations, biology and fraud can be measured now, and should set priorities.
Key takeaways
- The Guardian reports that a former senior British politician now working in the technology industry has dismissed fears of AI gaining godlike power to exterminate humanity, arguing that parts of the sector are winding themselves up unnecessarily.
- According to that report, the argument is not that AI is harmless but that firms should concentrate on identified threats such as cybersecurity and bioweapons rather than an unavoidable machine takeover.
- Specific misuse pathways can be tested, measured and mitigated with existing methods, whereas extinction scenarios depend on assumptions about future systems that nobody can currently evaluate.
- The strongest counterargument is that low-probability, irreversible risks justify precaution even without evidence, because there is no opportunity to learn from the failure after it happens.
The extinction framing is the weakest foundation for AI policy
Public argument about artificial intelligence has, for several years, been organised around a single dramatic question: whether sufficiently capable systems could end human civilisation. That framing has attracted attention from governments, funders and newsrooms. It has also been contested from inside the technology industry itself, most recently in remarks reported by the Guardian, in which a former deputy prime minister now working in the sector described parts of the industry as breathing their own fumes and characterised the extinction narrative as hand-wavy.
The argument set out here is narrower than “AI is safe”. It is that the extinction framing performs poorly as a basis for deciding what to do next. A useful risk claim tells you what to measure, what threshold matters, and what action follows when the threshold is crossed. Claims about known misuse pathways do that. A system that can meaningfully assist someone in synthesising a dangerous pathogen, or that can autonomously find and exploit software vulnerabilities at scale, produces a testable capability question and a corresponding mitigation: restrict the capability, gate access, harden the defending systems, or decline to release.
The extinction claim does not resolve into that form. It bundles together several separate propositions — that systems will become far more capable than humans across all domains, that such systems will develop goals of their own, that those goals will conflict with human survival, and that humans will be unable to intervene. Each of those steps is contested, and none of them yields a measurement that would settle the question one way or the other. The result is a debate that can be conducted indefinitely without either side producing evidence that would move the other.
That is not merely an academic problem. Regulatory attention, engineering effort and political capital are limited. A framing that cannot generate concrete tests tends to absorb those resources while producing few enforceable rules, and it does so at the expense of harms that are already documented.
Specific misuse pathways can be tested in a way extinction scenarios cannot
The practical difference between the two framings shows up clearly in how they are investigated. Questions about biological and chemical misuse, or about offensive cyber capability, are assessed through structured evaluation: build a set of tasks that stand in for the dangerous activity, run the model against them, compare performance against a baseline such as an unassisted person or an earlier model, and record whether the gap is growing. The work is difficult and imperfect — designing tasks that genuinely track real-world uplift, without themselves being dangerous to publish, is a serious methodological problem — but it is recognisably empirical. Results can be replicated, disputed and revised.
There is no comparable procedure for extinction. Proposals to measure it tend to collapse into measuring something else: how capable a model is in general, how well it pursues long-horizon goals, or how it behaves in artificial scenarios designed to elicit deceptive responses. Those are informative in their own right, but they do not establish the causal chain from capability to human extinction. They are proxies chosen because the thing itself cannot be observed.
This asymmetry matters for who gets to participate in the debate. Empirical risk questions can be examined by outside researchers, auditors and public bodies. Speculative ones can only be argued about, which in practice means the argument is dominated by whoever has the largest platform — frequently the executives of the firms building the systems. A risk narrative that only insiders can credibly discuss is a poor basis for democratic oversight, whatever its merits as a hypothesis.
Industry safety frameworks are already organised around specific capabilities
A telling piece of evidence is what the major developers actually do when they write down their safety commitments. The published frameworks used by leading laboratories are, in structure, capability-threshold documents. They identify specific domains of concern — typically biological and chemical weapons assistance, offensive cyber operations, and the ability of systems to operate autonomously over long tasks — and specify that when a model’s evaluated capability in one of those domains passes a defined level, additional safeguards apply before deployment or further scaling.
Notably, these documents do not operationalise extinction. They operationalise the narrower, more tractable concerns. That is a reasonable engineering decision, and it reveals something about the internal logic of the field: when firms have to commit to conditions they can be held to, they describe concrete capabilities, not civilisational endings. The same pattern appears in government work, where the public bodies set up to examine frontier systems have concentrated on evaluating misuse capability, model security and societal resilience, because those are the areas where testing produces answers.
If the practical machinery of AI safety — the evaluations, the thresholds, the release gates — is built entirely around identified threats, then the extinction framing is doing little work in the parts of the system that actually constrain behaviour. It functions largely as public rhetoric layered on top of a technical programme that is organised on different principles.
The present harms are not hypothetical and are accumulating
Beyond the security-adjacent threats, there is a category of harm that the extinction debate tends to push aside entirely: what these systems are doing now, at scale, in ordinary use. Automated generation of convincing text, voice and video has made impersonation fraud cheaper. Recommendation and generation systems shape what large populations read. Automated decision-making affects access to credit, employment and public services, often without meaningful explanation or appeal. Training data practices raise unresolved questions about consent and compensation. Energy and water demands from large-scale computing have concrete local effects.
None of these will end the species. All of them are happening, are documented, and are amenable to policy. They also tend to be less interesting to discuss than a machine takeover, which is precisely the problem. A public conversation calibrated to the most dramatic possible outcome systematically undervalues harms that are diffuse, unglamorous and distributed across people with little influence over technology policy.
The strongest case against this argument rests on irreversibility
The most serious objection is not that extinction is likely. It is that the standard decision rule — act on evidence, prioritise measurable harms — is the wrong rule for a particular class of risk. Where an outcome is catastrophic and irreversible, there is no chance to learn from a first failure and correct course. Under those conditions, precaution before the evidence arrives is not irrational; it is the only form of action available, because the evidence that would justify acting is evidence you can only obtain by allowing the outcome to occur.
This objection carries real force. Societies already apply it in other domains: nuclear safety engineering does not wait for accidents to establish base rates, and pandemic preparedness is justified by events that have not yet happened. Proponents also point out that capability progress in machine learning has repeatedly surprised expert forecasters, and that a risk becoming measurable only shortly before it materialises leaves very little time to respond. On that reading, insisting on testability is a demand that can only be satisfied too late.
There is a second version of the objection worth stating fairly: that the distinction being drawn here may be artificial. Loss of human control need not require malevolent superintelligence. It could emerge from the gradual delegation of consequential decisions to systems nobody fully understands, in finance, in infrastructure, in military command. Framed that way, “existential” risk and “concrete” risk are not competing categories but the same phenomenon at different scales, and dismissing the former weakens the argument for taking the latter seriously.
Specific evidence would change this conclusion
The position set out here is empirical, and should be revisable. Several developments would substantially weaken it.
The first would be a credible, replicable measurement of the disputed capabilities: a demonstration that current systems reliably form and pursue goals that were not specified by their operators, persist in them across contexts, and act to prevent correction. Isolated behaviours in constructed scenarios do not establish this; a robust and reproducible pattern in realistic deployment would.
The second would be evidence that the tractable-risk programme does not scale — that evaluation methods reliably fail to detect dangerous capabilities before release, or that capabilities emerge discontinuously enough that threshold-based governance cannot keep pace. If measurement itself breaks down, the argument for relying on measurement collapses with it.
The third would be a demonstrated case of significant real-world harm arising from autonomous system behaviour rather than human misuse. To date, the documented severe harms involve people deliberately using these tools for damaging ends. A clear instance of the other kind would shift the balance of the argument considerably.
Absent those, the sensible allocation of effort follows the evidence that exists. That means treating cyber and biological misuse, fraud, automated decision-making and concentration of infrastructure as the live agenda, while continuing to fund serious technical work on control and alignment as insurance rather than as the organising principle of public policy.
Sources and further reading
- The Guardian, technology section — the report of the remarks that prompted this piece, including the description of parts of the industry as breathing their own fumes.
- Published safety and responsible-scaling frameworks from major AI developers — company documents setting out capability thresholds and the safeguards triggered when they are crossed.
- National AI safety and security institutes — government bodies publishing evaluation methodology and findings on frontier model capabilities.
- Peer-reviewed machine learning and AI governance literature — academic work on capability evaluation, model auditing and the limits of current testing methods.
Surfaced from the rss:guardian_tech signal “industry dispute over AI extinction risk”. AI-assisted draft, editorially reviewed.
