One of the founders of modern AI says the technology could threaten humanity and that companies cannot police themselves. On balance, the evidence supports him: the risks are too serious to rely on voluntary rules alone.
Key takeaways
- A pioneering AI researcher has repeated his warning that advanced AI systems could pose an existential threat to humanity, the Guardian reports.
- The same researcher rejected the idea that the AI industry can be trusted to regulate itself.
- Commercial incentives, the limited understanding of how large AI systems work, and the record of other high-risk industries all support external oversight of advanced AI.
- The main argument against binding regulation is that it could slow useful innovation, act on risks that are still speculative and favour the largest companies.
The case for external oversight of advanced AI
The Guardian has published a long interview with a computer scientist often called a “godfather” of AI, whose research helped lay the foundations of today’s systems. According to the newspaper, he repeated his recent warnings that AI models pose an existential threat to humanity. He also dismissed claims that the industry could regulate itself. He described advanced AI as something like an alien form of intelligence. He said that when human interests and machine interests conflict, he is prepared to put humans first. In his view, society’s biggest challenge may be accepting that humans are no longer the only intelligent beings around. The interview also included sharp political criticism, including of how US political leaders use the word “superintelligence”. This article does not cover those personal remarks. It looks at the policy question behind them.
That question can be put simply. If a technology might one day act in ways its makers cannot predict or control, and if the possible harm is very large and cannot be undone, then the people who profit from building it should not be the only ones who decide how safe is safe enough. Supporters of this view do not claim catastrophe is certain. They argue that when the downside is severe and the uncertainty is high, voluntary promises are not enough. Independent rules are needed, and they need to be enforceable.
This is not an argument for stopping AI research. It is an argument about who should set the limits and who should check that they are kept. Self-regulation means companies write their own standards, test their own products and decide for themselves whether to publish what they find. External regulation means a separate public body sets minimum requirements and can impose penalties when they are broken. The three sections below each set out one strand of evidence for the second approach. A further section then gives the strongest case against it.
Commercial incentives pull against caution
The first strand of evidence is about incentives. Building leading AI systems costs a great deal. It needs large amounts of computing power, scarce specialist staff and long periods of investment before any money comes back. Companies that spend this much are under pressure from investors, customers and rivals to release products quickly and to keep improving them. A firm that pauses to do more safety testing risks losing ground to one that does not.
Economists call this a collective action problem. Each company may sincerely want the whole industry to be careful. Yet each one also has a reason to move faster than the others. Voluntary commitments do little to fix this, because nothing forces any firm to keep to them once keeping them becomes costly. A pledge made in a calm period can be quietly dropped when competition heats up.
The interview reported by the Guardian makes the same basic point. Expecting the industry to regulate itself, the researcher suggests, means expecting companies to act against their own short-term commercial interests without any outside pressure. That is not an accusation of bad faith against particular firms. It is an observation about how markets reward speed. Shared, binding rules remove some of that pressure, because every competitor faces the same requirements and no single firm is punished for being careful.
The inner workings of large AI systems are poorly understood
The second strand of evidence is technical. Modern AI systems are not written line by line the way traditional software is. They are trained. Huge numbers of internal settings are adjusted automatically as the system processes very large amounts of data. What comes out can perform impressively, but even its builders cannot fully explain how it reaches a particular output.
Researchers working on what is often called interpretability are trying to open up these systems and understand the steps inside them. This is still an active and unfinished field. As a result, it is hard for anyone, including the developer, to guarantee in advance how a system will behave in situations it was not tested on. Unexpected abilities and unexpected failures can both appear as systems get larger and more capable.
This is where the researcher’s “alien beings” image does its work. His point is not that AI systems are alive in a biological sense. It is that their intelligence may not resemble human thinking, and that people should not assume they share human goals or limits. If the people building a system cannot fully say what it will do, then the developer’s own assurance that it is safe carries less weight. Independent testing, required disclosure of known problems and limits on deployment in sensitive settings are all ways of dealing with that uncertainty without relying only on the developer’s judgement.
Other high-risk industries are not left to police themselves
The third strand of evidence is historical and comparative. In many industries where failures can cause serious or widespread harm, societies have decided that self-regulation is not enough. New medicines usually have to pass independent assessment before they can be sold. Aircraft must meet certification standards set by public authorities. Nuclear facilities are inspected by bodies separate from their operators. In each case the companies involved keep a great deal of technical knowledge and influence, but they do not have the last word on safety.
These arrangements usually came about because people saw that the costs of failure would land on the public, not just on the company. A faulty product in an ordinary market might hurt a firm’s reputation and sales. A failure in a high-consequence field can harm people who never chose to take that risk. The argument for regulating advanced AI follows the same reasoning. If the possible harms include large-scale social disruption, or the more extreme outcomes some researchers warn about, then many people who have no say in a company’s decisions would bear the cost.
The comparison has limits. AI is a general-purpose technology used in many different products, not one kind of device, which makes simple licensing harder. Even so, the wider principle holds. Where the stakes are high and the people affected are numerous, outside oversight has been the normal response, not the exception.
The strongest case against binding regulation
The opposing case deserves to be taken seriously, and several parts of it carry real weight.
First, the most extreme risks are still speculative. No AI system has caused the kind of catastrophe some researchers describe, and experts disagree sharply about how likely such outcomes are and when they might happen. Regulating against hypothetical harms can produce rules that are badly aimed. They may target the wrong features of the technology while missing harms that are already happening, such as bias, misinformation or invasions of privacy.
Second, regulators may not have the expertise to keep up. The field moves quickly, and the people who understand frontier systems best mostly work inside the companies that build them. Rules written today could be out of date by the time they take effect. They could also be too vague to enforce, or so rigid that they block useful work.
Third, heavy regulation can protect the companies already at the top. Large firms can afford compliance teams and legal advice. Smaller companies, universities and open research projects often cannot. Rules that look protective could end up concentrating power among a few big developers, which some would see as a risk in its own right.
Fourth, there is international competition. If one jurisdiction imposes strict rules and others do not, development may simply move elsewhere. The technology would still advance, but under weaker oversight, and the stricter jurisdiction would have less influence over how it develops.
Finally, AI has large potential benefits in areas such as medicine, science and productivity. Rules that slow progress have costs as well, and those costs are real even if they are harder to see than the risks.
Together these points make a reasonable argument for caution about how AI is regulated. They do not show that the industry’s own judgement is enough. Many of them can be dealt with through careful design: rules scaled to a system’s capability, independent technical bodies, and international coordination.
Evidence that would change the conclusion
Several developments would weaken the case for binding external oversight. If interpretability research reached the point where developers could reliably check how a system works and predict how it will behave, one of the main sources of uncertainty would shrink. If voluntary industry commitments were regularly checked by independent auditors and visibly kept even when it cost the companies something, self-regulation would look more believable. If the capabilities of frontier systems stopped improving, the urgency of the most serious warnings would fade.
Other evidence would strengthen the case. That would include documented examples of companies cutting back safety work under competitive pressure, or systems behaving in ways their developers did not expect and could not explain.
It is not known from the reported interview what specific regulatory measures the researcher would favour, or how he would weigh the trade-offs above. The policy debate is likely to turn less on whether advanced AI should be overseen and more on the form that oversight takes.
Sources and further reading
- The Guardian: the interview with a pioneering AI researcher about existential risk and industry self-regulation.
- Academic research on AI interpretability: work on understanding the internal mechanisms of trained neural networks.
- Public policy analysis on technology regulation: comparisons between AI governance and oversight of medicines, aviation and nuclear power.
- Economics literature on collective action problems: how competition affects voluntary safety commitments.
Surfaced from the rss:guardian_tech signal “AI existential risk warning”. AI-assisted draft, editorially reviewed.
