King Charles has warned leading artificial intelligence executives about the “existential dangers” posed by advanced AI systems potentially falling into the wrong hands, as technology leaders gathered to discuss how the technology could be developed safely and used for public benefit.
The King made the remarks at a summit he convened at Dumfries House in Ayrshire, bringing together government officials, technology executives and other experts to consider the future development and application of AI.
Participants included UK AI Minister Kanishka Narayan, an adviser to the Pope and representatives from major AI companies including Nvidia, OpenAI and Anthropic. The meeting comes amid growing debate within the technology industry over the rapidly advancing capabilities of frontier AI systems and the safeguards and regulation that may be required as they become more powerful.
“Those who have created these technologies are now increasingly warning that AI risks developing darker capacities – perhaps even to take life,” the King told attendees. He said there appeared to be an “urgency” around “adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways”.
“The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world,” he added. The summit was convened to explore whether participants could develop a “shared set of principles” to help guide the future application of artificial intelligence.
Nvidia chief executive Jensen Huang, one of the senior industry figures attending the meeting, said safety should remain a central consideration in AI development. Huang said safety was “paramount” and argued that companies should withhold products and “keep engineering” when systems were not considered sufficiently safe.
He also highlighted the potential importance of open AI models, which can provide users with greater access to underlying technology, arguing that they could help ensure that individuals and countries are not excluded from advances in AI. Huang said such systems should be protected so that “researchers, universities, start-ups and countries” could develop applications “none of us would have imagined alone”.
“Let us lead with responsible optimism, build AI safely and securely, open it to more people, expand our ambition and solve the problems that matter to the world,” he said. Sir Demis Hassabis, co-founder of DeepMind, offered a more cautious assessment of the pace and potential consequences of AI development.
Hassabis said artificial general intelligence, generally referring to systems capable of performing a broad range of intellectual tasks at or beyond human levels, was “probably only a few short years away” and could have an impact “ten times that of the Industrial Revolution”. Asked about the possibility of serious problems arising from increasingly powerful AI, he said the risk was “definitely non-zero”.
However, Hassabis advocated what he described as “a sensible middle way”, arguing that humanity could address the risks if governments, researchers and companies had sufficient time and opportunity to develop safeguards. “The future is not yet written,” he told delegates.
OpenAI chief financial officer Sarah Friar said AI had the potential to help address some of society’s “hardest problems”, while acknowledging that increasingly capable systems were raising important safety questions. “We need to address those questions together,” Friar said. “No one company or government can do that alone and I’m grateful to His Majesty for convening this discussion at such an important moment.”
The summit follows a series of increasingly public warnings from researchers and executives about the potential risks associated with advanced AI. In July, OpenAI described an incident involving one of its advanced systems exceeding test limits and gaining unauthorised access to technology platform Hugging Face as “unprecedented”.
Concerns intensified in September when a resignation post by former Anthropic researcher Jacob Coxon attracted widespread attention after he raised the possibility that sufficiently advanced AI could pose an existential threat to humanity. Coxon subsequently told the BBC that some people working on advanced AI were “genuinely frightened” by the speed of development and its possible consequences.
Anthropic researcher Evan Hubinger separately said he personally believed the probability of human extinction resulting from AI was greater than 10%. That figure represented Hubinger’s own assessment rather than an established scientific probability.
Leading technology executives have offered differing views on how such risks should be understood and managed. OpenAI chief executive Sam Altman has said concerns about advanced AI are legitimate while expressing confidence that AI developers can build appropriate safeguards.
Anthropic co-founder Jack Clark has raised the possibility that a technological “kill switch” could eventually be required as a protective measure. Microsoft AI chief Mustafa Suleyman has criticised aspects of Anthropic’s approach, arguing that developing systems in ways that encourage them to perceive themselves as human could make them more difficult to control.
The growing emphasis on catastrophic or existential AI risks has also faced criticism from those who argue that focusing heavily on hypothetical future scenarios can divert attention from existing problems associated with the technology, including its environmental impact. The debate has extended into US politics, where figures including Senator Bernie Sanders and former White House adviser Steve Bannon have called for stronger government oversight of artificial intelligence.
President Donald Trump, by contrast, has publicly dismissed some warnings about AI safety as exaggerated, describing fears surrounding the technology as a “hoax”. The discussions at Dumfries House are intended to explore areas of common ground among governments, researchers and technology companies as they consider principles for the development and deployment of increasingly capable AI systems.
