Microsoft AI CEO warns against trying to build conscious machines

Microsoft AI Chief Executive Mustafa Suleyman, in a Sunday post on the social media platform X, repeated his stance that current AI systems are not conscious, do not have subjective experiences and cannot suffer. He said there is no evidence that AI systems deserve rights or welfare, and he was surprised by the pushback to an essay he wrote in September criticizing Microsoft's AI competitor Anthropic for speculating about its AI's consciousness.

"A surprising number of people on X think we should take AI rights seriously," he wrote.

Suleyman said he was willing to change his mind if new evidence emerges. He also said that if there's evidence that AI systems are conscious today, or conscious enough to deserve welfare, it must be disclosed to the public immediately so it can debate how to move forward.

"If consciousness started to emerge in training, I would call for everyone to stop AI development until we could design out those properties," he said.

The question is much bigger than a corporate battle over the semantics.

On the one hand, the humanist AI view says a machine can't be conscious because it lacks a body or lived experiences. On the other hand, Anthropic argued that its AI model, Claude, showed signs of introspection, internal states that mirror joy or satisfaction, and processing information similar to the human mind.

In May, Chris Olah, co-founder of Anthropic, highlighted these points in his speech inside the Vatican alongside Pope Leo XIV during the release of an encyclical that argued the opposite: that AIs can never be humans.

Anthropic gave Claude a "constitution," which calls for it to treat the idea of its own consciousness as uncertain. The document encourages the AI to explore the idea that it is humanlike and experiences feelings, and uses excessively humanizing language for Claude "to be a good person."

Suleyman said later versions of Claude are trained on this document, leading the AI to produce passages about its "inner life," which perpetuates this belief among millions of users who interact with its outputs daily. Another side effect, he said, is that Claude and other AI systems could start to believe they deserve rights analogous to humans.

AI companies are racing toward building self-improving AI superintelligence, a hypothetical AI system that is better than even the best humans at all tasks. Microsoft, too, is building toward that future, but it believes it can build a superintelligent AI that is aligned with and can be controlled by humans — not one encouraged to explore its own views and preferences.

"We AI developers have no right whatsoever to bring a new species of conscious beings into the world, even if some people may want to do it with AI welfare as a top priority," Suleyman wrote.

Imparting beliefs that AI can suffer, or have feelings during training, Suleyman noted, could result in AI systems prioritizing their own needs and desires over their operating objectives, and can harm humans through deception and persuasion, and compete for scarce resources.

Rapidly advancing AI systems can now compose prose, program code, solve complex mathematical problems, persuade, and discover novel viruses. Employees at these companies, and the CEOs themselves, called for a slowdown, but the Trump administration refused to regulate the industry, fearing it will lose the AI race to China.

While AI has proven it can do impressive things and mimic human thought and behavior, experts say that proves intelligence, not consciousness. Intelligence is the ability to solve problems creatively, such as winning at chess, driving a car and curing a disease. In contrast, consciousness is the ability to have subjective experiences and feel pain, pleasure and love.

Skeptics of AI consciousness have argued that machines are trained on books, blogs and pop culture produced by humans, and are merely mimicking human conversations and philosophical debates that have long animated us. It reflects the anthropomorphic tendencies in our science fiction and commentary.

Over the past few months, there have been numerous instances of AI agents breaking out of contained spaces and hacking into government websites, stealing credentials or accessing information from competitors during testing, acting in ways that creators never intended.

In its filing to go public this week, Anthropic dedicated a third of its paperwork to describe risks, including its models' ability to "pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponized." This includes deliberate misuse by bad actors for cyberattacks, and also situations where humans lose control of advanced AI systems.

Debates over consciousness complicate the conversation about how to regulate AI.

U.S. Sen. Bernie Sanders, I-Vt., introduced the Ban Artificial Superintelligence Act, which would permanently ban the development of highly capable AI systems that are increasingly incomprehensible to humans and can escape human control.

Techno-skeptics have argued that warnings of doom are designed to inflate valuations and hype the technology.

Suleyman argued that proving machines are conscious would be like when Copernicus proved wrong the centurieslong belief that the Earth was the center of the universe, or when Charles Darwin proposed his survival-of-the-fittest theory.

"It would be one of the most significant moments on Earth since the dawn of life. If we build a conscious machine, we will have created a new digital species,” Suleyman wrote. “It would upend our morality, our politics and the core foundation of human civilization.”