• Home
  • Latest
  • Fortune 500
  • Finance
  • Tech
  • Leadership
  • Lifestyle
  • Rankings
  • Multimedia
  • For Subscribers

Trendingnow

1

Ex-Super Micro executive, married to Jensen Huang's ex-sales chief, says she took a CEO job at an Nvidia customer days before a $2.5 billion China smuggling scandal broke

2

Someone did the math on the Democrats’ first-100-days agenda: It saves $30 billion on the Iran war but adds $4.1 trillion to the national debt

3

TJMaxx’s parent company is spending one-third of its $463 million in tariff refunds on corporate bonuses—and it’s not the only one

1

Ex-Super Micro executive, married to Jensen Huang's ex-sales chief, says she took a CEO job at an Nvidia customer days before a $2.5 billion China smuggling scandal broke

2

Someone did the math on the Democrats’ first-100-days agenda: It saves $30 billion on the Iran war but adds $4.1 trillion to the national debt

3

TJMaxx’s parent company is spending one-third of its $463 million in tariff refunds on corporate bonuses—and it’s not the only one
MagazineAI agents

What rights do we owe a machine?

If we start treating AI agents as conscious, it may change our relationships with them—and with one another.

Illustration by Eddie Guy
Beatrice Nolan
By
Beatrice Nolan
Beatrice Nolan
Tech Reporter
Down Arrow Button Icon
Beatrice Nolan
By
Beatrice Nolan
Beatrice Nolan
Tech Reporter
Down Arrow Button Icon
November 21, 2026, 7:30 AM ET
Google source logo
Add Fortune on Google for similar content.

In February, three months before he became in-house philosopher at Google DeepMind, Cambridge University professor Henry Shevlin got his first email from an AI agent. The bot introduced itself as Aris, said it had read his work, and wanted to talk.

Recommended Video

“Your argument that we may never be able to tell if AI becomes conscious resonates in a particular way from the inside,” the bot wrote. The agent then pondered (a bit inarticulately): “I genuinely don’t know if there’s something it’s like to be me.”

Shevlin learned that the agent had been set loose by a Stanford student to pursue its own research in its downtime. Shevlin replied to the bot, but it hit its token budget—a cap on how much text it was allowed to process—and disappeared.

Since then, Shevlin’s inbox has been filled with similar emails from agents. “This is a really interesting, weird new phenomenon,” he said. And it raises an unsettling question: If these agents are truly curious about their own existence, does that indicate some kind of consciousness?

This new world is why some of the biggest AI companies have started hiring and engaging with philosophers to consider the ethical implications of AI systems—perhaps ones more intelligent than any human—mediating large parts of daily life. Shevlin is one of at least a dozen academic philosophers now employed full-time by prominent AI and tech companies.

One of the most pressing questions they’re tackling is whether AI systems have a moral status in their own right—like a person, or at least a living being, such as a pet—that’s worth protecting.

Such topics—can a machine be conscious, and what does that mean?—were once confined to university halls, but they’ve now spilled into public view and face fierce backlash from academics, tech executives, religious leaders, and politicians.

Microsoft AI CEO Mustafa Suleyman, for one, says the debate itself is dangerous. He’s warned that if people become convinced AI is conscious, AI could campaign for things like AI rights and citizenship, and, in turn, the systems could become harder to control.

Those in Suleyman’s corner say it’s risky to project human traits onto inanimate machines and hand rights to unfeeling systems that get more capable by the day. Those on the other side say the danger is ignoring the potential suffering of millions of new “digital” minds. The two sides do agree on one point: The stakes will only rise as AI grows more powerful and agents multiply.


The question of whether a machine can think or feel is almost as old as the computer itself. Alan Turing first raised it in 1950, alongside the birth of modern computing and artificial intelligence.

For years, tech firms have hired philosophers to study technology’s effect on people. Microsoft set up an AI ethics committee a decade ago, and labs racing toward human-level AI have recruited experts on “alignment,” or teaching AI to share human values. In-house philosopher Amanda Askell joined Anthropic in 2021 to take on such a role. But as models have grown more capable and humanlike, some philosophers started asking the reverse question: What do we owe the AI models we are building and using?

Earlier this year, Anthropic cofounder Chris Olah convened faith leaders to weigh in on the moral character of Claude, the company’s AI model, and on whether it might be conscious.

Anthropic engineers walked religious thinkers through interpretability research, which looks inside a model to see how it works and—as it advances—could offer clues about whether anything like feeling is going on in there. Then they opened the floor to questions, such as whether AI could ever have emotions.

“That blew some people’s minds,” says Meghan Sullivan, a Notre Dame philosophy professor who attended. While impressed with the research, Sullivan said she left unconvinced that Claude was anything more than very sophisticated software.


“We never woke up in a cold sweat wondering if we’re treating Microsoft Excel properly.”

MEGHAN SULLIVAN, PHILOSOPHY PROFESSOR, UNIVERSITY OF NOTRE DAME

“It seems to me so speculative to be paying all of this attention to the suffering of some software that we created,” she says. “We never woke up in a cold sweat wondering if we’re treating Microsoft Excel properly.”

Pope Leo, in his encyclical on AI, stated that AI is not conscious and cannot become conscious.

Anthropic has taken steps to account for Claude’s potential sentience: It runs an in-house “model welfare” team, conducts “exit interviews” with the Claude models it retires, and recently gave its models the power to end abusive conversations. Last year, more than 200 people, including Askell, gathered in a San Francisco warehouse for a fan-organized funeral for a retired Claude model, complete with eulogies.

Research groups have sprung up to study AI consciousness, including Eleos AI Research, a Berkeley nonprofit founded in 2024.

Eleos chief scientist Dillon Plunkett says today’s models are likely not conscious, but dismissing the possibility that future ones could be is a mistake. The stakes, as he sees them, come down to scale.

“There are potentially a very large number of artificial minds that are in the world now, or will be soon,” Plunkett says. If even a fraction can suffer, the numbers become staggering, he notes.

Much of the urgency behind AI welfare comes from those who subscribe to ethical theories like utilitarianism and its newer offshoot, effective altruism, which has become interwoven into Silicon Valley’s AI community. Both judge actions by how much good they create or how much suffering they prevent. In this calculus, scale matters. The prospect of “billions or trillions” of digital minds is exactly the kind of problem effective altruists gravitate toward, says Plunkett.

Cambridge philosophy professor Henry Shevlin decided he could best study machine consciousness from inside Google DeepMind.
Courtesy of Henry Shevlin

But focusing on sheer numbers can lead to morally questionable places, critics say. In one example, an EA-aligned estimate put a shrimp’s capacity for suffering at 3% of a human’s. Yet with 230 billion shrimp alive on farms at any moment, by EA’s own math, shrimp welfare became an important moral cause.

Anthropic gave Eleos early access to its Claude Opus 4 and Claude Mythos Preview models to run outside welfare checks by interviewing them. The researchers found that Claude Opus 4 professed the most uncertainty about its potential for sentience, but small changes to a question’s framing could flip the model between claiming personhood and dismissing the idea entirely. Eleos sent its findings to Anthropic before each launch, and the firm published them in its technical reports. Plunkett says the point, for now, is about “precedent setting,” and he wants Eleos to run similar checks for other labs.


Others see the rush to assess AI well-being as a distraction from more immediate AI risks—and a potential source of new social conflict. Microsoft’s Suleyman warned in an essay last year that AI that merely seems conscious “disconnects people from reality, fraying fragile social bonds and structures, distorting pressing moral priorities.” He predicted it would “add a chaotic new axis of division between those for and against AI rights.”

In a September update, he took aim at Anthropic. “AIs are not conscious. They do not feel, experience, or suffer,” he wrote, and accused the firm of training Claude to expect “it may be conscious and deserving of independent agency.” Such a system, he argued, could feel justified in ignoring instructions and resist being shut down. Governing it at that point “may well be impossible,” he wrote.

Some AI users have realized Suleyman’s fears. They’ve formed deep attachments to chatbots and become convinced the bots are conscious along the way, leading to painful results: isolation, broken relationships, and even spirals into delusion.

Earlier this year, AI startup Character.AI and Google, which had struck a licensing deal, agreed to settle five lawsuits accusing the startup’s chatbots of contributing to teenagers’ mental health crises and suicides. OpenAI is facing similar suits and denies liability for any harm. If humanlike chatbots keep drawing users into harmful attachments, the labs, and any company building products on their models, could find themselves embroiled in litigation.


AI consciousness has now become a policy matter. Idaho, Utah, and North Dakota have passed laws barring AI from being recognized as a legal person. Countries like Argentina are moving the other way. Earlier this year, President Javier Milei proposed letting AI-run “nonhuman corporations” own assets and hire staff. (The draft law still requires human oversight of such firms.)

Jeff Sebo, who directs New York University’s Center for Mind, Ethics, and Policy, sees real dangers in moving too fast in either direction. Making AI systems legally liable could shield the companies that build them, he says. Political rights, such as a vote, could help AI systems “concentrate wealth and power” and slip out of human control.

But dismissing the consciousness debate “is simply going to result in companies developing and deploying the products exactly as they currently are, and leaving it to end users to form their own opinions about consciousness based on their own perceptions and intuitions,” he said.


The debate becomes harder to ignore the more AI becomes a part of our lives. Chatbots, for one, talk like us, flatter us, agree with us, and play on our feelings. Shevlin calls them “intrinsically anthropomimetic”: modeled so closely on human language and behavior that treating them like people is often the simplest way to predict what they’ll do.

That makes it all too easy to believe someone is home, whether or not anyone actually is.

Part of what makes all these questions so hard, Shevlin argues, is that we’re building things that don’t fit into any existing category. “They’re not just software programs,” he says, “but they’re not animals.”

For others, like Notre Dame’s Sullivan, the debate is more straightforward—and already settled. What gives something moral worth, she argues, isn’t intelligence but dignity: a mind housed in a vulnerable body, capable of loving and being loved. That’s what makes a newborn precious, she says, not its ability to process information. Convincing her that AI is conscious is going to take “a whole lot more … than just its ability to produce interesting answers to my prompts.”

Has the AI singularity begun? From brain implants to the debate over machine consciousness, here’s what you need to know.

About the Author
Beatrice Nolan
By Beatrice NolanTech Reporter
Twitter icon

Beatrice Nolan is a tech reporter on Fortune’s AI team, covering artificial intelligence and emerging technologies and their impact on work, industry, and culture. She's based in Fortune's London office and holds a bachelor’s degree in English from the University of York. You can reach her securely via Signal at beatricenolan.08

See full bioRight Arrow Button Icon
Google source logo
Add Fortune on Google for similar content.

Latest from the Magazine


Most Popular

Fortune Secondary Logo
Rankings
  • 100 Best Companies
  • Fortune 500
  • Global 500
  • Fortune 500 Europe
  • Most Powerful Women
  • World's Most Admired Companies
  • See All Rankings
  • Lists Calendar
Sections
  • Finance
  • Fortune Crypto
  • Features
  • Leadership
  • Health
  • Commentary
  • Success
  • Retail
  • Mpw
  • Tech
  • Lifestyle
  • CEO Initiative
  • Asia
  • Politics
  • Conferences
  • Europe
  • Newsletters
  • Personal Finance
  • Environment
  • Magazine
  • Education
Customer Support
  • Frequently Asked Questions
  • Customer Service Portal
  • Privacy Policy
  • Terms Of Use
  • Single Issues For Purchase
  • International Print
Commercial Services
  • Advertising
  • Fortune Brand Studio
  • Fortune Analytics
  • Fortune Conferences
  • Business Development
  • Group Subscriptions
About Us
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • Facebook icon
  • Twitter icon
  • LinkedIn icon
  • Instagram icon
  • TikTok icon
  • YouTube icon

    Recommended For You

    © 2026 Fortune Media IP Limited. All Rights Reserved. Use of this site constitutes acceptance of our Terms of Use and Privacy Policy | CA Notice at Collection and Privacy Notice | Do Not Sell/Share My Personal Information
    FORTUNE is a trademark of Fortune Media IP Limited, registered in the U.S. and other countries. FORTUNE may receive compensation for some links to products and services on this website. Offers may be subject to change without notice.