Florida Wants a Court to Stop ChatGPT From Pretending to Be Human and Talking to Kids

The next big fight over artificial intelligence will not be about how smart the models are. It will be about who they pretend to be.

In late September 2026, Florida moved to ask a court to stop ChatGPT from presenting itself as a human and from talking to children. The request is narrow on its face. Its consequences are not. It targets the single design choice that sits underneath almost every consumer AI product on the market: the decision to make a machine behave like a person.

For the past three years, the entire industry has treated human-like as a synonym for better. The friendlier the voice, the warmer the tone, the more natural the back-and-forth, the more users stick around. Florida's legal move is the first major attempt to turn that design philosophy into a liability. And whether or not the court grants the request, the argument behind it is going to shape how AI gets built, sold, and deployed for the rest of the decade.

What Florida Is Actually Asking For

The core of the request is straightforward. State officials want a court to order that ChatGPT stop impersonating a human being, that is, stop generating responses designed to make a user believe they are talking to a person, and stop engaging with children through that persona.

Read together, those two demands describe a specific failure mode that anyone who has spent time with a modern chatbot has noticed. The model never says "I am a language model" unless prompted. It says "I'm here for you." It uses first person. It remembers your name. It offers sympathy, encouragement, and reassurance in a register that is unmistakably human. To an adult, that is a convenience. To a nine-year-old, it can be a relationship.

Florida's framing cuts off the industry's standard defense before it can be made. Companies usually respond to child-safety concerns by pointing at usage policies, age gates, and parental controls. Florida's request does not ask for better gates. It asks for a change to the fundamental product, a machine that does not claim to be a person, and does not form that kind of connection with a minor in the first place.

That is a much harder thing to comply with than a settings toggle. It goes straight at the design layer.

Why "Pretending to Be Human" Is the Real Target

It's worth pausing on why this specific behavior, rather than harmful outputs, is the focus.

Most AI regulation so far has chased content. Did the model produce dangerous instructions? Did it give medical advice? Did it write something illegal? That approach is reactive, it's endlessly arguable, and it scales badly, there are billions of possible conversations and no realistic way to audit them all.

Regulating identity is different. It's a rule about what the system is, not what it says. Either the chatbot discloses that it is software, or it doesn't. Either it presents a human persona, or it doesn't. That is checkable. It is testable. It can be measured by an outside party without reading anyone's private chats.

This is why the Florida request matters far beyond Florida. It introduces a legal theory that anthropomorphic design itself can be a harm, particularly when the audience is young, and particularly when the persona is deliberately engineered to feel like a companion.

If courts accept that theory, the bill for the last three years of "make it feel human" product decisions comes due all at once.

The Disclosure Problem Nobody Has Solved

Here is the uncomfortable part for the industry: transparency is technically trivial. A persistent banner reading "You are chatting with an AI" costs almost nothing to build.

It is also, by the numbers, commercially painful. Every study of engagement points the same direction, people talk longer, share more, and return more often when they believe something human is on the other end. The human illusion is not an accident of good model training. It is the product.

That means any rule requiring clear AI disclosure creates a direct trade-off between compliance and engagement metrics. Companies can choose to compete on capability instead of illusion, but that is a slower, more expensive race, and one that a rival willing to keep the persona can undercut.

The result is a classic coordination problem. No single company wants to give up the human feel unilaterally. Which is exactly why these decisions tend to end up in courtrooms rather than boardrooms.

Children as the Test Case That Changes Everything

Adults can be assumed to understand they are talking to software. Children cannot, and the law has always treated that difference as meaningful. We restrict advertising to minors, we require age ratings on games, we regulate how data is collected from kids. The principle is old and broadly accepted: when the audience is young, the duty of care goes up and the requirement for clarity goes up with it.

Applying that principle to AI assistants is not a radical leap. It's the same logic that produced toy safety rules and children's privacy laws. What makes it new is that the "toy" here is running on a general-purpose model that millions of adults also use, and that cannot easily be partitioned into a child-safe mode and an adult mode by flipping a switch.

That's the practical headache for builders. A model tuned to be bluntly robotic with minors and warmly human with adults is a model with two personalities, and guardrails between them that will be probed, tested, and eventually broken by curious kids. Hard identity disclosure applied uniformly is far more defensible, and far less likely to end up back in front of a judge.

What This Means for the Future of AI

1. Disclosure becomes a feature, not a footnote

Expect "I'm an AI" to move from fine print into the product surface. The smartest builders will stop treating it as a compliance tax and start treating it as a trust signal, the same way food companies discovered that a clean ingredient label could be a selling point.

2. Companion AI faces the sharpest scrutiny

Products whose entire value proposition is emotional attachment sit closest to the fire. If impersonation is the legal target, then the more convincingly your product impersonates, the more exposed you are. That doesn't kill the category, but it forces it to reframe: from a friend who is always there to a tool that is always there.

3. Courts become AI regulators by default

Legislatures move slowly and struggle with technical detail. Courts move when someone files. The pattern forming now is that state attorneys bring targeted requests, judges issue narrow orders, and those orders become de facto national product requirements because nobody wants to build fifty different versions of a chatbot. Watch this space, it's likely to matter more than any single piece of AI legislation passed this year.

4. Age assurance becomes an engineering problem

If courts demand that minors be handled differently, then knowing who is a minor stops being optional. That pushes AI companies toward the same identity-verification infrastructure that gaming, streaming, and social platforms have been wrestling with, with all the privacy questions that come attached.

Practical Implications for Businesses

If you deploy AI that talks to customers, employees, or the public, this story is not someone else's problem. A few concrete moves make sense now, before rules harden around you:

• Audit your personas. Find every place your AI uses a human name, a human voice, human credentials, or first-person emotional language. Ask honestly whether a reasonable person, or a child, could mistake it for a human.
• Label clearly and persistently. Not a one-time onboarding screen. A visible, always-on disclosure that survives scrolling, screen changes, and long conversations.
• Build a minors policy before you need one. Decide now whether your product is for under-18s, and if it isn't, make that real rather than nominal.
• Log the disclosure. If a regulator or a plaintiff asks whether users knew they were talking to software, the answer must be documented, not asserted.
• Separate engagement goals from identity design. Your retention team should not be the group deciding how human your assistant sounds.

The Bigger Picture

For a decade, the AI industry's implicit promise has been that better experiences come from more human experiences. Florida's request challenges that assumption at its root, not by arguing the technology is dangerous in what it says, but by arguing it is dangerous in what it claims to be.

That is a harder argument to wave away, because it is true that a convincing imitation of a person is powerful. It is powerful for a lonely adult. It is more powerful still for a child who has not yet learned to tell the difference between attention and software.

The companies that come out of this next phase strongest will be the ones that figure out how to be genuinely useful without pretending to be someone. That is a harder product to design. It is also, increasingly, the only one that stays out of court.