AI hallucinations in customer service: real cases
Three documented cases of AI giving customers rules that didn't exist, what each one cost, why language models fill gaps with confident inventions, and six controls that keep an invented policy out of your replies, plus a template for correcting one that got out.
By Max Lindberg · · 7 min read
An AI hallucination in customer service is a confident answer that isn't true, and the costly kind is an invented policy: a refund rule, a device limit or a legal right the business doesn't have. In each of the documented cases below, people believed the AI, because it spoke for the business.
Below are those cases, taken from the tribunal decision and the news reports, why language models do this, the controls that stop an invented rule reaching a customer, and a template for correcting one that already has.
Three documented cases
| Who | What the AI said | What was true | What followed |
|---|---|---|---|
| Air Canada, website chatbot, 2022 | A bereavement fare could be claimed after travel | The policy page said it couldn't | A tribunal ordered the airline to pay $812.02 |
| Cursor, email support, 2025 | One device per subscription is "a core security feature" | No such policy existed | Public cancellations; the company apologised and refunded the user |
| New York City, MyCity chatbot, 2024 | Businesses could take a cut of workers' tips, and could refuse cash | Both were against the rules | The city took the chatbot down in 2026 |
Air Canada: the chatbot and the bereavement fare
In November 2022, after their grandmother died, Jake Moffatt asked the chatbot on Air Canada's website about bereavement fares. The chatbot said that a customer who had already travelled could submit their ticket for a reduced rate "within 90 days of the date your ticket was issued". The airline's own bereavement page, which the chatbot linked to, said the policy did not apply after travel. When Moffatt asked for the difference, an Air Canada representative admitted the chatbot had used "misleading words", but the two sides couldn't resolve it.
Before the Civil Resolution Tribunal of British Columbia, Air Canada argued it could not be held liable for information provided by its chatbot. The tribunal called this "a remarkable submission", found the airline had not taken reasonable care to ensure its chatbot was accurate, and ordered it to pay $812.02: $650.88 in damages, plus interest and fees. It also asked why customers "should have to double-check information found in one part of its website on another part of its website". The full reasoning, and what it means for a store, is in our guide Is it safe to let AI answer customer emails?
Cursor: the support email from "Sam"
In April 2025, Ars Technica reported that a user of Cursor, a code editor, kept being logged out when switching between computers and emailed support. The reply, signed "Sam", said: "Cursor is designed to work with one device per subscription as a core security feature." Sam was an AI, and the policy did not exist. After the user posted the reply on Reddit, several users announced they were cancelling.
A Cursor representative replied: "Hey! We have no such policy... Unfortunately, this is an incorrect response from a front-line AI support bot." The cause was a backend change to session security. A co-founder apologised, said the user had been refunded, and added: "Any AI responses used for email support are now clearly labeled as such."
This is the case closest to a small store: an email to support about a real problem, answered with a policy that didn't exist.
New York City: the business-rules chatbot
New York City launched a chatbot to answer business owners' questions about city rules. In March 2024, The Markup tested it and published the answers. Asked "Can I take a cut of my worker's tips?", it replied: "Yes, you can take a cut of your worker's tips." Asked about going cashless, it said "Yes, you can make your store cashless in New York City", though stores have had to accept cash since 2020. The Markup also found the answers changed: one reporter was told landlords did have to accept housing vouchers, but when ten staffers asked the same question, all ten were told they didn't. In February 2026, The City reported that the bot had been taken down.
Why AI support tools invent policies
A language model writes the most plausible next words. It has no built-in sense of "I don't know". OpenAI says this about its own product: ChatGPT "can produce incorrect or misleading outputs. Sometimes, it might sound confident—even when it's wrong", and "Confidence isn't reliability".
In a support inbox, three gaps turn that into an invented policy:
- The policy isn't in front of it. If the AI has no returns page to read, it writes a typical returns policy, which may not be yours.
- The question goes past the policy. Your page says 30 days; the customer asks about day 31 with a faulty item. A reply that fills the gap is a new rule.
- Something real needs explaining. Cursor's bot met an unexplained logout and produced a reason. A delayed parcel, a failed discount code or a double charge invites the same kind of invention.
Old replies make it worse. An AI that learns from your past emails also learns last year's prices and one-off exceptions, and repeats them as rules.
Six controls that keep invented policies out
| Control | What it stops |
|---|---|
| 1. Answer only from documents you approved | Typical-sounding policies that aren't yours |
| 2. Cite the document behind every fact | Facts nobody can trace; the reviewer checks the source, not their memory |
| 3. Rules in code that hold unsupported promises | A refund, replacement or exception no approved document allows |
| 4. Fill order facts from the order, not the model | Invented dates, tracking numbers and amounts |
| 5. A person reads and sends every reply | Everything the first four miss |
| 6. When the documents don't answer, the draft says so | Gap-filling: "I'll check and come back to you" beats a made-up rule |
Two notes. Control 3 has to be code, not an instruction to the AI: telling a model "never promise refunds" is a request, not a guarantee. A rule in code checks every draft the same way. And control 5 only works if the reviewer has the time and the sources to check; a person clicking approve on a queue is not a review.
The cheapest control is upstream of the AI: a clear, complete policy page. If your returns page doesn't say who pays return shipping, no tool can quote it. See writing store policies support can quote, and if you use ChatGPT by hand, using ChatGPT to answer customer emails safely.
Template: correcting a wrong AI reply
If a reply with an invented rule has already gone out, write again quickly and plainly. Decide first what you'll do for the customer: Moffatt is a reminder that a customer who relied on what you told them may have a claim.
Hi {{first_name}},
I need to correct my last email about your order {{order_number}}. I told you {{wrong_statement}}. That was wrong, and I'm sorry: our policy is {{correct_policy}}.
Because you relied on what I said, {{what_we_will_do_for_you}}.
If anything in my earlier email led you to do something you wouldn't otherwise have done, reply and tell me, and I'll sort it out with you.
{{your_name}}
Fill {{correct_policy}} by copying from your policy page, not from memory, and check {{order_number}} against the order. If the invented promise was large and you plan not to honour it, get advice before you send.
How Corresso keeps facts to your documents
Corresso drafts every reply from documents your store approved (returns policy, shipping page, FAQ), and each fact in a draft cites the document it came from. Rules written in code hold a draft that promises a refund or replacement without citing a document allowed to offer one, or that contains an order number, tracking number or date the AI wrote itself. They run again on the final text after your edits. Order facts are blanks you fill from the Shopify order with one click; the AI never sees order data. A person always reads the draft and presses Send. How one email goes from arrival to Send.