Readers can't tell the difference between short stories generated by ChatGPT and those written by humans. They even rate the AI-generated texts higher, but only as long as they don't know a machine wrote them.
In three experiments with more than 2,500 total participants, test subjects did no better than chance at telling human-written and ChatGPT-generated fictional short stories apart.
In the first experiment, each of the 1,682 participants read one of six short stories, each about 1,000 words long. Three came from well-known literary magazines and short story collections. The other three were generated using ChatGPT 4.0, with prompts based on the theme, style, and narrative perspective of the human originals.
Half the participants were told the story was written by a human. The other half were told it came from ChatGPT. That information was accurate for only half the participants in each group, according to researchers Sydney Sears and Deena Skolnick Weisberg in their study published in the journal Judgment and Decision Making.
ChatGPT's stories were rated significantly higher than the human-written texts on both perceived quality and immersion. For quality, the mean score for AI stories was 1.54 compared to 0.97 for human stories on a scale from minus 3 to plus 3. For immersion, the gap was 1.42 versus 1.00.
Participants' own attitudes toward AI also shaped their ratings. Regardless of who actually wrote the story, participants gave higher scores when told a human was the author.
Participants with a positive attitude toward AI generally gave higher ratings across the board. When they were also told the story came from ChatGPT, their scores rose even further. Among AI-skeptical participants, this effect flipped. An earlier study on AI-generated poems found a similar bias.
In two more experiments with 905 total participants, the researchers made the task harder. Each person read both a human-written and an AI-generated story, then had to figure out which was which. Even with a direct comparison, participants performed no better than chance.
Self-reported experience with AI systems correlated positively with the ability to correctly identify the stories' origins. Self-reported experience with fiction, on the other hand, didn't help participants tell them apart.
Higher ratings don't necessarily mean better writing
AI-generated texts tend to be smoother, easier to read, and more emotionally upbeat than human-written texts. According to the authors, these traits could explain the higher ratings without the AI stories actually being better in a literary sense. People tend to prefer material that's easier to process.
High-quality literary fiction, by contrast, is often intentionally hard to access and pushes readers to work for meaning. A story can be high quality but not very engaging, and vice versa, the researchers suggest.
The short story format also likely works in AI's favor. Telling a coherent story in 1,000 words is a very different challenge than doing so across hundreds of pages. Still, the researchers' conclusion is clear: AI can generate creative works people perceive as at least on par with human work, yet people don't believe AI is capable of that.
All data and materials from the study are freely available on the Open Science Framework.
A study from last October by Stony Brook University and Columbia Law School shows that reader expertise matters only up to a point. With simple prompts, professional readers clearly preferred the human-written texts. But when models were specifically trained on individual authors' styles, the experts preferred the AI-generated texts eight times more often for style imitation and twice as often for writing quality.
Read on for the full picture.
Subscribe for hype-free coverage.
- Access to all THE DECODER articles.
- Read without distractions – no Google ads.
- Access to comments and community discussions.
- Weekly AI newsletter.
- 6 times a year: “AI Radar” – deep dives on key AI topics.
- Up to 25 % off on KI Pro online events.
- Access to our full ten-year archive.
- Get the latest AI news from The Decoder.