How to spot AI writing

You can discover ai’s hallmarks by comparing the writing of man and machine. To do this you need a baseline that is distinctive and familiar. The Economist turned to prose that we’re sure is human and that readers will recognise: our own. We designed a study to ask top llms—Openai’s Chatgpt, Anthropic’s Claude, Google’s Gemini and xai’s Grok—to write versions of our articles without consulting the web. (As a prompt, we gave them the ai-generated summaries that we have experimentally added to some of our articles.) This gave us a corpus of human and ai creations and we compared them across 55,940 sentences and 1.2m words. To make sure we were detecting ai quirks rather than our own, we also checked the ai texts against journalism from cnn, the New York Times and the Washington Post. Excerpts from hit novels published between 1950 and 2022 offered another test. Our findings are surprising. ai prose is distinguishable by word and punctuation choice as well as sentence and paragraph structure. But its hallmarks are not what you might expect, partly because its writing style has changed with software updates. That does not mean that llms are great writers: their prose lacks lucidity and elegance and is often formulaic. So those aspiring to be impressive (human) storytellers should avoid the following peculiarities in their own prose.

Much of this language could be described as what George Orwell called “pretentious diction”. He railed against writers who “dress up simple statements” with complicated words and jargon to sound clever. Such pontificating penmen, Orwell observed, also believe that “Latin or Greek words are grander than Saxon ones”. (Bots agree: more Latinate suffixes crop up in their writing than in human texts.)

Then look at punctuation. Many believe LLMs stuff their prose with em-dashes, but that is not true after the most recent updates. Today only Claude uses more em-dashes than human writers, with ChatGPT using markedly fewer than any other writer in our study. Humans rejoice — and start using dashes again.

A better way to spot AI-generated writing would be to look for texts without much punctuation at all. LLMS are very Joycean about it: they use fewer commas and semicolons than humans (and hardly any parentheses). They use less punctuation in part because they write longer sentences — “and” is their most overused word — and in part because they do not quote experts.

Really interesting article from the Economist (via Daring Fireball).