I’ve spent the last few months running a pet experiment. I’d take a draft written by GPT-4, refine it slightly, and drop it into our team Slack or paste it into a shared doc without telling anyone. Then I’d wait. The question I wanted to answer: can people actually tell when the words weren’t typed by a person?
The short answer is yes — but not the way you’d think. It’s not the grammar or the vocabulary that gives it away. It’s the rhythm. The lack of friction.
What gives AI writing away
Early on, I wrote a short memo for our sprint retro using an AI tool. Clean sentences. No typos. Perfectly structured. My co-founder read it and said, “This feels weirdly smooth. Did you use a template?” He didn’t suspect AI — but he noticed something off. That’s the pattern I started seeing again and again: AI text often feels too neat. Every sentence transitions cleanly into the next. Arguments hit their beats. There’s no throat-clearing, no backtracking, no “actually, I’m not sure about that part.” Real humans write in zigzags. We trail off. We rephrase. We admit confusion in the middle of a sentence.
I tested this more formally with a group of friends. I gave them five paragraphs — three AI-generated, two human-written — and asked them to pick which ones were fake. They were right about 70% of the time. The most consistent tell: AI writing doesn’t have “bad” moments. It never writes something that makes you think, “well that’s a weird way to put it.”
Now, I’m not saying AI can’t fool people. It absolutely can — especially for short, factual bits like “what is the capital of Mongolia” or product descriptions. But once you go beyond three sentences, the artificial smoothness becomes visible. At least to a careful reader.
The thing most people miss
Here’s what surprised me: the best AI writing I’ve seen — the stuff that actually passes — isn’t the most “natural.” It’s the stuff that includes deliberate mistakes. Awkward phrasing. A slightly off metaphor. The kind of thing a human writer would leave in because they were too tired to fix it, or because they thought it added personality.
My teammate once generated a slide title for a product launch: “Our new AI can draw pictures from text.” That’s fine. Boring but fine. But when I rewrote it as “Your cat’s next birthday card? Make it with AI” — which is slightly clunky, slightly too cute — people reacted like a human wrote it. Because a human probably would.
That’s the paradox. To sound human, you have to be willing to sound imperfect. And that’s hard for AI by design, because language models are trained to minimize perplexity — to choose the most probable next word. The most probable word is rarely the one that shows personality, doubt, or a slightly weird opinion.
While building Huanjian we keep bumping into this problem. Our AI presentation tool is great at generating slide drafts, but we spend twice as long editing them to add what we call “human wobble” — the pauses, the tangents, the little expressions that make the writing feel like it belongs to someone. It’s not about removing AI. It’s about making the output less perfect.
Honestly, I’m still not sure if “passing as human” is even the right goal. Maybe we should aim for something else: clarity, usefulness, a voice that readers actually like — whether it came from a person or a machine.