- Capital Gains
- Posts
- Pangram Is An Oracle Until There's a Good Counterexample
Pangram Is An Oracle Until There's a Good Counterexample
Statistical models turn out to be very good at detecting the outputs of statistical models
Know someone who might like Capital Gains? Use the referral program to gain access to my database of book reviews (1), an invite to the Capital Gains Discord (2), stickers (10), and a mug (25). Scroll to the bottom of the email version of this edition or subscribe to get your referral link!
Emily Bender has an article on why we should expect AI text detectors like Pangram to have a high false-positive rate. However, there's a near absence of known false-positives. There's substantial motivation to find them: students don't like getting flunked because their teachers accused them of cheating, whether or not they cheated. There's also a monetary incentive: there was a cash bounty for finding a text of more than 750 words, verifiably published before widespread LLM use that gets a high made-by-AI score from Pangram. And there was a winner! It's a pretty funny poem from 2006, and it bears no resemblance to any of the AI-generated student essays, minor literary prize-winning stories, podcast scripts, political speeches, etc. that people are complaining about. (Bender's article also makes some culture war arguments about why bad people think AI detection is possible. Given that it clearly is, her speculation is irrelevant, like trying to figure out whether it's the Illuminati or the Trilateral Commission who are spreading rumors that the sky is blue. You don't have to ask! Just look up!)
So, if Bender's reasoning tells us why lots of false positives should exist, and they don't, the purpose of her article is to outline misconceptions people might have about AI that lead them to emphatically believe and evangelize completely incorrect things about AI.
An important thing to note upfront is that Bender is not a layperson. She actually has a great deal of experience in the business of using computers to analyze text. However, it's also worth noting that this experience is with a different model of the best way to do this. If you were asking the question of how to teach a computer to write a poem a few decades ago, the natural thing to do would be to come up with concepts and rules. You might figure out syllable counts and a rhyming dictionary in order to generate nonsense that would sound like a poem to someone who didn't speak the language, layer in parts of speech and sentence structure, come up with abstractions that describe these at a high level so you could even generate your own, etc. The result you'd get, or at least hope for, is some poem-bot for which every choice can be explained by successively asking what makes something a poem at higher and higher levels of abstraction.
An uglier, less appealing strategy is to take a huge amount of poetry, slice the words up into tokens (maybe you break "parts" into "part" and "s," for example), and then start asking the model to guess the next word of some partly-completed poem from your training set, reinforcing whichever parts of the model contribute the most to a good guess and suppressing the ones that don't. Your a priori view might be this technique will require too much data to be feasible, or will lead to ugly results, or something like that. Which is true, for some values of "too much," or "ugly." But it also turns out to be true that it is a technique that can produce something recognizably like a poem (or an essay, or a zero-day exploit, or a patch for one, or many, many other things). This is a surprising fact about the world, and the last decade-plus has been a continuous process of discovering the astonishing capabilities of the "build a model blind to anything but statistical patterns, reward it for doing things like what you want it to do, and wait for the capabilities you care about to emerge." But: it works. You can see that it works. You can use free LLMs that, from the perspective of ten years ago, seem like the kind of technology that might have come along in the 2050s, if it were even possible at all. It's just an astonishing fact about the world that sufficiently powerful statistical models can do things that look like the outputs of human creativity.
It's annoying that this is true, because it means that we can replicate some outputs of human creativity without getting much insight into how humans create. Oh well! The pulley replicated the effects of having lots of muscles a long time before it told us anything about how muscles work.1 But as grating as it is to anyone who appreciates beautiful equations that cause a chaotic phenomenon to snap into perfect focus: they work. They generalize. They do things they weren't explicitly trained to do, and show capabilities beyond those of the people who created the training data in the first place, albeit quite unevenly across domains.
And this approach generalizes in another way, in that you can use it to do things other than generate text. You can use it to predict the answer to a question about text. You can feed a model a large number of pairs whose first member is a text and whose second is a true/false value, and ask it to predict true/false values for future chunks of text. The model doesn't have to know what those labels mean (in fact, we can go on a long and pointless tangent about what, if anything, it means for them to "know"), but it can happily classify them. Maybe they are or aren't French, they are or aren't above some Flesch-Kinkaid reading level score, they did or didn't receive a passing grade or a literary prize or a nasty personal attack in the comments section.
In Pangram's case, the labels are whether it is true or false that the resulting text is the output of an LLM. Now, we just got done talking about how counterintuitive it is that LLMs can do things humans do, like write a limerick or a love story, and we're suddenly talking about how they do things differently. But they do! Anyone with experience reading LLM-generated prose—anyone who regularly uses chatbots, or started to pick up on "It's not X, it's Y", the rule of three, and "quietly" working their way into a surprising number of documents, is aware that LLMs have verbal tells. Some of these tells are human-readable, but some will be more subtle, perhaps involving the distribution of sentence lengths, preferred adjectives, how they handle transitions to new topics, or something else. There could be a long list of them, not a single one of which is telling on its own (speakers from Jesus Christ to JFK have used contrastive parallelism as a rhetorical technique. It's not just an output, it's an input).But put them together and you can make a definitive case, in the same way that one personal detail (e.g. hometown, school, the name of a single one of your friends) doesn’t deanonymize someone, but a surprisingly small number of them can uniquely identify someone.
If you were to mechanistically interpret such a model, it might look something like this: First, count up the frequency of LLM-y sentence construction, paragraph structure, word choice, narrative arc, etc. Next, compare that to other constructions, structures, words, arcs, etc. that LLMs, for whatever reason, don't reproduce as well. If the ratio's high, it's probably from an LLM. The techniques LLMs use work, just in small doses, and LLMs are typically evaluated in small increments. So one possible reason the text sounds the way it does is that It's easier to ask a thousand people to each read 500-word extracts and rate their literary quality, and much harder to have someone read a 500,000-word LLM-generated doorstopper of a novel and give coherent thoughts. So it's as if you trained a model to produce good recipes by having it rate individual bites, so when you asked it to produce an ideal recipe, it suggested Salted Simple Syrup Soup, topped with a pat of butter. Delish!
It's hard to read prose and see a statistical distribution. Unless you're an LLM, in which case that's all you see. It's like two people standing next to each other at an art gallery, one appreciating a masterpiece and the other diligencing an asset class. LLMs are sentence-completers because they are pattern-detectors. Pangram's approach should pick up on patterns like this, just as you could probably train a model to find tells about someone's first language—it might not spot Joseph Conrad, but it might notice that people who speak a language without articles, like Russian or Japanese, have a slightly higher error rate when it comes to using them correctly in English, and might further disambiguate based on subtle differences in vocabulary, echoes of the other language's word order, or still subtler tells. We know that human-written prose has some kind of statistical distribution to it because you can train a model that produces that prose, even on topics no one has written about before. And we know that doing that requires the ability to spot patterns, even patterns no human would notice, even, possibly, patterns we couldn't fully understand.
So, Bender has a concern: Pangram is trained on some documents, whose provenance we don't know perfectly well. It spits out some answers. But maybe it's trained on particular texts, and misses some nuances. Maybe those nuances end up overrepresented in LLMs—perhaps because so many answers were being rated by whoever was the most affordable English-speaking labor globally. It might be that there's some overlap between what Pangram flags as AI and what a college-educated person in Nigeria would consider idiomatic formal English.
It would mean that that writing would be a great place to look for false positives. Somebody did, checking 187 books published by African writers both before and after AI using two AI detectors. Pangram and GPTZero both said that 0.00% of the 7.4m words written before AI were produced by AI. The Blue Fighter, which has been flagged as AI, uses a notably different vocabulary than other books, and LLM readers will recognize them—it starts a sentence with the word "Not" 3.9x more often than the previous record-holder. So, what we learned from this is: it was an interesting hypothesis that, upon investigation, was falsified.
Part of her complaint is to say that some of Pangram's marketing (accurately) talks about the low false-negative rate from one study, and (again, accurately) talks about the lower false-positive rate from another, without disclosing the two together. Then she concedes that they do have a more formal tech report that discloses both, at which point her objection changes to the fact that maybe it performs worse out-of-sample.2 But, the stated false positive rate in the report is 0.0041%. In the zero-false-positive example, they looked at 25,996 essays. The expected number of false positives is 1.07, and there's a 34% change of having zero. So, in fact, that marketing material makes a statistical claim that is no-asterisks-required statistically indistinguishable from what they say in research reports addressed to the ML community. That's just not the case for a large fraction of the ads that use a percentage sign for something other than an interest rate.
Then she argues that they present the results in a way that's misleadingly precise, declaring some texts either 100% human or 100% not. That's more a UI critique than a problem with the product, and the critique appears to be that in the UI, the part that says "We believe that this entire text is human-written," (emphasis added) should be in a bigger font, and maybe the text should be black rather than gray. This is just a natural way to talk about statistical models! Gmail believes that some emails are spam and some are not; banks believe that some charges are fraudulent and some are not, so it lets some through completely and blocks others. In Pangram's case, a false positive (i.e. tagging something as AI-generated) is a worse experience than a false negative. It's a bigger deal for a student to be falsely accused of academic dishonesty than for one to skate by with a little cheating, even though the cheating is bad. And someone working on a product that can have people's fates in its hands will be biased towards mercy, most of the time. This choice is up to them, because they're the ones setting the cutoff. They could catch more AI cheating at the cost of falsely accusing a small fraction of that number of people—and they're 83x as willing to let AI go undetected than to label it incorrectly.
One thing Bender doesn't seem to grasp is the fundamental question of why we even care about LLM-generated text. In one sense, I don't have any problem with it at all: I consume enormous volumes of it every day, in the form of answers to questions, summaries of documents that help me decide whether or not to read them, editing suggestions for text, etc.3 A document of some length is, implicitly, a claim that the person who wrote it put in effort commensurate with that length. In the pre-LLM days, it was possible to format something in LaTeX and pretend that it was an academic article, even if it wasn't, but using LaTeX was a pain. It was a small fraction of the cost of doing research but a large fraction of the cost of making nonsense look impressive, so a nicely-formatted PDF was either slight evidence that the content was worth a skim or evidence that someone had put effort into a joke.
LLMs break this relationship. They make it trivial to produce something that looks like it took a lot of effort. And it's rude to present it in a way that implies that it required effort. (Especially if it's supposed to be emotional effort, but that's a rarer case.) The LLM-generated text might be good, but the writer is, implicitly, saying: reading this is worth your time, but writing it wasn't worth mine. It's a lot worse if they're doing this in the context of some kind of competition, where they're depriving human beings of recognition for their creative efforts while taking credit for the hard work of AI researchers, GPU designers, prolific Reddit users, private credit analysts, and all the other unsung heroes whose contributions to the canon of award-winning short stories have been heretofore unrecognized.
There are people who put lots of work into getting LLMs to produce the prose they want, and who at least claim to work just as hard to produce a thousand words that get flagged as 100% AI as someone else who worked to write text manually. Some of them may be completely truthful! But it's a bit like telling someone that, after you're done with work, you wait until dark, put on a superhero outfit, and spend the entire night preventing crimes, but that you can never reveal your secret identity. Okay! Maybe that's true, but it's a very easy thing to say, impossible by design to verify, and would make the person claiming it enviable if it were true. It's thus a weak signal of being Batman and a stronger signal of having a tendency to fib.4
That's fine as a standalone claim. It's just quirky. But if you decide that your status as a master crimestopper entails being treated respectfully by everyone you meet, and if state funding commensurate with Bruce Wayne's lifestyle also seems sensible, we're going to want to see some evidence. Not a comprehensive list of every villain you've ever caught, of course. Just, as a helpful starter, one example. And if you insist that the standard of proof for being Batman is zero evidence of ever preventing a single crime, well, we won't take you seriously because it just isn't a serious claim. It's not something anyone who's thought about it would believe. And when you really ask them, the skeptics don't believe it, either.
The people saying they don’t believe Pangram works don’t act like they believe what they’re saying. They don’t propose some sort of challenge, where each side puts up money and bets on whether some newly-written text is going to be flagged as AI-generated even though a human just wrote it. Personally, if the stakes are high enough, I’d be happy to do this—we can buy copies of today’s WSJ/FT/NYT, turn to the business section, pick an article at random, have me write a thousand-word essay about it, ask an LLM to do the same, and bet on which one gets flagged. And we can do it again and again and again until whoever’s betting against Pangram has either lost all of their money or conceded my point. But that won’t happen. Pangram’s critics follow the cognitive pattern of the earliest LLMs, the ones it was fair to deride as "stochastic parrots": they can put one word after another in a way that makes grammatical sense and roughly sticks to a given topic. But whatever the exact process is that produces that output, it's a mistake to call it "thinking."
We've mentioned Pangram and AI-generated writing a few times in The Diff, but the best pieces to read look at the broader question of how AI breaks the connection between effort and output, what we gain from that, and what we lose.
It's important to think about what we'll lose with AI, even if the reason we lose those things is that we got something more valuable.
In 2023, we looked at which writing jobs are at risk from generative AI ($).
Text doesn't perfectly reflect reality, but it's such a handy tool for describing it that LLMs, which work well because we've implicitly encoded such a world model, actually encourage us to write down more of what we know.
Share Capital Gains
Subscribed readers can participate in our referral program! If you're not already subscribed, click the button below and we'll email you your link; if you are already subscribed, you can find your referral link in the email version of this edition.

Join the discussion!
1 It turns out that your eye and finger muscles use pulleys, but the pulley wasn't discovered by dissecting a corpse, at least as far as we know.
2 It's interesting to compare the output of LLM skeptics today to the outputs of LLMs in the GPT-3-or-so era. They were popping out one token at a time, and sometimes they would meander their way into realizing they'd been wrong about something, but would have no choice but to diligently plough ahead. With the release of o1, they could reason through multiple possibilities, prune the branches of thought that didn't lead anywhere, and produce a tight output. So it was around the end of 2024 that publicly-available LLMs achieved superhuman performance at the task of arguing that LLMs and other AI tools don't work.
3 On the last: a good way to use LLMs, especially for me, is to ask for help on: structure, weak points, unsupported assertions, etc. If you write in Google Docs, or paste a draft into it, you're also using LLMs for grammar and spelling. For that second category, it's generally fine to just accept edits. (For whatever reason, Google seems to over-rely on LLMs rather than older, more typo-based methods. So they're great at spotting incorrect homonyms, missing or doubled words, etc. But they've degraded in their ability to spot transposed characters, since those can lead to hard-to-tokenize words. It would be nice if they used the old method as a fallback when they don't have a good LLM-based suggestion.) On structure, substantiating claims, etc., what LLMs basically do is cope with writers' blind spots. Articles in Capital Gains and The Diff are generally written in a straight shot, which naturally leads to bouncing from one strong argument to another while not quite getting around to admitting weak ones. One of the strengths of LLMs is that they can't help but be average—you can use them as a nice oracle for "is this rude?" or "how would one ask that?" or "do professionals in this field use the technical term or colloquial one for a given term?" So they do a nice job of implicitly producing comments that either complain about or argue with a piece. But I don't use them to write the text itself. That's my job! If I wanted readers to get an LLM output, I'd give them the prompt.
4 This is less of a problem in verifiable domains. There isn't much demand for a "Pangram for code" right now, because outside of academic settings and job interviews, it doesn't really matter how the code was generated as long as it does what it's supposed to do. (That includes not just technical benchmarks and passed tests but a more holistic "works in a way that the person responsible for using or maintaining it understands." If you're using an LLM to analyze a dataset, it's good if it produces some readable Pandas code that you can check and understand, and it would probably be a poor tradeoff to just give you straight assembly even if that were slightly faster and used less memory. Part of the intended final output is an understanding of roughly how the results were obtained, and the limitations that entails.) Even so, getting the right output can involve hand-written code. Code is an efficient notation for thought, and there's something about writing down thoughts that seems to produce more of them. And LLMs' taste in code is presumably similar to its taste in literature: always at the high end of average, but limited by what it's trained on. To the extent that software is a competitive advantage in a given industry, the code most useful for training isn't part of broad models, and diffuses in proportion to how quickly it loses economic value and vice-versa. In those industries, there's still room for a mix of fine-tuning and hand-written code alongside the standard coding agents.
Reply