Foomax

Beware the Man of Many Studies

September 2026 · LessOnline

Two waves of study-skepticism in rationalist writing, fifteen years apart — and why the second one matters more. Part 2 of a series drawn from 37.6 million words across 34 rationalist blogs.

I. Fifty million dollars and nothing to show for it

Between 1974 and 1982 the United States government randomly gave 7,700 people medical care that was either free or not free, for up to five years, and measured who ended up healthier. The RAND Health Insurance Experiment cost fifty million dollars. The free-care group consumed 30 to 40 percent more medicine. They were not, in any measurable way, healthier.

Robin Hanson — an economist who trained in physics and helped invent prediction markets — wrote about RAND in 2008 and petitioned the government to run it again, bigger. Nothing happened. So he did what a blogger can do instead: he went through the medical literature one review at a time and reported, in three-hundred-word posts with the deadpan of a coroner, what it showed. Supplements Kill. Beware Active Placebos. Peer Review Is Random. Skip Cancer Screens: screening “consistently leads to more cancer detection and more cancer treatment, it consistently doesn’t lead to lower mortality.”

Thirteen years later he could finally post the experiment he’d petitioned for — in India, not America: 52,292 people in 435 villages randomised into hospital insurance, “very few statistically significant impacts … on health.” He is not a man who gloats. He just updates.

II. Two waves

Count everything — every post by every author on the guest list of LessOnline, the rationalist blogosphere’s annual festival, sorted into themes by a topic model — and one theme turns out to be not a subject but a practice: writing whose question is how much should I believe this study? Around 900 posts, spread across seven blogs.

It comes in two waves. The first is Hanson’s: 85 posts in 2007, then 60, 78, 68, 53 — nearly all his, nearly all about medicine. Then a trough: a dozen a year or fewer from 2014 to 2020, three in 2019. Then a second wave from different hands: 56, 51, 62, 52, 50 across 2021–2025. (One caution: the blog that carried the theme through the 2010s, Slate Star Codex, isn’t in the corpus — only its 2021 successor — so part of the trough is a missing bookshelf.)

III. The hinge

In May 2015 the physicist Steve Hsu posted 191 words on the first big replication effort in psychology — 100 findings, 39 reproduced. What he added was a diagnosis of the readers rather than the studies. Researchers “might pay lip service” to the result, he wrote, but “they typically have not updated their posteriors to reflect the low reliability of research results, even in the top journals.”

That is the hinge between the waves. The first asked does the medicine work? — a question about the world. The second asks what should a reader do with a literature this unreliable? — a question about the reader, which turns out to be much harder.

IV. The second wave argues with itself

Its founding document is a rebuttal. In June 2023 the pseudonymous Cremieux published Beware the Man of Many Studies — 4,000 words that begin by praising Scott Alexander’s famous 2014 advice (don’t trust one study; read the whole literature) and then turn on it. The subtitle is the argument: “Low-quality literatures mean meta-analyses are frequently worse than single studies.” A hundred bad studies averaged together do not become one good study. They become a confident wrong answer with a narrow confidence interval.

Scott, meanwhile, was policing the other flank. Against That Poverty And Infant EEGs Study (2022) dismantles a paper the Times had celebrated as “basically a typical null-result-having study.” But All Medications Are Insignificant In The Eyes Of God And Traditional Effect Size Criteria (2023) argues that the fashionable “effect size 0.30 is meaningless” critique of antidepressants would also condemn drugs that obviously work — so “downweight all claims about ‘this drug has a meaningless effect size’ compared to your other sources of evidence, like your clinical experience.”

Notice what happened. In the first wave the skeptic was one economist reading against the grain. In the second, skepticism is the default — so the interesting posts are the ones policing skepticism itself. Cremieux: the meta-analysis you were told to trust can be worse than the study you were told to distrust. Scott: the heuristic you were told to apply will tell you Ambien doesn’t work. Skepticism has become a tool that needs its own calibration.

V. Why this is the pressing one

The community that built this discipline now spends most of its words on artificial intelligence, and the AI argument is conducted almost entirely in this theme’s currency. Timelines are forecasts with error bars. Capabilities are benchmark scores published by the labs being benchmarked. Safety claims are evaluation results with sample sizes and funders. Every failure mode this theme catalogued in medicine is present in the AI evidence base, at higher stakes and faster tempo.

Hanson petitioned for an experiment in 2008 and got one in 2021. That is how long good evidence on a hard question takes. The AI question does not have thirteen years. So a skill learned one Cochrane review at a time, by people who did not know what they were training for, is now the load-bearing skill of the community’s central argument. And Hsu’s line is the whole theme in a sentence: the hard part was never showing the studies were unreliable. The hard part is that people “have not updated their posteriors” — and updating is the one thing this community claims, above all else, to know how to do.

LLM-to-read