Wednesday, January 07, 2015

Winner's Curse

Before saying anything more about genetic studies, I want to talk about a problem that's quite common in scientific studies. It's colloquially (and somewhat inappropriately, since the analogy doesn't hold 100%) known as the "winner's curse."

In economics, the idea of "winner's curse" comes from the observation that the winner of an auction often overpays. Maybe you've noticed this phenomenon yourself, if you've participated in auctions? It's not just that people get into a frenzy of bidding and drive prices too high. That's not really the core idea here. The core idea is more like this: Suppose a laptop goes on auction and there are ten people in the room. All will bid on it. Every one of the ten persons has an estimate (a top bid) in mind for the true value of the laptop. The average of those ten estimates might be (let's say) $300. However, an average bid doesn't win an auction, does it? The highest bid does. And that might be considerably in excess of $300.

The "winner's curse" phenomenon comes into play in science a lot, because (for example) when a systematic review is done for various studies that found the "effect size" for a given type of medical treatment, the effect size often turns out to be larger in smaller studies. In theory, a treatment should give roughly the same effect size in all different sizes of study. If aspirin is a successful treatment for headaches 50% of the time, a study involving 30 people should show that, and a study involving 1000 people should show that. Instead, what you often see is (here I'm making numbers up) aspirin works 60% of the time in a small study and 50% of the time in a large study.

You can see how this might happen. Suppose four different teams of scientists (who don't talk to each other) decide to investigate the effectiveness of a vitamin milkshake as a hangover cure. And suppose that the "true effectiveness" of this "cure" (over a large enough number of studies and subjects) is actually about 20%. However, our four teams, working with very small study populations (hence, a high potential for statistical noise) find effectiveness of 11%, 13%, 22%, and 30%. The teams that got the lowest numbers probably won't publish their results. The team that got 30% probably will.

This sort of thing happens in science a lot and helps explain why, for example, some of the early twins studies on schizophrenia found high concordance rates (over 60%) for schizophrenia in identical twins versus fraternal twins, in relatively small studies, whereas later, much larger studies have found rates as low as 11%.

"Winner's curse" is a well-known ascertainment problem in science that affects studies of many kinds, including some of the recent large genome-wide associataion studies that have produced findings that failed to replicate when other teams decided to do similar sorts of investigations. I'll be talking more about that about soon.

Note: For a more technical discussion of "winner's curse" in genetic studies and what can be done about it, see Sebastian Zöllner and Jonathan K. Pritchard, "Overcoming the Winner’s Curse: Estimating Penetrance Parameters from Case-Control Data," Am J Hum Genet. Apr 2007; 80(4): 605–615.

I'm working on a no-nonsense, "skewer the sacred cows" book about mental illness. If you'd like to follow the progress of the book, sign up for the mailing list, and be sure to check back here regularly. Thanks!