The official site of bestselling author Michael Shermer The official site of bestselling author Michael Shermer

Fake, Mistake, Replicate

published September 2006
A court of law may determine the
meaning of replication in science
magazine cover

In the rough-and-tumble world of science, disputes are usually settled in time, as a convergence of evidence accumulates in favor of one hypothesis over another. Until now.

On April 10 economist John R. Lott, Jr., formerly of the American Enterprise Institute, filed a defamation lawsuit against economist Steven D. Levitt of the University of Chicago and HarperCollins, the publisher of Levitt’s 2005 book, Freakonomics. At issue is what Levitt meant when he wrote that scholars could not “replicate” Lott’s results, referring to Lott’s 1998 book, More Guns, Less Crime. Lott employed a sophisticated statistical analysis on data from state-level variation in “carry and conceal” laws, finding that states that passed laws permitting citizens to carry concealed weapons saw statistically significant declines in robbery, rape and homicide compared with states that did not pass such laws.

As is typical with such politically charged research, considerable controversy followed publication of Lott’s book, with a flurry of conference presentations and journal papers, some of which replicated his results and some of which did not. For example, in a series of papers published in the Stanford Law Review, Lott and his critics debated the evidence.

In Freakonomics, Levitt proffered his own theory for the source of the 1990s crime decline — Roe v. Wade. According to Levitt, children born into impoverished and adverse environments are more likely to land in jail as adults. After Roe v. Wade, millions of poor single women had abortions instead of future potential criminals; 20 years later the set of potential offenders had shrunk, along with the crime rate. Levitt employed a comparative statistical analysis to show that the five states that legalized abortion at least two years before Roe v. Wade witnessed a crime decline earlier than the other 45 states. Further, those states with the highest abortion rates in the 1970s experienced the greatest fall in crime in the 1990s.

One factor that Levitt dismissed is Lott’s, in a single -passage in the middle of a 30-page chapter: “Lott’s admittedly intriguing hypothesis doesn’t seem to be true. When other scholars have tried to replicate his results, they found that right-to-carry laws simply don’t bring down crime.”

According to Lott’s legal complaint, “the term ‘replicate’ has an objective and factual meaning”: that other scholars “have analyzed the identical data that Lott analyzed and analyzed it the way Lott did in order to determine whether they can reach the same result.” When Levitt said that they could not, he was “alleging that Lott falsified his results.”

I asked Levitt what he meant by “replicate.” He replied: “I used the term in the same way that most scientists do — substantiate results.” Substantiate, not duplicate. Did he mean to imply that Lott falsified his results? “No, I did not.” In fact, others have accused Lott of falsifying his data, so I asked Lott why he is suing Levitt. “Having some virtually unheard-of people making allegations on the Internet is one thing,” Lott declared. “Having claims made in a book published by an economics professor and printed by a reputable book publisher, already with sales exceeding a million copies, is something entirely different. In addition, Levitt is well known, and his claims unfortunately carry some weight. I have had numerous people ask me after reading Freakonomics whether it is really true that others have been unable to replicate my research.”

“Replicate” is a verb that depends on the sentence’s object. “Replicate methodology” might capture Lott’s meaning, but “replicate results” means testing the conclusion of the methodology, in this case that having more guns in society results in less crime. The problem is that such analyses are so complicated that the failure to replicate more likely indicates modeling mistakes made during the original research or in the replication process rather than fakery.

Mr. Lott, tear down this legal wall and let us return to doing science without lawyers. Replicating results means testing hypotheses — not merely duplicating methodologies — and this central tenet of science can only flourish in an atmosphere of open peer review.

topics in this column: , , , ,

11 Comments to “Fake, Mistake, Replicate”

  1. J.P. McLaughlin Says:

    Mr. Lott’s complaint reminds me a bit of Bill Clinton’s whine: “That depends on what your definition of ‘is’ is.”

  2. William L Fell Says:

    Interesting article because I am a gun carrier. But, more interesting to me was J.P. McLaughlin’s little note referencing Bill Clinton. When Clinton said what he did about ‘is’ he was not whining. What he said was quite profound and an intense study in philosophy and linguistics. Most,if not all but a few scholars,think the statement was humorous, but that was due to much ignorance.

  3. Chad Eldien Says:

    While J.P. McLaughlin’s credentials in the area of linguistic commentary are beyond reproach, I’d like to recommend Steven Pinker’s writing as a good reference for Mr. McLaughlin and others that may be interested in the meaning of ‘is’; particularly, “The Stuff of Thought.”

  4. Mike Anzis Says:

    It doesn’t seem that Lott is trying to “do science” with lawyers but rather to protect his reputation by refuting an implication of falsification. However, regarding the social issue, are we dealing here with the classic confusion between correlation and causality? Lott’s analysis shows correlation between right-to-carry laws and reduced crime; Levitt’s between abortion and reduced crime. Both are plausable causal factors, but wouldn’t it take a good deal of controlled social experimentation (probably impossible) to prove causality of either?

  5. Dr.Q Says:

    sometimes somepeople confuse science.

  6. Ian WIlson Says:

    Much of the disputation in science seems to be the result of a lack of patience and a surplus of short term self interest. Assertiion that some method or conclusion may be iffy may be taken as an affront. But the goal is to find reliably observed conclusions… aka truth… aka observable. Careers are not what science is after. They may be handy in the short run. But in the end it is the confirming observations that build science.
    Patience, perspective beyond the moment, and keeping the long term goal in focus would probably remove adversarial actions from the pursuit of truth where the leading edge of science is cloudy. Hypothesis is science on the floor for discussion. Even the tried and tested may be found wanting.
    Let’s not take the joy out of science.
    Justice is not served well by the courts. Why would science be any different?

  7. John Says:

    When I was a grad student, the academic left was working hard to cast the scientific enterprise as a mere social construct – my adviser’s response was to urge us to bear in mind that *people* do science. Is there politics in science? Sure. Is there deception? Undoubtedly. Is this a problem? Not as much as you might think – erroneous results are often published (whether they be honest or not) but the self-correcting nature of science eventually ferrets these out.

    Lott is suffering from being on the short end of the scientific stick and he is taking desperate measures. If he could have won in the scientific ‘court of his peers’ he would – but he has already lost there. So he’s going to a civil court to be judged by non-experts.

    Perhaps this is a good thing: I find it highly unlikely that an American court would rule that criticizing a scientist’s results is grounds for a defamation of character suit. I hope the court has the judgment to counsel scientists (and other academics) to stick with their established systems of resolving these matters.

  8. Stu Says:

    Lott’s research is some of the most picked-apart, both loved and reviled depending on what side of the political spectrum one lies. Yet serious peer-review seems to back-up his figures, and they’re the ones, along with Kleck & Mustard’s, that are used by law enforcement. This is no different than any other scientific question; science eats its own young. If there are shortcomings to any hypothesis, they WILL be found out. Having examined Lott’s data myself, I would truly be impressed if the abortion hypothesis produces as uniform results.

  9. Michael A Thompson Says:

    I agree with the argument that we should try to keep the law out of science but I will have to say that guns DO reduce crime. To say otherwise is clearly unscientific. One has to simply look at the DOJ’s own statistics to show this statement true. Here is a more entertaining link to a report that refers to the DOJ’s report along with others:

  10. David R Tribble Says:

    I think Lott and Levitt both have valid, but different, points, and it’s a shame that they are fighting amongst themselves. Remember that both men have been vilified in the mainstream media for their views.

  11. J Morgan Says:

    Michael Shermer states above: >>According to Levitt, children born into impoverished and adverse environments are more likely to land in jail as adults.<< It has been quite awhile since I read Levitt’s FREAKONOMICS, but this claim does not jive with my recollection. I understood that the abortion of UNWANTED children, without reference or correlation to income, marital status, education, class, or anything else, was the most significant and measurable factor identified by Levitt as the probable cause for the decline in crime in the 1990s. Is Mr. Shermer assuming that those or the majority of those electing an abortion were (are) poor or live in adverse environments, or did Mr. Levitt actually show this? Please clarify.

This site uses Akismet to reduce spam. Learn how Akismet processes your comment data.