The following analysis is from Aaron C. Brown
, a risk analyst, statistician, and financial author. Brown teaches statistics at New York University and at the University of California at San Diego and writes regular columns for Bloomberg and Wilmott. He holds an M.B.A. in Finance and Statistics from the University of Chicago and an S.B. in Applied Mathematics from Harvard.
Note: To the best of your humble editor’s knowledge, Brown has no ties to the firearm industry or any pro-gun organizations.
The video above and the following were presented by Reason
Out of 27,900 research publications on gun laws, only 123 tested their effects rigorously.
A 2020 analysis by the RAND Corporation, a nonprofit research organization, parsed the results of 27,900 research publications on the effectiveness of gun control laws. From this vast body of work, the RAND authors found only 123 studies, or 0.4 percent, that tested the effects rigorously. Some of the other 27,777 studies may have been useful for non-empirical discussions, but many others were deeply flawed.
The 123 studies that met RAND’s criteria still had serious statistical defects, such as a lack of controls, too many parameters or hypotheses for the data, undisclosed data, erroneous data, misspecified models, and other problems.
The 123 papers identified by RAND tested 722 separate hypotheses about the impact of gun control policies for “statistical significance.” Peer-reviewed journals generally accept a result as statistically significant if it has a one-in-20 chance or less of being due to random chance. So if researchers run 100 tests on the relationship between two things that obviously have no connection to each other at all—say, milk consumption and car crashes—by pure chance, they can be expected to get five statistically significant results that are entirely coincidental, such as that milk drinkers get into more accidents.
In terms of the gun control studies deemed rigorous by RAND, this means that even if there were no relationship between gun laws and violence—much like the relationship between drinking milk and getting into car accidents—we’d nevertheless expect about five percent of the studies’ 722 tests, or 36 results, to show that gun regulations had a significant impact. But the actual papers found positive results for only 18 combinations of gun control measure and outcome (such as waiting periods and gun suicides). That’s not directly comparable to the 36 expected false positives, since some combinations had the support of multiple studies. But it’s not out of line with what we would expect if gun control measures made no difference.
Also concerning is the fact that there was only one negative result in which the researchers reported that a gun control measure seemed to lead to an increase in bad outcomes or more violence. Given the large number of studies done on this topic, there’s a high statistical likelihood that researchers would have come up with more such findings just by random chance. This indicates that researchers may have suppressed results that suggest gun control measures are not working as intended.
The reasons that we have no good causal evidence about the effectiveness of gun control are fundamental and unlikely to be overcome in the foreseeable future. The data on gun violence are simply too imprecise, and violent events too rare, for any researcher to separate the signal from the noise, or, in other words, to determine if changes in gun violence rates have anything to do with a particular policy.
One common research approach is to compare homicide rates in a state the years before and after gun control legislation was passed. But such legislation can take months or years to be fully implemented and enforced, if ever. Most modern gun control measures only affect a minority of gun sales, and new gun sales are a small proportion of all firearms owned. Very few of the people who would be prevented from buying guns by the legislation were going to kill anyone, and many of the people who were going to kill someone would do it anyway, with another weapon or by getting a gun some other way.
Therefore, the most optimistic projection of the first-year effect of most laws on gun homicides would be a reduction of a fraction of a percent. But gun homicide rates in a state change by an average of six percent in years with no legislative changes, based on FBI Uniform Crime Reporting (UCR) data
going back to 1990. As a statistician’s rule of thumb, this kind of before-and-after study can only pick up effects about three times the size
of the average year-to-year change, meaning that such studies can’t say anything about the impact of a gun law unless it leads to an 18 percent change or greater in the gun murder rate in a single year. That’s at least an order of magnitude larger than any likely effect of the legislation.
Despite popular media reporting, the incidence of gun violence is too rare for meaningful research to bear reliable statistics concerning legislation.