From Washington Post's Wonk Blog:
It's an impressive looking graph. Too bad that the actual numbers are trickier to interpret. The graph was put together by the Enliven Project, and they provide a discussion of where the data came from.
Much of it appears to be a guesstimate. For instance:
For those of you who have asked, here is the background on the stats we used:The link to the 5-25% figures is to British data. The link to the 9% prosecuted is to US data, and as far as I can tell from the latter link, the 9% seems to be the percentage likelihood of going to prison for rape in the last year the statistics give us, not the percentage of a prosecution.
- Some reports suggest that only 5-25% of rapes are reported to authorities. Other suggest that close to half are reported. We assumed 10%, which is dramatic, but possible.
- Of the rapes that are reported, approximately 9 are prosecuted.
Amanda Marcotte mentions a few other problems with the underlying data, especially with the numbers of little men in it, because much of rape is caused by a small number of men raping repeatedly. She also points out that the graph overstates the number of unreported rapes in the United States.
I don't think that the creators of this graph intended anything more than to start a conversation about the magnitude of unreported rape, the magnitude of cases where someone is falsely accused and so on. It's pretty certain that a lot of rapes go unreported, for instance.
But they did choose to pick fairly extreme estimates for each number, and now they are going to be taken to task for the kind of statistical accuracy problems which even professional research might ultimately be unable to avoid, except by giving extremely wide safety margins for the various items. But that's how it goes.
The link (pdf) the Enliven Project gives to false rape accusations is very useful, however, because it puts together a larger number of studies concerning those numbers and thus places the favorite of the MRA sites into a more proper perspective. I quote from the pdf:
In the most frequently cited study on this topic, Professor Eugene Kanin (1994) reported that 41% of the 109 sexual assault reports made to one midwestern police agency were deemed to be false over a nine-year time period. However, the determination that the charges were false was made solely by the detectives; this evaluation was not reviewed substantively by the researcher or anyone else. As Lisak (2007) describes in an article published in the Sexual Assault Report:
Kanin describes no effort to systemize his own ‘evaluation’ of the police reports—for example, by listing details or facts that he used to evaluate the criteria used by the police to draw their conclusions. Nor does Kanin describe any effort to compare his evaluation of those reports to that of a second, independent research— providing a ‘reliability’ analysis. This violates a cardinal rule of science, a rule designed to ensure that observations are not simply the reflection of the bias of the observer (p. 2).2
In other words, there is no way to explore whether the classification of these cases as false was simply made as a result of the detectives’ own perceptions and biases, without any real investigation being conducted.This concern is compounded by the fact that the practice of this particular police department was to make a “serious offer to polygraph” all rape complainants and suspects (Kanin, 1994, p. 82). In fact, this practice “has been rejected and, in many cases, outlawed because of its intimidating impact on victims” (Lisak, 2007, p. 6).The reason is because many victims will recant when faced with apparent skepticism on the part of the investigator and the intimidating prospect of having to take a polygraph examination.Yet such a recantation does not necessarily mean that the original report was false.
In reality, there is no way that an investigator can make an appropriate determination about the legitimacy of a sexual assault report when no real investigation has been conducted—and the victim is intimidated by the department’s policy of making a “serious offer to polygraph” all rape complainants.As we will discuss at length below, the determination that a report is false can only be made on the basis of findings from a thorough, evidence-based investigation.
As a result of these and other serious problems with the “research,” Kanin’s (1994) article can be considered “a provocative opinion piece, but it is not a scientific study of the issue of false reporting of rape. It certainly should never be used to assert a scientific foundation for the frequency of false allegations” (Lisak, 2007, p. 1) In contrast, when more methodologically rigorous research has been conducted, estimates for the percentage of false reports begin to converge around 2-8%.
For example, in a multi-site study of eight U.S. communities involved in the “Making a Difference” (or “MAD”) Project, data were collected by law enforcement agencies for all sexual assault reports received in an 18- 24 month period. Of the 2,059 cases that were included in the study, 140 (7%) were classified as false.This is particularly note- worthy because a number of measures were taken to protect the reliability and validity of the research. First, all participating law enforcement agencies were provided training and technical assistance in an ongoing way to ensure that they were applying consistent definitions for a false report. In addition, a random sample of cases was checked for data entry errors. More information on the MAD Project is available at http://www.evawintl.org.
To date, the MAD study is the only research conducted in the U.S. to evaluate the percentage of false reports made to law enforcement.The remaining evidence is therefore based on research conducted outside the U.S., but it all converges within the same range of 2-8%.
For example, Clark and Lewis (1977) examined case files for all 116 rapes investigated by the Toronto Metropolitan Police Department in 1970. As a result, they concluded that seven cases (6%) involved false reports made by victims.There were also five other reports made by someone other than the victim that were deemed by these
researchers to be false (e.g., a relative or boyfriend).
Grace, Lloyd, and Smith (1992) conducted a similar analysis of the evidence in all 348 rape cases reported to police in England and Wales during the first three months of 1985. After reviewing the case files, reports from forensic examiners, and the statements of victims and suspects, 8.3% were determined to constitute false allegations.This study was sponsored by the British Home Office.
A similar study was then again sponsored by the Home Office in 1996 (Harris & Grace, 1999).This time, the case files of 483 rape cases were examined, and supplemented with information from a limited number of interviews with sexual assault victims and criminal justice personnel. However, the determination that a report was false was made solely by the police. It is therefore not surprising that the estimate for false allegations (10.9%) was higher than those in other studies with a methodology designed to systematically evaluate these classifications.
The largest and most rigorous study that is currently available in this area is the third one commissioned by the British Home Office (Kelly, Lovett, & Regan, 2005).The analysis was based on the 2,643 sexual assault cases (where the outcome was known) that were reported to British police over a 15-year period of time. Of these, 8% were classified by the police department as false reports.Yet the researchers noted that some of these classifications were based simply on the personal judgments of the police investigators, based on the victim’s mental illness, inconsistent statements, drinking or drug use.These classifications were thus made in violation of the explicit policies of their own police agencies.The researchers therefore supplemented the information contained in the police files by collecting many different types of additional data, including: reports from forensic examiners, questionnaires completed by police investigators, interviews with victims and victim service providers, and content analyses of the statements made by victims and witnesses.They then proceeded to evaluate each case using the official criteria for establishing a false allegation, which was that there must be either “a clear and credible admission by the complainant” or “strong evidential grounds” (Kelly, Lovett, & Regan, 2005). On the basis of this analysis, the percentage of false reports dropped to 2.5%.
Finally, another large-scale study was conducted in Australia, with the 850 rapes reported to the Victoria police between 2000 and 2003 (Heenan & Murray, 2006). Using both quantitative and qualitative methods, the researchers examined 812 cases with sufficient information to make an appropriate determination, and found that only 2.1% of these were classified as false reports. All of these complainants were then charged or threatened with charges for filing a false police report.
Of course, in reality, no one knows—and in fact no one can possibly know—exactly how many sexual assault reports are false. However, estimates narrow to the range of 2-8% when they are based on more rigorous research of case classifications using specific criteria and incorporating various protections of the reliability and validity of the research—so the “study” does not simply codify the opinion of one detective who may believe a variety of myths regarding false reporting.
Note that all the scientifically better estimates of false rape accusations are less than ten percent of reported rapes, the one US study giving us a figure of 7%. The 2% figure is the extreme lower bound of these estimates, and probably shouldn't have been picked for the graph, just as ten percent reporting rates shouldn't have been picked for the graph. On the other hand, Marcotte discusses additional problems with the way the concept of a false accusation is treated.
It's pretty sad if the problems behind this graph are going to result in the assumption that the best possible estimates wouldn't give us a fairly sad-looking graph, too.