Response to ProPublica: Demonstrating accuracy equity and predictive parity
The Northpointe Suite • An equivant product
The website ProPublica recently published a story that focused on the scientific validity of COMPAS, raising questions about racial bias. As a result of the article and the subsequent national attention that it garnered, Northpointe launched an in-depth analysis of the data samples used by ProPublica. Drawing from the results of our analysis of ProPublica’s data, Northpointe unequivocally rejects the ProPublica conclusion of racial bias in the COMPAS risk scales.
Predictive modeling is a specialized field within statistics and the appropriate use and interpretation of valid predictive models require a solid understanding of the techniques and methodological nuances common to this type of work. Our detailed review of how ProPublica conducted their analysis revealed several statistical and technical errors such as misspecified regression models, mis-defined classification terms and measures of discrimination, the incorrect interpretation and use of model errors, and more. These errors led to a false conclusion of racial bias; we do not believe the conclusions drawn are in fact supported by the data samples used in any way.
“COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity” is our formal response to the ProPublica article. This technical report presents the technical results after a careful review of the statistical methodology that was used by ProPublica as well as the results of our own thorough analysis of the same data. An explanation of the errors identified by our review is included as Appendix B, and we also provide a thorough technical discussion of three key findings of our methodologically correct analysis of the data:
- ProPublica focused on classification statistics that did not take into account the different base rates of recidivism for blacks and whites. Their use of these statistics resulted in false assertions in their article that were repeated subsequently in interviews and in articles in the national media.
- When the correct classification statistics are used, the data does not substantiate the ProPublica claim of racial bias towards blacks.
- The interpretation of the results in the samples used by ProPublica demonstrate that the General Recidivism Risk Scale (GRRS) and Violent Recidivism Risk Scale (VRRS) are equally accurate for blacks and whites.
Please CLICK HERE to download our report which includes a detailed and technical discussion of the findings.
An article has been pubished in the Federal Probation Journal by Flores, Bechtel, & Lowekamp (September 2016) that refutes Probublica’s claims of racial bias against COMPAS. False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And It’s Biased Against Blacks.” Click here to download the article from the Federal Probation Journal web site.
Also, please read the National Institute of Corrections (NIC) publication “Myths and Facts: Using Risk and Need Assessments to Enhance Outcomes and Reduce Disparities in the Criminal Justice System” for more information.