January 17, 2018
Northpointe, Inc. d/b/a equivant, issues the following in response to the article in the January 17, 2018 Science Advances online open access journal article titled “The accuracy, fairness, and limits of predicting recidivism”.
The cursory review of the article indicates serious errors related to misidentification of the COMPAS risk model and a lack of an external/independent validation sample. The author/s have made an erroneous specification of the COMPAS risk model as using “137 inputs”. This part of the study is highly misleading. It falsely asserts that 137 inputs are used in the COMPAS risk assessment. In fact, the vast number of these 137 are needs factors and are NOT used as predictors in the COMPAS risk assessment. The COMPAS risk assessment has six inputs only. Risk assessments and needs assessments are not to be viewed as one and the same.
We also question the testing procedure utilized with the brief 2-question risk scale. The scale was likely developed and then tested on the same large data sample. This is often known as giving only a limited internal validation. This commonly leads to “over-fitting” that falsely heightens the apparent predictive accuracy. This does not meet the test for external validation, thus adding further ambiguity to the findings.
The findings of “virtually equal predictive accuracy” in this study, instead of being a criticism of the COMPAS assessment, actually adds to a growing number of independent studies that have confirmed that COMPAS achieves good predictability and matches the increasingly accepted AUC standard of 0.70 for well-designed risk assessment tools used in criminal justice. Given the well-known advantages of ensemble methods such as crowd-sourcing, random forests, and other multiple method approaches, this finding confirms the valid performance of the COMPAS risk model across multiple geographical areas, independent State Department researchers, different university teams, and diverse methods.
equivant will be requesting the data used in the referenced study as well as peer review results documented by external sources. Our Research division will review the data upon receipt. It is critical to ensure the appropriate methods, fair comparisons, and conclusions were made as the article is given wide circulation online.