What is validation?

Validation is the degree to which an instrument (a survey, for example) measures what it is intended to measure. In the justice system, validity is invaluable because it means you can trust what a tool is telling you.

Here’s an example. If a coworker asks you how warm it is outside, how would you determine the answer? Would you look out a window? Or would you use a thermometer? Looking out the window might help you approximate the correct answer (e.g., if it’s a sunny July day in the Northern Hemisphere, looking out the window may tell you it’s warm, and that may be accurate), but you wouldn’t be able to provide a precise response, nor would you necessarily be correct in your approximation. Windows are not valid instruments for measuring temperature.

Likewise, survey questions do not always measure what we might think (or hope) they’re measuring. Consider this question that could be found on an elementary school exam testing for reading comprehension:

Q. Which of the following items would NOT be found in a backyard?
a. swing set
b. lawnmower
c. couch
d. rabbit

The correct answer is clearly C, right? And any small child who answers anything other than C obviously doesn’t have the desired reading comprehension skills, right?

Not so fast. What about the child from an urban area who has never seen a backyard? What about the child who just moved here from a town where it’s quite common to sit on the old couch in the yard, and there aren’t any rabbits because the climate doesn’t support their habitat?

A well-intentioned person could use this question to measure reading comprehension, but it isn’t, in fact, measuring what it’s intended to measure. That’s why it needs to be validated before it can be used to make a critical decision like whether the children taking the test should be in an accelerated or remedial reading program.

Likewise, before we make critical decisions in criminal justice that have long-lasting impacts for individuals and communities, it’s essential that we’re using tests that measure what we intend for them to measure.

How do researchers validate an assessment instrument?

There are several different ways to look at validation, and different tests help researchers determine how valid a particular instrument is. In terms of risk and needs assessments in the criminal justice space, there are five key types of validity that researchers typically consider:

• Content validity – How essential, useful and relevant each question is to measuring the theoretical concept.
• Construct validity – How well test questions represent the overall domain of the test. Or rather, how well they stay on-topic. If a test is designed to measure depression, it should be measuring only depression, not closely related constructs like anxiety.
• Predictive validity – How well a tool predicts future performance.
• Internal validity – How well the tool adheres to basic cause-and-effect. That is, how confidently we can say A causes B. If the tool muddles the variables and it’s possible that C causes B, or that D and E are interfering, then a tool has low internal validity.
• External validity – How confidently we can generalize the results to different people in different places at different times.

The best assessment tools are continuously being validated. Society changes, people change, and the justice landscape changes. The Northpointe team is always validating and re-validating the COMPAS Risk/Needs Assessment in different locations around the country to ensure its validity.

Even though it’s not related to validity in the scientific sense, we also put measures in place to ensure data quality. This gives us an added level of confidence when we run validity tests on COMPAS. The two data quality tests we have in place are testing for lies and for randomness. If someone is trying to game the test or present themselves in the best possible light, we’ll know. Likewise, if someone is randomly choosing answers, we’ll know. This adds to our overall confidence in COMPAS.

What does this mean for my agency?

It means you need to start asking questions about the tools you use. If you’re using COMPAS, ask us anything! We’re happy to walk you through a high-level overview or a nitty-gritty look at exactly how we’re validating our tools.

If you’re not using COMPAS, consider the source of the tools you’re using and the rigor with which they were created and validated. We have over 30 years of scientific data backing every one of our tools, and it pains us to see critical, life-altering decisions being made with home-grown surveys.

Even if the science doesn’t convince you, the pragmatics should. The bottom line is that scientifically validated tools are more likely to stand up in court.

For more information on the COMPAS Risk/Needs Assessment or our validation process, please  contact us.