Someone new to surveying may wonder, “Can I just make up my own survey questions from scratch?” The answer is yes and no. Sometimes it is that simple: You want to gauge awareness? Ask, “Have you heard of XYZ?” and, with a good sampling plan, you’ll probably get an accurate sense for XYZ-awareness in your population of interest.

Other times, however, a question that makes sense on its face does not translate to the answers you want. Asking about intentions, for example, like “How often do you plan to donate/go to the gym/visit XYZ?” often results in inaccurate estimates as they are subject to wishful thinking and planning fallacies (i.e., everything takes longer than you think), among other biases contributing to forecasting errors. One tack that can help in this case is to ask about behavior in the recent past. Although memory is subject to biases too, recalled past behavior is typically a better predictor of future behavior than intentions, e.g., “How often did you visit XYZ in the past month?”

But questions about intentions are by no means the only questions that yield misleading responses. Biases can arise for many reasons, which are often difficult to anticipate, based on the question order and specific phrasing of the question text and response options (see my post on minimizing bias). For this reason, it is good practice, when possible, to use pre-validated measures.

What are Pre-Validated Measures?

In order to be “validated,” measures go through a thorough vetting process, testing that they accurately reflect the theoretical concept they’re intended to measure—what is called construct validity. Key elements of overall construct validity include:

  1. Convergent validity – the measure is related to theoretically related measures
  2. Discriminant validity – the measure is NOT too closely related to theoretically distinct measures
  3. Predictive validity – the scale predicts theoretically expected outcomes

Let’s say, for example, you run a program for youth and want to assess how well participants feel they “fit in” within your program. You could search published measures using key terms like “social fit” and “belonging,” or an experienced researcher could give you the lay of the land for existing scales related to that construct. A pre-validated belonging scale may be closely related to self-esteem and authenticity (convergent validity), unrelated to extraversion (discriminant validity), and a strong predictor of overall well-being and mental health (predictive validity). Rather than making up items that may or may not accurately reflect your construct, you can use the specific well-vetted language of this existing scale. In addition, your background research could help you articulate to partners and funders why your construct of interest matters for long-term outcomes.

Intuitive and… Less Intuitive Measures

Critically, a valid measure may not always be face valid, i.e., sound like it should measure what it does. Some measures are extremely face valid. For example, agreement or disagreement to the single item, “I have high self-esteem,” is actually a well-validated measure for self-esteem. Other scales, on the other hand, do not measure what they seem to measure on their face. What does agreement to the following statements measure, for example?

  • “I always admit my mistakes openly and face the potential negative consequences.”
  • “In conversations I always listen attentively and let others finish their sentences.”
  • “I never hesitate to help someone in case of emergency.”
  • “I always stay friendly and courteous with other people, even when I am stressed out.”

Is this a measure of prudence and virtue? No, these are items from a validated and commonly used “social desirability” scale. Answering affirmatively to many of these over-the-top positive items indicates that you are motivated to answer in socially desirable ways or have high impression management concerns (not something people will always self-report accurately if asked directly).

In other words, the best way to measure a construct is sometimes straightforward—what you’d likely come up with off the top of your head—and sometimes not. If you’re going for a DIY approach, searching for published scales is a great place to start.

—

For further reading, check out these sample scale validation articles: