It’s pretty common for Corona to combine qualitative and quantitative research in a lot of our projects. We will often use qualitative work to inform what we need to ask about in qualitative phases of the research, or use qualitative research to better understand the nuances of what we learned in the quantitative phase. But did you know that we can also use qualitative research to help design quantitative research instruments through something called cognitive testing?
The process of cognitive testing is actually pretty simple, and we treat it a lot like a one-on-one interview. To start, we recruit a random sample of participants who would fit the target demographic for the survey. Then, we meet with the participants one-on-one and have them go through the process of taking the survey. We then walk through the survey with them and ask specific follow-up questions to learn how they are interpreting the questions and find out if there is anything confusing or unclear about the questions.
In a nutshell, the purpose of cognitive testing is to understand how respondents interpret survey questions and to ultimately write better survey questions. Cognitive testing can be an effective tool for any survey, but is particularly important for surveys on topics that are complicated or controversial, or when the survey is distributed to a wide and diverse audience. For example, you may learn through cognitive testing that the terminology you use internally to describe your services are not widely used or understood by the community. In that case, we will need to simplify the language that we are using in the survey. Or, you may find that the questions you are asking are too specific for most people to know how to answer, in which case the survey may need to ask higher-level questions or include a “Don’t Know” response option on many questions. It’s also always good to make sure that the survey questions don’t seem leading or biased in any way, particularly when asking about sensitive or controversial topics.
Not only does cognitive testing allow us to write better survey questions, but it can also help with analysis. If we have an idea of how people are interpreting our questions, we have a deeper level of understanding of what the survey results mean. Of course, our goal is to always provide our clients with the most meaningful insights possible, and cognitive testing is just one of the many ways we work to deliver on that promise.
When we do qualitative research, our clients often wonder how representative the qualitative data is of the target population they are working with. It’s a valid question. To answer, I have to go back to the purpose of conducting qualitative research in the first place.
The purpose of qualitative research is to understand people’s perceptions, opinions, and beliefs, as well as what is causing them to think in this way. Unlike quantitative research, the purpose is not to generalize the results to the population of interest. If eight out of ten participants in a focus group share the same opinion, can we say that 80% of people believe that particular opinion? No, definitely not, but you can be pretty confident that it will be a prevalent opinion in the population.
While qualitative data is not statistically representative of a population, we still have guidelines that we follow to make sure we are capturing reliable data. For example, we suggest conducting at least three focus groups per unique segment. Qualitative research is fluid by nature, so data gathered from across three groups allows us to see consistent themes and patterns across groups, and assess if there are any outliers or themes exclusive to one group that may not be representative of the unique segment as a whole.
Still not sure which methodology will best be able to answer your research questions? We can help you choose!
Nonprofits are among my favorite clients that we work with here at Corona for a variety of reasons, but one of the things that I love most is the passion that surrounds nonprofits. That passion shines through the most in our work when we do research with internal stakeholders for the nonprofit. This could include donors, board members, volunteers, staff, and program participants. These groups of people, who are already invested in the organization are passionate about helping to improve it, which is good news when conducting research, as it often makes them more likely to participate and increase response rates.
Prior to joining the Corona team, I worked in the volunteer department of a local animal shelter. As a data nerd even then, I wanted to know more about who our volunteers were, and how they felt about the volunteer program. I put together an informal survey, and while I still dream about what nuggets could have been uncovered if we had gone through a more formal Corona-style process, the data we uncovered was still valuable in helping us determine what we were doing well and what we needed to improve on.
That’s just one example, but the possibilities are endless. Maybe you want to understand what motivated your donors to contribute to your cause, how likely they are to continue donating in the future, and what would motivate them to donate more. Perhaps you want to evaluate the effectiveness of your programs. Or, maybe you want to know how satisfied employees are with working at your organization and brainstorm ideas on how to decrease stress and create a better workplace.
While you want to be careful about being too internally focused and ignoring the environment in which your nonprofit operates, there is huge value in leveraging passion by looking internally at your stakeholders to help move your organization forward.
It’s hard to believe that it’s already that time of year where you’re thinking not only about turkeys, in-laws and shopping, but also reflecting on what you’ve accomplished in the last year. For me, this November marks my one-year anniversary with Corona Insights, and since Gregory joined our team this fall, I’m officially no longer the newbie!
It’s also the time of year when many start thinking about what the future may hold for the year ahead. While I wouldn’t recommend calling me for your late night psychic readings (as I feel my fortune-telling skills are still a little below average), I can predict what you can expect if you hire Corona Insights for your research needs.
- You’ll learn something new…One of the most exciting things about research is finding that nugget of information that you never knew existed. After conducting research, you’ll have a far better understanding of what your customers are thinking, some of which you’ll be hearing for the first time!
- And you’ll confirm what you already know. Our clients are experts in their fields, so it’s no surprise that some information uncovered by our research confirms what they already know, and that’s a good thing! Having research and data to back-up what you already believed about your organization can be instrumental in making sound decisions to guide your organization forward.
- You’ll be surprised by what people know…When I first started at Corona, I sometimes worried that people would not know enough about a topic or have enough opinions to fill the time of a full focus group, but my worries quickly melted away. People love to be asked their opinions, and are usually more than willing to share, even if they don’t know a lot about the topic. Learning about people’s perceptions and opinions (even from those who don’t know a lot about your organization) and where those thoughts come from is extremely valuable.
- And probably frustrated by what they don’t. I’m sure there is nothing more frustrating for our clients observing focus groups than hearing participants say they haven’t heard of their organization, or have misconceptions, despite all their best marketing efforts. Understandably, clients often feel the urge to run in to the group and tell participants the real story. But, from Corona’s perspective, it’s a GOOD thing to hear about what people don’t know, as well as their misconceptions. After all, it’s hard to enhance your work when you don’t know what you most need to improve upon. While it may be frustrating to hear, it’s probably part of why you decided to conduct research in the first place!
Above all I can predict (even without consulting my Magic 8 ball) that at the end of the day, you’ll come away with actionable insights.
At Corona Insights, we are always researching best practices for the work we do. In the world of qualitative research, this often means best practices for conducting focus groups. Over the years, we have learned many tips and tricks for conducting focus groups, which includes incorporating exercises into our discussions. There are a wide range of exercises and activities we use depending on the topic of discussion. These exercises can include everything from drawing ideas to ranking priorities to testing messaging or ads.
Incorporating exercises into focus groups serves several important purposes:
- It gives participants an opportunity to think about topics in a different way. It can sometimes be hard for participants to fully think through a topic when they are expected to quickly answer a question. Allowing them more time to think about the topic in an exercise helps encourage different thinking and promotes answers and opinions that are below top of mind.
- It encourages those who are quiet to express their opinions. Some participants are quieter by nature, so it can be a challenge to hear their opinions in a group of 8-10 other people. Incorporating exercises, and having participants share their thought process for completing the activity ensures that even the most shy of participants are participating.
- It makes the group more interesting. Sitting in a room listening to someone ask questions for two hours can be exhausting, especially if the group takes place in the evening. Incorporating exercises into a focus group breaks up the model of the moderator asking questions and participants answering, and hopefully makes the group more fun!
- It helps breaks up group think. Sometimes if there are strong personalities in the room, or if a topic is particularly controversial, participants can act as if they agree with each other on certain topics, even if this is not truly the case. This can also happen if most participants haven’t given the topic a lot of previous thought, and a few participants have more knowledge on the topic than others. Having the participants work and think individually during an exercise ensures that the group is not being influenced by group think.
So, if you ever attend a Corona focus group, don’t be surprised to see participants doing more than just answering questions (and hopefully having more fun and expressing more thoughtful and insightful opinions because of it).
As it finally starts to feel like summer, one of my favorite activities is sitting outside on the patio reading a good book. My fellow Coronerds must agree with this sentiment as we recently kicked off summer with our very own Corona Book Club, fearlessly led by Senior Associate, Matt Bruce. As we started discussing this quarter’s book, I got to thinking that the ingredients for a successful book club really mirror the ingredients for a successful research project.
Those of you who may have participated in a book club on your own personal time may have noticed that some book clubs thrive, while others flop. I would argue that success starts with clearly defining a purpose. Is the purpose of the book club to socialize, to learn new writing techniques or to learn about a specific topic? If all members of the club don’t agree on a purpose, it probably won’t be successful. Imagine the chaos that may ensue if half of the members haven’t read the book because they came just to socialize, while the other half planned to have an in-depth discussion about literary techniques. If you don’t have a clear purpose for the book club, you won’t be able to agree on a book or have a meaningful discussion.
The same is true for research projects. We’ve talked a lot in the past about beginning with the end in mind—that is, being clear about why the research is being conducted in the first place, and how it will be used once complete. When our clients are clear about the purpose behind the research project, and all stakeholders are on the same page, we are better able to guide the research by creating research goals, asking the right questions and using appropriate analysis techniques so that the research is actionable.
Wondering about the purpose of Corona’s book club? It’s to continually learn about ways to improve our processes to make our work more useful to our clients, of course!
Last month, I attended the RIVA Training Institute in Rockville, Maryland to get a fresh look at how Corona tackles qualitative research. It was heartening to find that our qualitative research practices match industry standards, but as with any good training, I came away with some new insights that will undoubtedly help make us better researchers.
One thing in particular that stood out to me was taking a look at how we phrase questions during focus groups and interviews. Perhaps you have been in a situation where you feel like you’re asking a really valuable question, but aren’t getting comprehensive responses. Why? Well, the answer is actually in the question. The problem may, in fact, be the word, “why”?
Of course, qualitative research is all about uncovering answers to “why” questions, but using the word “why” may actually prevent you from getting the answers you need. The word “why” tends to put people on the defensive. As an example, think back to when you were a teenager, and you were late for your curfew. What’s the first thing your mom asked? “Why are you late? Why didn’t you call? Why don’t you show some respect?” You’re put on the spot and immediately feel the need to defend yourself. In general, there are fairly negative connotations with the word “why”, so although it might be somewhat subconscious, people may be less likely to open up and give deeper responses if they’re asked to explain themselves in this way.
Of course, we still need to know the answer to these “why” questions, but using a different word choice will likely elicit more responses that are below top-of-mind. Consider using phrases such as, “What are some reasons…” or “What influenced you to…” as alternatives. It sounds simple, but it works wonders! So, next time you ask a “why” question, but don’t get the answers you need, consider rephrasing. You may be surprised to find it will not only help you get better answers as researcher, but also in other aspects of your life.