RADIANCE BLOG

Category: Qualitative Research

Does This Survey Make Sense?

It’s pretty common for Corona to combine qualitative and quantitative research in a lot of our projects.  We will often use qualitative work to inform what we need to ask about in qualitative phases of the research, or use qualitative research to better understand the nuances of what we learned in the quantitative phase.  But did you know that we can also use qualitative research to help design quantitative research instruments through something called cognitive testing?

The process of cognitive testing is actually pretty simple, and we treat it a lot like a one-on-one interview.  To start, we recruit a random sample of participants who would fit the target demographic for the survey.  Then, we meet with the participants one-on-one and have them go through the process of taking the survey.  We then walk through the survey with them and ask specific follow-up questions to learn how they are interpreting the questions and find out if there is anything confusing or unclear about the questions.

In a nutshell, the purpose of cognitive testing is to understand how respondents interpret survey questions and to ultimately write better survey questions.  Cognitive testing can be an effective tool for any survey, but is particularly important for surveys on topics that are complicated or controversial, or when the survey is distributed to a wide and diverse audience.  For example, you may learn through cognitive testing that the terminology you use internally to describe your services are not widely used or understood by the community.  In that case, we will need to simplify the language that we are using in the survey.  Or, you may find that the questions you are asking are too specific for most people to know how to answer, in which case the survey may need to ask higher-level questions or include a “Don’t Know” response option on many questions.  It’s also always good to make sure that the survey questions don’t seem leading or biased in any way, particularly when asking about sensitive or controversial topics.

Not only does cognitive testing allow us to write better survey questions, but it can also help with analysis.  If we have an idea of how people are interpreting our questions, we have a deeper level of understanding of what the survey results mean.  Of course, our goal is to always provide our clients with the most meaningful insights possible, and cognitive testing is just one of the many ways we work to deliver on that promise.


How representative is that qualitative data anyway?

When we do qualitative research, our clients often wonder how representative the qualitative data is of the target population they are working with.  It’s a valid question.  To answer, I have to go back to the purpose of conducting qualitative research in the first place.

The purpose of qualitative research is to understand people’s perceptions, opinions, and beliefs, as well as what is causing them to think in this way.  Unlike quantitative research, the purpose is not to generalize the results to the population of interest.  If eight out of ten participants in a focus group share the same opinion, can we say that 80% of people believe that particular opinion?  No, definitely not, but you can be pretty confident that it will be a prevalent opinion in the population.

While qualitative data is not statistically representative of a population, we still have guidelines that we follow to make sure we are capturing reliable data.  For example, we suggest conducting at least three focus groups per unique segment.  Qualitative research is fluid by nature, so data gathered from across three groups allows us to see consistent themes and patterns across groups, and assess if there are any outliers or themes exclusive to one group that may not be representative of the unique segment as a whole.

Still not sure which methodology will best be able to answer your research questions?  We can help you choose!


Turning Passion into Actionable Data

Nonprofits are among my favorite clients that we work with here at Corona for a variety of reasons, but one of the things that I love most is the passion that surrounds nonprofits.  That passion shines through the most in our work when we do research with internal stakeholders for the nonprofit.  This could include donors, board members, volunteers, staff, and program participants.  These groups of people, who are already invested in the organization are passionate about helping to improve it, which is good news when conducting research, as it often makes them more likely to participate and increase response rates.

Prior to joining the Corona team, I worked in the volunteer department of a local animal shelter.  As a data nerd even then, I wanted to know more about who our volunteers were, and how they felt about the volunteer program.  I put together an informal survey, and while I still dream about what nuggets could have been uncovered if we had gone through a more formal Corona-style process, the data we uncovered was still valuable in helping us determine what we were doing well and what we needed to improve on.

That’s just one example, but the possibilities are endless.  Maybe you want to understand what motivated your donors to contribute to your cause, how likely they are to continue donating in the future, and what would motivate them to donate more.  Perhaps you want to evaluate the effectiveness of your programs.  Or, maybe you want to know how satisfied employees are with working at your organization and brainstorm ideas on how to decrease stress and create a better workplace.

While you want to be careful about being too internally focused and ignoring the environment in which your nonprofit operates, there is huge value in leveraging passion by looking internally at your stakeholders to help move your organization forward.

 


Informal Research for Nonprofit Organizations

While Corona serves all three sectors (private, public, and nonprofit) in our work, we have always had a soft spot for our nonprofit clients.  No other type of organization is asked to do more with less, so we love working with nonprofits to help them refine their strategies to be both more effective at fulfilling their missions and more financially stable at the same time.

However, while we are thrilled for the opportunities to work with dozens of nonprofits every year, we know that there are hundreds of other organizations that we don’t work with, many of which simply don’t have the resources to devote to a formal marketing research effort. I’m a huge fan of the Discovery Channel show MythBusters, so I’ll share one of my favorite quotes:

http://www.tested.com/art/makers/557288-origin-only-difference-between-screwing-around-and-science-writing-it-down/
Image from Tested courtesy of DCL

While few would argue that the results found in an episode of MythBusters would qualify as academically rigorous research, I think most would agree that trying a few things out and seeing what happens is at least better than just trusting your gut instinct alone.  Likewise, here are a few ideas for ways that nonprofits can gather at least some basic information to help guide their strategies through informal “market research.”

Informal Interviews

One-on-one interviews are one of the easiest ways to gather feedback from a wide variety of individuals.  Formal interview research involves a third-party randomly recruiting individuals to participate from the entire universe of people you are trying to understand, but simply talking to people one-on-one about the issues or strategies that you are considering can be very insightful.  Here are a few pointers on getting the most out of informal interviews:

  • Dedicate time for the interview. It may seem easy to just chat with someone informally at dinner or at an event, but the multitude of distractions will reduce the value you get out of the conversation.  Find a time that both parties can really focus on the discussion, and you’ll get much better results.
  • Write your questions down in advance. It’s easy to go down a rabbit hole when having a conversation about something you are passionate about, so be sure to think through the questions you need to answer so that you can keep the conversation on track.
  • Record the conversation (or at least take notes). Take the MythBusters’ advice, and document the conversation.  If you’ve talked to a dozen people about your idea, it will be impossible for you to remember it all.  By having documentation of the conversations, you can look back later and have a better understanding of what your interviewees said.

Informal focus groups

Similar to interviews, in an ideal world focus groups should be conducted by a neutral, third-party with an experienced moderator who can effectively guide the group discussion to meet your goals.  However, as with interviews, you can still get a lot of value out of just sitting down with a group and talking through the issues.  In particular, if you have an event or conference where many people are together already, grabbing a few to talk through your ideas can be very informative.  Our suggestions for this type of “research” are similar to those for informal interviews, with slight differences in their implications:

  • Dedicate time for the discussion. As mentioned before, it may be tempting to just say “We’ll talk about this over dinner” or “Maybe if we have time at the end of the day we can get together.”  You’ll get far better results if everyone can plan for the conversation in advance and participate without distractions.
  • Write your questions down in advance. Even more so than for interviews, having a formal plan about what questions you want to ask is imperative.  Group discussions have a tendency of taking on a life of their own, so having a plan can help you to guide the discussion back on topic.
  • Document the results. Again, you may think you can remember everything that was said during a conversation, but a few months down the road, you will be very thankful that you took the time to either record the conversation or take notes about what was said.

Informal Surveys

Surveys are, perhaps, the most difficult of these ideas to implement on an informal basis, but they can nevertheless be very useful.  If you’re just needing some guidance on how members of an organization feel about a topic, asking for a show of hands at a conference is a perfectly viable way of at least getting a general idea of how members feel.  Similarly, if you have a list of email addresses for your constituents, you could simply pose your question in an email and ask people to respond with their “vote.”

The trickiest part is making sure that you understand what the results actually represent.  If your conference is only attended by one type of member, don’t assume that their opinions are the same as other member types.  Likewise, if you only have email addresses for 10 percent of your constituents, be careful with assuming that their opinions reflect those of the other 90 percent.  Even so, these informal types of polling can help you to at least get an idea of how groups feel on the whole.

~

Hopefully these ideas can give your nonprofit organization a place to start when trying to understand reactions to your ideas or strategies.  While these informal ways of gathering data will never be as valuable as going through a formal research process, they can provide at least some guidance as you move forward that you wouldn’t have had otherwise.

And if your issues are complex enough that having true, formal research is necessary to ensure that you are making the best possible decisions for your organization, we’ve got a pretty good recommendation on who you can call…


Who’s Excited for 2016?

Oh man, where did 2015 even go? Sometimes the end of the year makes me anxious because I start thinking about all the things that need to be done between now and December 31st. And then I start thinking about things that I need to do in the upcoming year, like figuring out how to be smarter than robots so that they don’t steal my job and learning a programming language since I’ll probably need it to talk to the robots that I work with in the future. Ugh.2015 Calendar

Feeling anxious and feeling excited share many of the same physical features (e.g., sweaty palms, racing heart, etc.),  and research has shown that it is possible to shift feelings of anxiety to feelings of excitement even by doing something as simple as telling yourself you are excited. So, let me put these clammy hands to use and share some of the things that I am excited about for 2016:

  • Technological advancements for data collection. Changes in phone survey sampling are improving the cell phone component of a survey. Also, we have been looking at so many new, cool ways of collecting data, especially qualitative data. Cell phones, which are super annoying for phone surveys, are simultaneously super exciting for qualitative research. I’m excited to try some of these new techniques in 2016.
  • Improvements in the technology that allows us to more easily connect with both clients and people who work remotely. We use this more and more in our office. I’m not sure if in 2016 we will finally have robots with iPads for heads that allow people to Skype their faces into the office, but I can dream.
  • Work trips! I realize that work trips might be the stuff of nightmares at other jobs. But Coronerds understand the importance of finding humor, delicious food, and sometimes a cocktail during a work trip.
  • New research for clients old and new. This year I’ve learned all sorts of interesting facts about deck contractors, the future of museums, teenage relationships, people’s health behaviors, motorcyclists, business patterns in certain states, how arts can transform a city, and many more! I can’t wait to see what projects we work on next year.
  • Retreat. For people who really love data and planning, there is nothing as soothing as getting together as a firm to pore over a year’s worth of data about our own company and draw insights and plans from it.

Alright, I feel a lot better about 2016. Now I’m off to remind myself that these clammy hands also mean that I’m very excited about holiday travel, last minute shopping, and holiday political discussions with the extended family…


Magic 8 Ball Says…Outlook Good!

It’s hard to believe that it’s already that time of year where you’re thinking not only about turkeys, in-laws and shopping, but also reflecting on what you’ve accomplished in the last year.  For me, this November marks my one-year anniversary with Corona Insights, and since Gregory joined our team this fall, I’m officially no longer the newbie! Future

It’s also the time of year when many start thinking about what the future may hold for the year ahead.  While I wouldn’t recommend calling me for your late night psychic readings (as I feel my fortune-telling skills are still a little below average), I can predict what you can expect if you hire Corona Insights for your research needs.

  • You’ll learn something new…One of the most exciting things about research is finding that nugget of information that you never knew existed. After conducting research, you’ll have a far better understanding of what your customers are thinking, some of which you’ll be hearing for the first time!
  • And you’ll confirm what you already know. Our clients are experts in their fields, so it’s no surprise that some information uncovered by our research confirms what they already know, and that’s a good thing! Having research and data to back-up what you already believed about your organization can be instrumental in making sound decisions to guide your organization forward.
  • You’ll be surprised by what people know…When I first started at Corona, I sometimes worried that people would not know enough about a topic or have enough opinions to fill the time of a full focus group, but my worries quickly melted away. People love to be asked their opinions, and are usually more than willing to share, even if they don’t know a lot about the topic.  Learning about people’s perceptions and opinions (even from those who don’t know a lot about your organization) and where those thoughts come from is extremely valuable.
  • And probably frustrated by what they don’t. I’m sure there is nothing more frustrating for our clients observing focus groups than hearing participants say they haven’t heard of their organization, or have misconceptions, despite all their best marketing efforts.  Understandably, clients often feel the urge to run in to the group and tell participants the real story.  But, from Corona’s perspective, it’s a GOOD thing to hear about what people don’t know, as well as their misconceptions.  After all, it’s hard to enhance your work when you don’t know what you most need to improve upon.  While it may be frustrating to hear, it’s probably part of why you decided to conduct research in the first place!

Above all I can predict (even without consulting my Magic 8 ball) that at the end of the day, you’ll come away with actionable insights.


How to Choose your own Adventure when it comes to Research

One of the things we’ve been doing at Corona this year that I’ve really enjoyed is resurrecting our book club. I enjoy it because it’s one way to think about the things we are doing from a bigger picture point of view, which is a welcome contrast to the project-specific thinking we are normally doing. One topic that’s come up repeatedly during our book club meetings is the pros and cons of different types of research methodology.

Knowing what kind of research you need to answer a question can be difficult if you have little experience with research. Understanding the different strengths and weaknesses of different methodologies can make the process a little easier and help ensure that you’re getting the most out of your research. Below I discuss some of the key differences between qualitative and quantitative research.

Qualitative Research

Qualitative research usually consists of interviews or focus groups, although other methodologies exist. The main benefit of qualitative research is that it is so open. Instead of constraining people in their responses, qualitative research generally allows for free-flowing, more natural responses. Focus group moderators and interviewers can respond in the moment to what participants are saying to draw out even deeper thinking about a topic. Qualitative research is great for brainstorming or finding key themes and language.

Qualitative data tend to be very rich, and you can explore many different themes within the data. One nice feature of qualitative research is that you can ask about topics that you have very little information about. For example, you might have a question in a survey that asks, “Which of the following best describes this organization? X, Y, Z, or none of the above.” This quantitative question assumes that X, Y, and Z are the three ways that people describe this organization, which requires at least some knowledge. A qualitative research question for this topic would ask, “How would you describe this organization?”. This is one of the reasons why qualitative research is great for exploratory research.

The primary weakness of qualitative research is that you can’t generate a valid population statistic from it. For example, although you could calculate what percent of people in focus groups said that Y was a barrier to working with your organization, you couldn’t generalize that estimate to the larger population. However, if you just wanted to identify the main barriers, then that would be possible with qualitative research. So even if 30% of focus group participants reported this barrier, we don’t know what percent of people overall would report that same barrier. We would only be able to say that this is a potential barrier. It’s important to think carefully about whether or not this would be a weakness for your research project.

Quantitative Research

The main goals of quantitative research are to estimate population quantities (e.g., 61% of your donors are in Colorado) and test for statistical difference between groups (e.g., donors in Colorado gave more money than those in other states). With quantitative research, you’re often sacrificing depth of understanding for precision.

One of the benefits to quantitative research, aside from being able to estimate population values, is that you can do a lot of interesting statistical analyses. Unlike a small sample of 30 people from focus groups, a large sample of 500 survey respondents allows for all sorts of analyses. You can look for statistical differences between groups, identify key clusters of respondents based on their responses, see if you can predict people’s responses from certain variables, etc.

There usually is not one single best way to answer a question with data, so thinking through your options and the benefits afforded by those options is important. And as always, we’re here to help you make these decisions if the project is complicated.


Question What You Know

What I’m about to disclose may seem weird, or familiar, depending on the kind of person you are.  Last week our Corona Book Club met to discuss our recent pick, and as I sat down to put together my thoughts on it, I was reminded of an ad campaign from my youth.  I googled the slogan (this is not so weird) and couldn’t find the ad online, so instead I pulled out my huge 3-ring binder where I save things like interesting ads and magazine articles and dug out my own personal copy from circa 1995 (that may be weird).

(Even more strange is that I only have a photocopy of this particular ad, so I have no idea where it came from, and it appears to be half of a two-page ad, so I’m not even sure what the ad is for. My guess is it was for Carnival Cruise Lines or Cirque du Soleil, but it could just as easily have been for perfume or Nike or Waterford Crystal – 90’s ads were full of angsty inspirational prose.  In fact, my google search turned up another blogger writing about one of my favorites – yes, also part of my hard copy collection.)

I digress.  As I wasQuestion what you know saying, the ad text, which reads: “What appears to be new may in fact be familiar. What appears to be familiar may in fact be new.  So question what you know … because you may not really know it at all.” resonated with a point made in the book: “Don’t treat everyday life as boring or obvious; do treat ‘obvious’ actions, settings and events as potentially remarkable.”.

The book is David Silverman’s, A very short, fairly interesting and reasonably cheap book about qualitative research.  In addition to his exhortation to pay attention to things that may seem unremarkable, he also encourages researchers to explore other research methods and other sources of data.  He points out, accurately, that most commercial qualitative research is limited to interviews and focus groups.  He suggests we broaden our horizons to consider observational research methods, analysis of natural language, written documents, and so on.  He provides examples to show the value and possibilities in these alternatives.

Having read this book, it seems to me that my binder of nearly-vintage ads could be a useful data source for studying the Gen X persona.  Intrepid grad students can request a copy.

And if you recognize this ad, please tell us about it!


Incorporating Exercises into Focus Groups

At Corona Insights, we are always researching best practices for the work we do.  In the world of qualitative research, this often means best practices for conducting focus groups.  Over the years, we have learned many tips and tricks for conducting focus groups, which includes incorporating exercises into our discussions.  There are a wide range of exercises and activities we use depending on the topic of discussion.  These exercises can include everything from drawing ideas to ranking priorities to testing messaging or ads.

Incorporating exercises into focus groups serves several important purposes:

  • It gives participants an opportunity to think about topics in a different way. It can sometimes be hard for participants to fully think through a topic when they are expected to quickly answer a question.  Allowing them more time to think about the topic in an exercise helps encourage different thinking and promotes answers and opinions that are below top of mind.Light Bulbs
  • It encourages those who are quiet to express their opinions. Some participants are quieter by nature, so it can be a challenge to hear their opinions in a group of 8-10 other people.  Incorporating exercises, and having participants share their thought process for completing the activity ensures that even the most shy of participants are participating.
  • It makes the group more interesting. Sitting in a room listening to someone ask questions for two hours can be exhausting, especially if the group takes place in the evening. Incorporating exercises into a focus group breaks up the model of the moderator asking questions and participants answering, and hopefully makes the group more fun!
  • It helps breaks up group think. Sometimes if there are strong personalities in the room, or if a topic is particularly controversial, participants can act as if they agree with each other on certain topics, even if this is not truly the case. This can also happen if most participants haven’t given the topic a lot of previous thought, and a few participants have more knowledge on the topic than others.  Having the participants work and think individually during an exercise ensures that the group is not being influenced by group think.

So, if you ever attend a Corona focus group, don’t be surprised to see participants doing more than just answering questions (and hopefully having more fun and expressing more thoughtful and insightful opinions because of it).


Wondering if you’re normal? The answer is probably yes.

Katherine Brown is Corona’s 2015 summer intern. She has one year left at Macalester College where she has chosen to study anthropology because it gives her an excuse to talk to people from all over the place and call it work. After living in Germany for a calendar year and Turkey for an academic one, Katherine is able to pass for a German and haggle for coffee sets in Turkish. She’s also well on her way to writing a senior thesis about the huge German Turkish population. When she’s not busy looking for people she’s never met to tell her about their lives in detail, she’s likely spending time with some portion of her Denver-based extended family in restaurants, the mountains, or both. 

Wondering if you’re normal? The answer is probably yes.
Written by Katherine Brown

Early this summer in the midst of looking for jobs and internships, I decided to make a documentary about my extended family. When I mentioned this idea to my eighteen year old cousin, she immediately wanted in. Since I’m all about teamwork and she’s more like a sister than a cousin to me anyway, it didn’t take much convincing at all before we were a small team of two.

Our first (and, I will add, only) meeting was a discussion of how to start the project. It’s hard to make a documentary about your own opinionated and very close family without offending anyone. We decided the best thing to do would be to write up a short questionnaire, send it out to the family, see where people had strong opinions, and go from there. Coming from an anthropology background and hours of training on how to frame questions, how to stay neutral, and how to make people comfortable enough to share information, coming up with five questions for my family seemed like nothing. “What are the biggest generational gaps?” “Do you see major points of contention? What are they?” I wanted to look for patterns, anything we could latch onto and say “YES! That’s it! Let’s dig deeper there!” My cousin, who just graduated from high school, was set on a different approach. She felt that it was crucial to know just one thing.

“Is our family normal? Why or why not?”

It took me a while to figure out why this one question seemed so crucial. Maybe she was looking for someone to confirm her suspicions that our family is not normal. Maybe she wanted everyone to tell her that we are, in fact, normal and that she didn’t need to worry. I mostly just put it off to the fact that she hadn’t had the training in asking questions that I have.

A couple of weeks later, I started working at Corona as an intern for the summer. By then, I’d more or less forgotten about the incident with my cousin. My life became filled with interviews and survey data. It wasn’t until this week that I had a chance to step back again and think about my cousin’s question. Apparently all of this first-hand exposure to research has been teaching me something, because I suddenly understood why she had been so focused on the question of normal. She knew what we were really trying to get at through this documentary.

“Am I normal?”

Through my coursework and experience interviewing people, I’ve always focused on the process. It never occurred to me that all the data analysis (whether from surveys or interviews or focus groups) I’ve done is really just a quest to find “normal”. Of course, each data set is slightly different, focusing on finding normalcies among slightly different populations. But no matter what kind of data I am looking at, people love to emphasize their own ‘normalness’.

In my academic research on German Turks, many interviews will start with people telling me that they are “not really the kind of person I’m looking for”. Their story is different from the ones I’m used to hearing because they don’t come from Eastern Anatolia. Or because they “did not have a choice when they moved to Germany”. Or because they “didn’t even move back to Turkey on purpose”.  And yet, in each interview I hear more or less the same thing. The details are different, of course. But there are notable patterns in what people emphasize when I ask what it means to be a German Turk. These people who present themselves as unique or abnormal almost always end up giving me details about how they are just like all the other German Turks. I hear “Anyone who has had experience in Germany will tell you that…” in almost every interview, including the ones that begin with the phrase “I’m not really the kind of person you’re looking for”. Those normal habits of abnormal people are exactly what I’m looking for when I’m building my senior thesis. Those habits are the ones that matter.

My exposure to qualitative research at Corona has shown me similar results, albeit without the introductory phrase of “I’m not the right person”. Research here is often conducted to find out people’s opinions. I’ve noticed that most people believe that their own ideas are also the most widely-held opinions, the ‘normal’ opinions. Answers to an interview question sometimes begin with pauses implying how ‘obvious’ the answer is going to be. In the next interview someone may pause for the same reason, then completely contradict the last person’s answer. What I’ve learned from this is that everyone is normal. Everyone knows the ins and outs of their industry. Everyone knows what everyone else thinks. At least until you talk to everyone else.

Even quantitative survey data allows room for people to remark upon their normal lives. People will circle survey questions and comment “why are you even asking this?”, either forgetting or oblivious to other people’s perceptions of normal. In open ended responses people will write that they do “the normal things” or go somewhere “right next to where you get off the train”. That version of normal may or may not be the same as mine. Many of these people are filling out the surveys quickly, and do not remember that they’ve never met me. Why would any one person’s experience be anything BUT normal?

I will spoil the end of the unfinished family documentary.  Everyone in my family responded that we are normal. Their responses to my open-ended, non-judgmental, broad, neutral questions varied pretty greatly. And yet, there were patterns. In my experience, following those patterns would lead to a pretty representative profile of a ‘normal’ family with the same education, socio-economic status, geographic location, and generational breakdown as my own. On her quest to find normal, my cousin may be beginning to realize that everything and everyone is normal. On my quest to become a better researcher, I’m beginning to realize that what I’m looking for is the answer to the question my cousin was so set on asking.

“Are you normal? Why or why not?”