You may have recently seen the phrase “statistically significant” in an article to note that an occasional glass of red wine can increase life expectancy (let’s hope so) or that your daily coffee intake is causing some kind of malady (worth it). Statistical significance is critical for how we understand important differences in groups and determine whether interventions had effects, but what does it actually mean? How is it determined? When should it be calculated in market research? Whether you haven’t thought about statistical significance since high school stats or see it on a daily basis and just want to make sure you are interpreting it correctly, here is a quick overview of the concept and the tale of the lady tasting tea.
What Does “Statistically Significant” Mean?
In short, statistical significance means that the difference we observe between two groups is unlikely to be attributed to random chance. Note that this tells us nothing about the importance of this difference. In fact, many statistically significant findings are trivial in terms of their actual importance. Why do we compare differences to random chance? How do we determine if a difference is distinct from random chance? Let’s talk tea.
As data storytellers, we’re often charged with the task of revealing useful and hard-hitting data in a manner that is cohesive and captivating. Sharing data in a meaningful way is difficult and blending different types of data together can make it even more challenging.
Great visualization tools for quantitative data have made it easier to share quantitative findings. However, it is important to remember the value that qualitative data bring, and the importance of highlighting participant or customer voice. This event will explore how qualitative and quantitative findings can be effectively blended to tell a more interconnected, nuanced, and comprehensive story. We will highlight real-life examples from research in the private and public sectors. Your presenters include:
NPS is perhaps the most well-known metric in the world of satisfaction research, but as with many acronyms and jargon, what it actually is sometimes gets lost in translation. So, whether the term is new to you, or you just need a refresher, here is a quick overview of NPS.
As a wannabe astronaut since I was about 5 years old, I’m a big fan of SpaceX. It’s super cool to see the progress that has been made just in the last decade when a new, energetic startup truly pushes the capabilities of spaceflight. When I read over discussions about their progress toward manned spaceflight, though, I’m reminded that government agencies (and the organizations that work with them like SpaceX) love acronyms. For someone not deeply embedded in the industry, it’s easy to become lost in discussions about the F9 launch from LC-39A that was NET May 27th and would launch astronauts to the ISS, which is in LEO, and the booster that would come back and land on the ASDS named OCISLY.
Thankfully, the world of marketing and market research isn’t near that full of jargon. However, we do have our own blind spots to terminology that is obvious to those of us who work in the industry but may be completely unknown to those who don’t. For the next month or two, therefore, we’ll be focusing on some of these industry terms and trying to explain what they really mean for our clients.
To start out this discussion, let’s chat about VOC research. At its core, the concept is simple. VOC research is intended to bring the “voice of the customer” to the table when leaders are discussing and debating strategies. The idea is that customers are at the core of nearly any organization’s strategy, so it is vital to make sure that you are constantly listening to what they have to say and considering how you can better meet their needs.
had your share of emails about COVID-19, its impacts, and what organizations
are doing to cope (and to help you cope, too). As many are right now, we’re
taking this seriously and doing our part to stop the spread of COVID-19.
Insights is not typically a consumer-facing company, our work does bring us
into contact with our community, from conducting in-person research, to
community activities with our clients, and of course our clients themselves.
It’s a rapidly evolving situation, but here is where we stand as of today.
Research participation. Recruiting for in-person research (e.g., focus groups largely) has largely been postponed with few exceptions. Beyond just the challenges, we feel this is the right thing to do at this time as we want participants, clients, and staff to be safe and comfortable in their participation. We have not yet seen impacts via other research modes. We’ll continue to monitor the situation and current recommendations with our partners to assess additional needed changes.
Research impacts. Beyond just participation, we’re recommending assessing the impact, if any, on the project itself, such as the impact the current situation will likely have on some research topics.
Project timelines. Whether it’s due to a reason above, or clients themselves being swamped, we are helping mitigate potential delays as best we can. We’ve always been flexible for our clients’ needs and that’s true now more than ever.
What Corona is
Proactively managing projects. Corona is advising clients about the impact COVID-19 is having, or will likely have, on our work together, as well as how we recommend addressing it.
Altering research modes. Corona’s has a breadth of research tools and we are working with our clients to determine if and when it is appropriate to either change how we’re collecting data (e.g., moving more research online) or delaying research if we think the current impacts can’t be mitigated enough.
Recognizing the increased possibility of response bias. With an unprecedented level of societal changes happening to tackle COVID-19, we want to do what we can to avoid collecting data in a non-normal state right now. For instance, we recommended delaying one project that has to do with transportation, including public transit. Due to transit being used less right now, we will wait until behaviors return to normal before continuing with the study. We’re happy to talk through this with any clients who might have questions.
Working from home. Remote offices are nothing new for Corona except now it’s mandated. Staff will only be in the office as needed for critical functions, and we will determine the need for in-person meetings on a per client basis (most, perhaps all, have already moved to virtual or been rescheduled).
For years, qualitative market research has been dominated by focus groups and one-on-one interviews. Each methodology offers benefits. However, as timelines shrink and research objectives expand, it may be time to rethink our “go-to” qualitative methods. This blog will discuss the shortcomings of traditional methods and how refreshed methodological approaches can overcome these pitfalls.
We all know the stereotype—a focus group room with a two-way
mirror providing a thin veil between participants and researchers. It has long
been argued that this barrier maximizes participants’ comfort and allows for
authentic, unbiased, discussions. The two-hour focus group is typically
designed as a question-answer format with activities peppered throughout to
break up the monotony. Historically, activities ask participants to complete a
task individually and then share their thoughts and responses with the group.
This approach is especially effective when delving into emotions or personal
reactions. However, researchers are increasingly designing focus group activities
that are group-based and co-created.
At Corona, we have eagerly implemented co-creation
activities for marketing campaign and message testing projects. In 2019, we
were working with a local association developing messaging for a new nonprofit giving-based
program called Refund What Matters. Participants were asked to get in groups of 2-3 and create a print
advertisement for the program. More specifically, they were asked to come up
with a visual for the print ad, a tagline, and a hashtag. The client observed
as the participants worked together to create their advertisements. The advertisements
revealed the inner thoughts and motivations for nonprofit giving and the
discussion between participants while creating the advertisements provided
insight into how Colorado residents may try and pitch this giving program to
their family members, peers, and coworkers.
A while ago, I started seeing this chart floating around the internet that categorizes things based on good vs. evil and
lawful vs. chaotic. Apparently, I was not hardcore nerd enough to recognize
that the chart was from the game Dungeons
and Dragons. And the hilarious thing is that this chart seems to be able to
group all sorts of things and people in an intuitive way. You can even take a
quiz to find out which group you fall into.
I think one of the reasons that this chart is so popular is
because human brains love patterns and ways of organizing pieces of
information. From psychology research we know that if you have some type of mental
framework for organizing information, like a schema or a story, it is suddenly
way easier to remember that information, process that information, figure out
whether a new piece of information belongs, etc.
Conversations about research design often devolve into
arguments about the relative merits of quantitative and qualitative approaches.
One easy way to resolve these debates is to employ tools from both methods when
answering the question at hand. Mixed methods research, or work that uses both
qualitative and quantitative analysis, has become increasingly common in social
and behavioral sciences over the last decade. As evident in the graph below, a
rapidly growing share of abstracts mention mixed methods and Google’s NGram data (a count of a
word’s instances across millions of scanned books) shows an 800% increase in
the term “mixed methods” over a similar, 14 year period. Recent articles have
highlighted a similar rise in popularity in market
health research, and ad
Add to that the Telephone Consumer Protection Act (TCPA), which creates challenges for research via cell phones in particular, and rules around pharma in some cases (mainly around if you’re allowed to incentivize doctors to participate in research), and our operating environment seems to get more complex by the day.
All of these
regulations, as well as smaller or state-specific rules, have had significant
ripple effects. Even if you don’t think you do business within these geographic
boundaries, you are likely impacted by their reach. Even in our home state of
Colorado, we have had new laws surrounding consumer data.
As we begin a blog series about Colorado,
we must first take note of the diversity of the state. We have mountains, plains, and valleys, of
course, but we also have large metro areas and small towns, agricultural economies
and service economies, and myriad other variations on the Colorado theme. The concept of “Colorado” is very different
depending on where you are in Colorado.
After 20 years of working in Colorado
demographics, I’m always intrigued to see how different organizations break out
the state into regions for their work.
There are some strong patterns the regional analyses I’ve seen over the
years, but there are always variations. What
is the overall pattern, though? If we
had to come up with a consensus set of ten regions, what would they be? I decided to try to find out.
I found regional breakouts of Colorado that have been produced by a dozen large organizations or programs around the state. Those organizations divided the state into anywhere from 4 to 13 regions , all on a county-level basis.