Models, or more specifically, statistical models meant to predict human behavior, are inherently wrong. George Box is credited with the saying, “All models are wrong, but some are useful,” even though he may never have said those exact words. (He and another author did write in their book, “Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.”)

Why are they wrong?

Modeling human behavior – predicting what will someone do under certain conditions – is inherently complex. No model can perfectly capture the intricacies of real-world decision making (at least yet, and probably for some time to come).

Models by their nature simplify the problem at hand. What variables do we believe will impact the decision and what variables can be reasonably measured? We therefore remove details that we don’t think are critical or that we don’t think can be quantified and included.

Take as an example trying to predict someone buying a membership for a museum. So many things can impact that decision and those things can quickly change. There are of course prices and benefits. Is there a special offer? How much are they engaged in the topic of the museum? How far away are they from the museum? What has been their past experience at the museum? How often do they expect to go and how often do they think it’s worth going (is there anything new)? How much extra income do they have at the moment? How many other memberships do they have? How might becoming a member change their perception of themself? Do they even know about membership? Did they receive your marketing about it?

The list goes on and on and, critically, will be different for each person and can change, sometimes quickly.

So, we pick the factors we can realistically measure through surveys or past member behavior or other sources, and model those factors to predict future behavior.

Sometimes they appear right

With all that can be wrong in modeling outcomes they can often be right, or at least directionally right. Often, the variables we can’t measure, or don’t know to measure, are less central to the prediction or are correlated with other variables that we have included, so by capturing one we capture several.

Even when wrong they can still be useful

Even with inaccurate modeling, the act of modeling can be useful. By going through the exercise of building the model we can start to understand what factors are driving outcomes. And critically, by including variables that you may be able to impact, it can help you determine where to focus your efforts.

Back to the museum example, you may see that awareness is actually the biggest hindrance to attracting more members. Or maybe it’s the perception that there isn’t a reason to come back anytime soon. Also, by modeling behavior with different benefits and price points, you can begin to understand what features are more important in a membership and the price sensitivity your audience may (or may not) have.

How we Build models at Corona

To maximize the useful part of models, we deliver models in the form of a tool that allows our clients to continue to experiment. By allowing some inputs to be modifiable – such as price – it turns a model from providing a static prediction to an ongoing decision-making tool. We may combine survey data, past membership data, Census data, and more to help predict outcomes now and into the future under different scenarios.

Reach out if you want to discuss modeling for your organization. It may not be exactly right, but we will strive for it to be useful.