BWF Client Advisory — May 17, 2017
Staying Up to Date
At its core, predictive models represent a snapshot in time. Whether that snapshot was taken one week or one year ago, it is the basis of your predictions and just like a printed photograph, this snapshot will degrade over time. This is not dissimilar to a written policy or business process, the act of creating it is only the first step and the real work is in regularly reviewing and updating it. It is critically important that predictive models be monitored, evaluated, and refreshed on a regular basis. Unfortunately, only a small number of top-level fundraising teams operate under this paradigm.
An Ongoing Business Process, Not A One-Time Project
Every predictive model will degrade over time, regardless of the original accuracy. This degradation occurs for a number of reasons related to the dynamic nature of the data.
- First, as new records are added to the data set, the model’s “snapshot” of your data becomes less and less reflective of the true population. Think of this as a family portrait that was taken before a birth or marriage—that old portrait isn’t an accurate representation of the family anymore.
- Second, as your organization captures new data that wasn’t available when the models were first built, the potential accuracy of the models will suffer. Typical examples include event attendance, social media data, and online giving days. If adding this new data to your models will make them more accurate, the versions you have currently aren’t as accurate as they could be.
As a model’s accuracy degrades, its ability to provide value to your organization becomes increasingly limited. Inaccurate models affect decision making, have negative impacts on ROI and efficiency, and can erode trust in the modeling process itself, all of which will ultimately impact the funds raised by the organization.
When and How to Refresh
Knowing when to refresh your predictive models can be as nuanced as the act of building the models in the first place and requires an intimate level of knowledge about your data. There is no “Golden Rule” or best practice to follow, although if you think it may be time to refresh your models, it probably is. In general, monitoring the following elements over time can give you a good indication as to the timing of a refresh.
- Rate of Data Acquisition/Churn. Being able to quantify how much your data has changed since your models were created is a key factor in determining how stale your snapshot is. Some organizations will enter tens or even hundreds of thousands of gifts in a six-month period—a strong indicator that the giving data used to build an existing model is no longer representative of the current population. Other organizations might not see that many gifts in more than a year and could rely on existing models for longer periods of time.
- Significant Change at the Organization. A major change of direction, leadership, or some other emergent fundraising opportunity could also be a reason to refresh your predictive models. For example, a new initiative to build a new soccer stadium on campus may be reason enough to refresh an existing model focused on capital gifts. It stands to reason that the generic model could be made more specific—and accurate—by re-focusing it on the soccer stadium project explicitly. Likewise, if your board has recently made donor acquisition one of its key priorities, you would do well to think about refreshing an existing acquisition model.
- External Factors. Major changes to the environment external to your organization must also be taken into account when evaluating whether a model has reached the end of its lifecycle. A major economic shift, such as the Great Recession, can have significant implications on your constituents’ likelihood to support your organization. Similarly, elections or other political events may present unique opportunities to your organization and can be strong indications that your models are in need of refreshing.
The “How” of predictive model refresh can be trickier than it sounds. Should your organization simply revise an existing model or re-build it from scratch? Is your model static or scored dynamically on a nightly basis? For all organizations, the question of refreshing predictive models must first start with a re-evaluation of the business objective: What was the need for modeling in the first place? Is that need still valid? Has it changed? You should also evaluate whether new additional models are needed on top of the existing models, and whether they are a higher priority than refreshing the existing models.
All of this can be done by creating a monitoring and evaluation plan to periodically review your models’ effectiveness and quantify your model lifecycles. At the very least, your plan should cover the following questions.
- Monitoring. Are your models still reasonably accurate based on their starting points? How stable are your models over time? Are they prone to large shifts in scoring? Do new records score as expected based on known data?
- Evaluation. What is the model’s ROI, and is the model still producing results for your organization? If the model’s goal is to identify prospects for certain types of gifts, how effective is the model? What gains in efficiency result from the model? Are prospect researchers more effective and efficient when utilizing the model? What is the organizational impact of the model?
For most non-profit organizations, reviewing your monitoring and evaluation plan every 3–6 months is probably sufficient to stay ahead of the naturally occurring degradation experienced by all predictive models.
To learn more about the analytics services offered by Bentz Whaley Flessner, or to discuss your predictive modeling needs with a member of the BWF Insight team, please contact us at firstname.lastname@example.org or (952) 921-0111. Together, we transform philanthropy.
Copyright © 2017 Bentz Whaley Flessner & Associates, Inc.