Evaluating Outputs and Outcomes

What is it?

Simply put, evaluation assesses whether or to what degree a given effort is achieving its intended outcome. For hunting and shooting sports R3, the ultimate intended outcome is to maintain or increase participation in and support for hunting and shooting sports. For more details and background about R3 outcomes, see the chapters on R3 Foundations and Definitions and Planning and Strategy.

In the Planning & Strategy Chapter, we learn how to design R3 efforts to help participants through the Outdoor Recreation Adoption Model. In this chapter, we learn how to evaluate whether or not those programs meet their objectives.

Evaluation is a systematic process determining if your programs achieve their intended purpose.

 Evaluation results can be used to:

• Show successful outcomes to your stakeholders.
• Build additional support for R3 programming.
• Identify specific ways to improve implementation.
• Make strategic decisions about which programs to continue, modify, or discontinue.

It is common for practitioners and administrators to lose sight of evaluation as they focus on program development and implementation. However, if applied early and consistently, evaluation will help you collect the data you need to:

1. Stop conducting programs/efforts that aren’t achieving the outcomes you seek.
2. Do more programs/efforts that are working (or try new efforts that might work).
3. Make adjustments to programs/efforts that show promise.

Why is it important?

The R3 community has been implementing programs with no systematic evaluation for decades. That is, the R3 community has been doing all manner of “stuff” with little or no idea whether it is working. With hunter numbers declining, the community can’t afford to continue with this approach.

Recent research¹ shows that hunting license sales will continue to decline as the license-buying population declines due to the effects of aging and other social, cultural, and demographic changes.

The situation for shooting sports is slightly different. The number of people purchasing and using firearms is growing, not declining. However, there is no obvious metric like license sales to measure overall participation. The need to carefully identify R3 objectives and measure progress against them is even more acute.

How can your organization help its practitioners?

• Insist that R3 efforts be evaluated against specific objectives to ensure effectiveness.
• Give the R3 Team the resources and authority to conduct meaningful evaluation.
• Ensure staff have access to data needed to measure outcomes (dashboards, CRMS, license sales data, customer demographic information, marketing campaign evaluation results etc.).
• Work toward implementing a full Customer Relationship Management (CRM) system (or at least collaboration among the agency divisions that touch customers).
• Ensure your state agency is participating in the Real-Time Hunting and Fishing License Data Dashboard and your state agency or organization uses an internal dashboard to track sales/engagement at the state/organization level.

Other types of Evaluation

There are specific evaluation methods and metrics for many of the disciplines that combine to form R3. Each of these chapters of the guide has an evaluation section of its own. Here are links to those other sections of the guide if you are looking for that specific assistance:

Create a Plan Format and Process

Revising your plan over time

Marketing Evaluation

How to Evaluate Partnership Engagement

How to Evaluate and Refine Your Education and Outreach Initiatives

What are you talking about? – Definitions

A big challenge the R3 community has grappled with is identifying what, specifically, constitutes an R3 effort. Too often, R3 practitioners inherit or are directed to implement programs or efforts that aren’t really R3 at all. There will always be political, economic, and other realities that practitioners must deal with, i.e., you may have to complete tasks that are not R3. Identifying the programs and efforts that are and are not R3 is the first step in prioritizing where you can most effectively spend your evaluation resources.

Additionally, there are terms and concepts used in the R3 community without common definitions. Some of them are measured in different ways. This makes it very difficult to evaluate efforts effectively and/or compare one effort’s results to another.

Working with R3 practitioners from around the country, the Western Association of Fish and Wildlife Agencies is leading an effort to define R3 terms and concepts more clearly². Practitioners would do well to read the paper “R3 Metrics” in its entirety. Key elements follow.

Overall, the purpose of R3 efforts is to maintain or increase participation in and support for hunting and shooting sports. This is most often done by influencing or modifying human behavior, which can be specifically defined according to the three Rs.

Recruitment – A behavioral influence (from an R3 effort or other external influence) resulting in the initial choice to participate in a target activity.

Retention – A behavioral influence (from an R3 effort or other external influence) resulting in continued year-to-year participation in a target activity.

Reactivation – A behavioral influence (from an R3 effort or other external influence) resulting in the renewed participation in an abandoned target activity.

Thus, most R3 efforts should be viewed as interventions or actions needed to initiate, support, or re-establish an individual’s adoption of desired behaviors.

An individual’s or group’s adoption of a new behavior is likely dependent upon two things. One is how well the behavior aligns with the individual’s existing values, attitudes, or social norms. The second is how well an R3 effort addresses the unique barriers (real and perceived) that restrict a particular audience from participating in the behavior³. Ideally, R3 efforts should be designed and implemented to address the specific barriers and subjective learning needs of a particular group or demographic. If the effort is designed without an understanding of the audience’s values, beliefs, and barriers to participation, it may fail to motivate individuals to participate in the future.

More to the point, given the diversity of those likely to engage in an outdoor activity, it is imperative that R3 efforts be evaluated for effectiveness. R3 practitioners need data illuminating how and when to modify their R3 efforts to increase efficacy and efficiency. This underscores the need for adaptive program design and implementation of R3 efforts that incorporate an evaluation system capable of documenting participants’ behavioral changes directly attributable to the R3 effort.

Plan for Evaluation From the Beginning

One of the primary reasons so few R3 efforts have been evaluated in any meaningful way is the fact that very few organizations think about evaluation until the effort is well underway or completed. To be effective, every R3 program or action should have evaluation woven into it from the very beginning. For every program, ask what is this program’s objective—what, specifically, is it trying to achieve? How does this program help us meet the overall R3 outcomes we seek?

Review the chapter Planning and Strategy for guidance on setting meaningful objectives. Defining your evaluation objectives up front establishes guidance for what questions about the program need to be answered. It helps to decide how to collect data to track metrics, how these data are analyzed, and how the results will be used and shared.

Even if your agency or organization has been doing a program for many years, if it doesn’t have a specific objective identified, stop everything, and assign an objective(s). Otherwise, you will have no way of knowing whether that program or effort is worth the time and effort you are putting into it. Sometimes, when you attempt to attach a specific objective to a program, you realize that it is not really R3 after all. Or, maybe it makes people feel good about your agency/organization or “makes us more visible in the community” or something along those lines. Perhaps the public relations objective is essential, and you may decide to continue doing it for that reason. If a program is not achieving R3 outcomes, recognizing that and calling attention to it is the first step toward making your true R3 efforts effective.

Once you have suitable objectives, the next step is to identify which metric you will measure. This determines whether you have been successful, to what degree you have been successful, and how often it will be measured. After you set objectives and metrics, the real work of evaluation can happen. This means looking closely at results over time and acting on the results. Adjust or eliminate the things that aren’t working, so you can do more things that make a difference.

Often, it is difficult to determine when enough is enough. As the plan develops, the amount of information wanted tends to grow larger and larger. This can lead to data collection instruments that are too long or demand staffing and fiscal resources you don’t have. Asking yourself who this evaluation is intended for and what information will be most useful to them will make the task of narrowing your focus more manageable. Is the evaluation intended to provide information to R3 practitioners, funding partners, agency directors, or commissions? An evaluation for program staff may look very different than one intended for funding partners or directors. What information is most meaningful to them? Different types and quantities of information will be more or less valuable in different situations.

Outputs and Outcomes

The difference between outputs and outcomes is important in evaluating R3 efforts. Simply stated, outcomes are the ultimate ends we seek from our efforts, and outputs are what we accomplish to achieve those outcomes. Most agencies and organizations select some variation of the following ultimate outcomes for their R3 programs:

1. Maintain/increase participation in hunting/shooting activity and/or license purchases.
2. Increase public acceptance/support of hunting/shooting activity.

Nearly all workshops, activities, communications, and other efforts you do to get people to do these things are outputs. Far too often, when agencies and organizations set out to evaluate their programs, they only measure outputs. Things like the number of programs delivered in a year, the number of participants going through programs, the number of training videos produced, clicks on websites, online impressions, and so on. Very few of these efforts lead directly to the outcomes we seek. For instance, it usually takes many interactions over time and across a variety of topics to create an independent hunter or shooter. While measuring outputs is very important, it is even more important that you do not stop there. It is insufficient to have dozens or even hundreds of attractive outputs if it doesn’t achieve the ultimate outcome. These are all part of effective R3 program evaluation. Ideally, you should measure as many outputs as possible to be sure they are tracking in the right direction. Then measure your outcomes to be sure the rest of it makes a difference.

Set yourself up for success. Are your outcomes and outputs realistic–can you achieve them in the amount of time you have? Are your outcomes about change–how will the participants be different because of the program? Can you think of existing or new ways to measure these outcomes?

Results Chains

Results chains are foundational to evaluation. They provide a framework for designing evaluation activities. A results chain visualizes the step-by-step process leading from a program’s inputs to the outcomes. It shows how the effort is intended to work. It will help you ensure that your activities align with your intended outcomes. It also helps identify appropriate metrics for measuring the success of your efforts.

Inputs are what we invest in our efforts. They include staff and volunteer time, money, equipment, and facilities. Outputs are what you use the inputs to create. They are the products and services you deliver and who you reach with these products and services. Finally, outcomes are the results of your outputs. You will primarily be interested in two types of outcomes: effort/program-specific outcomes and R3 outcomes. Effort- or program-specific outcomes look at how the output affects participants’ awareness, knowledge, attitudes, skills, intentions, or motivations. R3 outcomes are the ultimate goal. They are how the effort or program results in R3 behaviors (recruitment, retention, reactivation).

Suppose your program or effort already has a results chain developed. In that case, it should be the starting point for planning your evaluation. If you don’t already have one, creating a results chain involves visually representing the sequence of events that leads from program inputs to outcomes.

Start by defining the inputs, representing the resources (personnel, equipment, etc.) that go into a program or effort. Layout your planned activities that will be implemented. From here, you will identify these activities’ direct and tangible results. These should be specific and measurable. Incorporate key indicators at each stage of the results chain. These will help you measure progress and success. You will need to clearly articulate the effort-specific outcomes and your R3 outcomes. Remember, your effort-specific outcomes may not represent the desired R3 outcome but can reflect progress toward desired R3 outcomes. Typically, several effort-specific outcomes must be achieved before achieving your R3 outcome. You may find that participants will need additional interactions beyond a single program or effort. When building your results chain, clearly demonstrate how each step contributes to the next stage of the chain.

As you lay out the results chain, ensure a logical flow from inputs to outcomes. Each step in the chain should make sense in the context of the program’s theory of change and goals. A theory of change is a systematic description of how and why a desired change is expected to happen in a particular situation (e.g., the Outdoor Recreation Adoption Model). It articulates the underlying logic and assumptions of a program.

From your results chain, you should be able to fill in the blanks below:

If we do (proposed activity), then based on (theory of change), we can logically expect (result) to happen as evidenced by (indicator).

The following is an example of what a results chain for a shooting sports R3 program might look like.

If you would like a more in-depth discussion about results chains, the Dunfee and Pedder 2017 webinar on R3 evaluations is a great place to start.

Don’t try this alone!

Program evaluation often requires a range of skills and knowledge areas. It includes research design, data collection and analysis, subject matter expertise related to the program, and understanding the context in which the program operates. Just as most marketing specialists shouldn’t assume they could conduct a research project on turkey population ecology, most R3 practitioners should not assume they can design and implement effective program evaluation. Specific expertise is required. Establishing an evaluation team is important for conducting a thorough, objective, and credible program evaluation. Include staff from outside the R3 program (e.g., licensing, social science, marketing/communications, GIS, etc.) who can critically support the evaluation process. The combination of diverse skills, perspectives, and expertise enhances the quality of the evaluation process and its outcomes.

Part of the work of an evaluation team is to determine the staff resources, data systems, abilities, and fiscal resources needed and available for the evaluation. Your team might be able to support all your evaluation needs, but if you are stuck or need an outside perspective, seek input as early as possible. Find a social scientist, an experienced evaluator, or a subject matter expert who can give advice and notice problems early in the process. More and more fish and wildlife agencies and conservation organizations have social scientists on staff. But they are often spread thin and working across many divisions and projects. Engaging them early in the process and articulating your program’s objectives and outcomes will help them understand how to provide help. These steps are critical. Depending on your budget, you may be able to hire a consultant to assist with some or all of the process. They may help frame the research questions, develop evaluation instruments, assist implementation, and/or perform data analysis and reporting.

Common R3 Metrics

The ultimate R3 outcome most organizations seek is participation in hunting and the shooting sports. For hunting, the obvious metric is license sales. Not everyone who buys a license participates, and not everyone who participates is required to buy a license. But the correlation is high, and license sales are a well-accepted proxy for hunting participation. License sales (or sales resulting from specific interventions) are one of the primary ways agencies and organizations can assess the success of hunting R3 efforts.

Following are some metrics commonly used to assess changes in license sales over time. Some are straight counts; others are percent change, where percent change is calculated:

Recruitment – A behavioral influence (such as from an R3 effort) resulting in the initial choice to participate in a target activity.

• Attain X number of unique new permit holders by [date].
• Increase by X percent the number of unique new permit holders by [date].
• Increase participant recruitment rate X percent by [date].

To be considered recruited, an individual should never have had a previous license (or range permit) issued by your organization.

Retention – A behavioral influence (such as from an R3 effort) resulting in continued year-to-year participation in a target activity.

• Increase by X percent the number of unique new permit holders who maintain those permits for Y years by [date].
• Reduce by X percent the number of participants who have lapsed for Y years by [date].
• Increase participant retention rate X percent by [date].

Individual organizations could define the number of years a person has to renew their license (or range permit) to consider them in your retention rate. The national dashboard calculates the retention rate as the percentage of prior year license/permit holders that renewed their license/permit in the reporting year.

Reactivation – A behavioral influence (such as from an R3 effort) resulting in the renewed participation in an abandoned target activity.

• Increase by X percent the number of lapsed participants who purchase a hunting license/range permit in the current year.
• Increase participant reactivation rate X percent by [date].

Individual organizations could define the amount of time a person has to be without their license (or range permit) to consider them in your reactivated when they purchase a new license/permit. The national dashboard considers individuals reactivated if they missed at least one entire year and then renewed their license/range permit.

Although the above metrics measure the primary R3 outcomes most agencies seek, they don’t tell us whether the outputs (our R3 programs and efforts) positively contribute to these outcomes. For that, we need additional proxy metrics. For instance, if we have a program designed to recruit new hunters (and we collect information that will allow us to match them to our license database), we could calculate a program conversion rate, that is, the proportion of first-time license sales purchased by our program participants (X unique, new participants in R3 programs account for Y percent of first-time license sales by [date]).

This still isn’t a direct metric because we can’t know whether our program or R3 effort was the cause of the recruitment. But it is reasonable that if this proportion is high, either our R3 effort is having some effect, and/or people participating in R3 programs are already more likely to purchase a license for the first time. We could use self-reported data (e.g., post-event survey) to learn more about this possible effect.

Although it isn’t perfect, this kind of evaluation can tell us a lot about whether or to what degree our R3 efforts result in the behavior change we seek.

License Dashboard

Thanks to a Multistate Conservation Grant sponsored by the Association of Fish and Wildlife Agencies, state agencies can now see their license sales trends at a higher scale. License dashboards were created to allow state agencies to view and share real-time license sales information in a standard format. Since license sales are one commonly used indicator of participation, understanding regional and national license sales trends helps agencies better understand their operating environment, and whether these trends exceed or underperform against set goals or regional averages. With these insights, states can identify when changes are needed or if current R3 efforts are on track. These kinds of real-time data are what the industry uses to make marketing and sales decisions on a daily basis. Now that this technology is in place for tracking license sales, state agencies should do the same.

Primary reasons agencies are encouraged to participate:

• Data Confidentiality: Participation in the real-time dashboard does NOT require the sharing of license buyers’ personal identifying information (PII).
• Frequency: Daily license sales are collected and reported in historical (ten-year), cumulative (year-to-date), monthly, and daily visualizations. Sales data undergo a standard, rigorous normalization process to ensure accurate and reliable hunting and fishing participation insights.
• With the shift to API technology, demands on state agency staff are minimal.
• Cost: The dashboards are funded by an AFWA multistate grant and FREE to all state agencies.

Click here to see the latest results from participating states. Go here for more information. If your state is not currently participating, contact Lisa Parks ([email protected]) for additional details or to sign up.

How to Identify Meaningful (and Reasonable) Evaluation Techniques

There are many different ways that R3 efforts can be evaluated (assuming specific objectives have been set). The trick is selecting techniques that give meaningful results at a reasonable cost. Data come in a variety of forms. Regardless of the manner of collection, data must be gathered and organized before analysis can begin. Before you start, ask yourself what the objectives of the evaluation are, what type of data you are collecting, and what methods you will use to gather, organize, and analyze the data. Answering these questions will help you avoid a very common problem—being unable to answer your evaluation questions. Accurate data are necessary to draw correct conclusions and maintain research integrity. If you are unsure if a technique suits your needs, seek help from a social scientist or experienced evaluator.

Ethical Considerations

Data collection involves gathering information from your program participants. This makes ethical considerations paramount to ensure data privacy, informed consent, and minimizing the potential impact on participants. There are online resources that can help you ensure that your evaluation is conducted with integrity and sensitivity. If your evaluation team includes a social scientist or experienced evaluator, they should be well-versed in this area as well.

Administrative Data – License Sales

If your evaluation efforts have license sales or range permit sales as a metric, external data may exist for measuring impact. License sales data and program-related records often include numbers and statistics that you can use to track programs and participants.

The points below are a general guide on using administrative data to evaluate an outcome:

• Define the outcome you seek and how you will measure it.
• Identify relevant administrative data sources, including your agency’s license sales database, license dashboard, a CRM system, and other program-specific records.
• Prepare and clean the data – This involves handling missing or duplicate data, correcting errors, and standardizing formats. This step is crucial for the reliability of your analysis.
• Link data sources – If your outcome metric requires information from multiple sources, you may need to establish a unique identifier to link data from different sources.

At this point, you can analyze, interpret, and share the results.

Primary Data Collection

If your evaluation efforts do not have license/permit sales as a metric and no relevant administrative data are available, you will need to collect data directly from participants. Following is a general overview of quantitative and qualitative methods. If your evaluation team lacks the expertise for social science primary data collection, consult with a social scientist or experienced evaluator. Poor design often leads to bad data, and bad data are often worse than no data at all.

You need to collect quantitative data for evaluations looking to precisely measure results or gain large-scale statistical insights. If the hope is to explore ideas, understand experiences, or gain detailed insights in a specific context, collect qualitative data. Often, we hope to do both, in which case we can use a mixed methods approach and collect both quantitative and qualitative data.

Qualitative Versus Quantitative Research

It’s impossible to determine which research technique is best for you without understanding your situation, available resources, and objectives. Social science research requires a complex skillset. This chapter of the Practitioner’s Guide is designed to give you enough of an understanding of the process to coordinate with your organization’s social science specialists or hire a consultant rather than doing the research yourself. However, if you have the background to do research yourself, the following sections will help you with the specifics. If you aren’t going to do your own research, you can probably skip these sections: