Design the Evaluation
What is an Evaluation Plan?
An evaluation plan flows from your logic model and helps you think through how to measure progress and achievements. It’s the blueprint for your evaluation. It lays out what you need to know to implement your evaluation and how to get that information.
- The specific evaluation questions you want to answer to know if your program is effective
- The measures or indicators you will use to know if you are moving toward your goals
- The practical and realistic data sources and methods you will use to get the information to answer your questions
- The people who will help you collect your data and the timeline and budget that will guide your work
An Evaluation Plan Ensures Your Evaluation Is:
How Do You Design an Evaluation Plan?
We’ve broken the steps of designing an evaluation plan into two parts on the next few pages. Each part has a tool with more information, shows an example in practice, and provides a template to work on each step.
Part 1: Develop
Design the Evaluation, Part 1: Develop Evaluation Questions and Indicators
What Are Evaluation Questions?
Developing evaluation questions is the starting point for a strong evaluation. They help you to focus in on the program elements that you and your stakeholders think are the most important to understand if your program is effective.
Evaluation questions should flow easily from your logic model:
- Process questions ask about implementation, or how the inputs, activities and participation sections of your logic model are working. These questions will help you understand whether the program is being implemented according to your plan and identify opportunities for improvement.
- Outcome questions ask about whether you have accomplished your outcomes as described in your logic model. These questions will help you understand progress and changes that are made as a result of your program. For example, have there been changes in people’s knowledge or health behaviors as a result of your short term outcomes?
An evaluation can ask both process and outcome questions. Typically your evaluation will focus more heavily on process questions in early years, but it’s important to start collecting data for outcome measurement early on.
Over time, the accumulation of outcome data will allow you to answer your outcome questions.
Case Study
The Health Connect program team sat down with their logic model and brainstormed a list of evaluation questions. It was tough to prioritize them, but they focused on those that would be the most useful and feasible to answer.
They chose a few indicators for each evaluation question and tried to pick a mix of qualitative and quantitative indicators to more completely answer the question.
Download the Design the Evaluation, Part 1: Develop Evaluation Questions and Indicators Case Study
What Are Indicators?
The next step is determining how to answer your evaluation questions. To do this you need to identify indicators, which are the evidence that you are making progress. They answer the question "if the outcome is achieved, how will we know it?"
Indicators also can help you see where you are making progress if you cannot yet show your program’s impact on the longer-term outcomes.
If this step is skipped, it can be difficult to make sure you pick the right data collection method to actually answer your questions. Here are some examples:
Walkability
An indicator of a residential area's walkability is the proportion of streets that have sidewalks.
Children's Health
An indicator of children's health status is the percentage of those who are obese.
Academic Achievement
An indicator of academic achievement is the high school graduation rate.
Part 2: Plan
Design the Evaluation, Part 2: Plan for Data Collection
What is Data Collection?
The next step in creating an evaluation plan is picking the source that will provide the best information, while also thinking about how easy or difficult it will be to get that information. Often there is more than one source for data and more than one way to collect it from that source.
What Data Sources Should We Use?
The data you use will depend entirely on your evaluation questions. Evaluation data generally come from three sources:
People
Clients, program staff, funders, staff of other organizations, community leaders, elected officials, general public, critics, etc.
Documents and electronic systems
Tracking sheets for service delivery, grant proposals, meeting minutes, publicity materials, existing data, previous evaluation reports, electronic health records, etc.
First hand observation
Directly observing and documenting the program's setting or key features, or how things work in practice like personal interactions and program activities
What Methods Should You Use to Collect Your Data?
Once you have explored potential data sources, the next step is to choose which method will be most effective. You can:
- Use existing data. This includes data already published or collected in the past by other parties, as well as data previously collected by your organization for other purposes. You might use existing program documentation, electronic health records, or local secondary data from sources like CottageHealth.Data2Go.
- Collect new data. Data you collect yourself (or have someone collect on your behalf) are called primary data. This might include interviewing people, holding focus groups, using online or phone surveys, or observing the program in action. We will walk through all the typical methods used to collect evaluation data in the next step - Collect.
How Do You Plan Your Data Collection?
Consider these factors to select the best data sources and methods for your evaluation:
- What are the advantages and disadvantages of different sources and methods?
- How intrusive will it be to collect the data in this way?
- Is there an existing data source you could use?
- Is this a culturally appropriate data collection method?
- Who will collect the data and when?
Case Study
The Health Connect program team brainstormed possible data sources and methods for each of their indicators and then prioritized which would work best for their evaluation.
For example, the program team wanted to know who the Community Health Worker (CHW) program serves. One indicator is the number of unduplicated clients. The team determined that these data were already available through program documents like intake forms and the CHW schedule.
As part of their evaluation, they would conduct a document review to establish the number of clients served.
Download the Design the Evaluation, Part 2 – Plan for Data Collection Case Study
