Collect Evaluation Data
How Will You Collect Your Evaluation Data?
You may not have to start from scratch—there are many questions and instruments that have been developed and tested by other organizations. Sometimes these will work without changes, and sometimes they need to be adapted to fit your evaluation. If existing resources will not work, you can design your own instrument.
Quality Check:
- Accurate – Is your method measuring what you intend to measure? Is it doing so consistently?
- Is your method reliable? Reliability is the degree to which your method produces consistent results over many uses. For example, a reliable survey question would be one where every person who took the survey thought the question was asking the same thing. Or a reliable observation tool would help any observers score or describe what they are seeing in the same way.
- Is your method valid? Validity is the degree to which your method actually measures what you intend it to measure. A simple example is that a ruler is a valid measuring device for length, but would not be valid for measuring weight. Often using more than one method for data collection can help you be more confident that you are measuring what you intend to measure.
- Ethical – Do you have a method to keep people’s information confidential? If the data you are collecting will be sensitive or personal, your process will need to build in safeguards to protect people's privacy and make sure the data are secure.
When Should You Collect Data?
When you collect data depends on what you want to learn and the data collection method you will be using. To answer some questions, you may only need to collect that data at one point during the program. To answer other questions, it may be useful to collect data before and after the program and then compare the results. Most of your data should probably be collected on a regular basis, with enough time between data points to show change over time.
How to Plan and Conduct a Focus Group
Why Conduct a Focus Group?
Focus Group Tool Overview
PLANNING
- Decide when
- Figure out logistics
- Identify the sample
- Recruit participants
- Identify and train a facilitator
- Develop a discussion guide
- Pilot test
COLLECTING DATA
- Use the data collection tips
- Track & document data
- Key follow up steps
Download the How to Plan and Conduct a Focus Group Tool
Focus groups are structured small group discussions with a set of questions that guide the group through a conversation about your program. They are an excellent way to get rich stories and descriptions of how your program is working from a group of people.
Focus groups are most appropriate to use for group process and brainstorming because people can build off of each other’s comments. Because of this, focus groups are often used to test messages, get ideas for program improvement, etc. Good focus groups also leave space for people to share individual points of view, but they are not an appropriate way to discuss topics that are sensitive, controversial, or personal.
Focus Groups Can Help You Explore:
- In-depth information about a particular issue or topic
- The range of perceptions, views and opinions about a particular issue or topic (not just what people agree about)
- Factors that influence opinions or behaviors
Typically, when they are used for evaluation, multiple focus groups are conducted. You could conduct focus groups with several groups of people that all share similar characteristics. You could also conduct multiple focus groups with diverse participants, as long as each focus group itself has similar characteristics or contexts. Then you can compare responses of different groups.
Planning and Developing Focus Groups
Focus groups can help you collect information from multiple people in a short period of time. However, they can be time consuming and difficult to arrange because they require coordinating the logistics of bringing together a group of people in-person and finding a strong facilitator. Be sure to allot time for planning and organizing.
How to Plan and Conduct Interviews
Why Use Interviews?
Interview Tool Overview
PLANNING
- Decide when
- Identify the sample
- Recruit participants
- Identify and train interviewers
- Develop interview protocol
- Pilot test
- Finalize logistics
COLLECTING DATA
- Track & document data
- Key follow up steps
Download the How to Plan and Conduct Interviews Tool
An interview is a one-on-one conversation with someone who has knowledge and experience about the specific topic you are interested in. Because the people who provide the data have perspectives that are important or “key”, interviews for data collection are often called “key informant” interviews.
Interviewing is Useful When:
- The topic is sensitive or people are likely to be inhibited by speaking about the topic in front of others
- People have a limited reading ability
- Bringing a group of people together is difficult
- You need in-depth understanding about complex topics (e.g., community participation, empowerment, or cohesiveness)
- You have a lot of open-ended questions and want to be able to ask for clarification or probe for understanding
How to Plan and Conduct Direct Observation
Why Choose Direct Observation to Collect Data?
Observation Tool Overview
PLANNING
- Decide who, what & when
- Focus
- Select format & type
- Develop a tool
- Identify & train observers
COLLECTING DATA
- Use the tips
- Key follow up steps
Download the How to Plan and Conduct Direct Observation Tool
Direct observation is the collection of information using your senses. By observing, you can document activities, behavior, and physical aspects of a situation without having to depend on peoples’ willingness or ability to respond accurately to questions.
Observation is Useful When:
- You are trying to understand an ongoing process or behavior, or an unfolding situation or event.
- There is physical evidence, or products or outcomes that can be seen.
- Written or other data collection methods seem inappropriate.
Observation can occur in public situations, such as observing peoples’ participation in a training or documenting how people use a community garden. Observation can also occur in more private settings like observing a patient visit in a clinic.
How to Plan and Conduct Surveys
Why Choose a Survey to Collect Data?
Survey Tool Overview
PLANNING
- Define the sample
- Decide how & when to administer
- Develop the questions
- Pilot test
COLLECTING DATA
- Distribute & collect
- Monitor & record responses
- Calculate response rate
- Key follow up steps
Download the How to Plan and Conduct Surveys Tool
A survey is a questionnaire (written or verbal) that is administered verbally or in writing to your focus population. Surveys are a good method to use to learn about:
- Knowledge—what people know or how well they understand something
- Beliefs—people’s attitudes, opinions
- Behaviors—what people do
- Attributes/demographics—who people are
In comparison to many of the other qualitative methods (interviews, focus groups, and observations), surveys provide more standardized and consistent data that can be more easily combined and analyzed. Surveys can help you to gather both qualitative and quantitative information. Also, survey responses can be anonymous, which is very helpful if you are asking about sensitive or personal information.
Case Study
The Health Connect program’s evaluation plan included a variety of measures that required data collection before, during, and after the program.
Some data were collected to answer questions about how the program was working. Other data were collected to understand the program’s outcomes, such as improvement in health behaviors related to healthy eating and physical activity.
The program team was also able to use existing electronic health record data for their long-term outcomes.
Below are examples of their data collection at different time points.
Baseline
Before delivering new program services, pull data on health outcomes related to diabetes for new clients, using existing electronic health record data.
During the Program
Collect program participation data, (e.g., number of clients served) and interview clients to see if their goals and action plans are useful.
After the Program
Once the program is complete, leverage existing electronic health record data to measure improvement in health outcomes related to diabetes and compare to baseline.
