Evaluation Questions: A Guide to Designing Effective Evaluation Questions - EvalCommunity (2024)

Monitoring and evaluation (M&E) is an essential part of any project or program. It helps us understand the progress and impact of our efforts and identify areas for improvement. But what are the key evaluation questions to ask in order to effectively monitor and evaluate a project or program? In this article, we’ll look at some of the main questions to consider when setting up an M&E system.

Table of Contents

• What are evaluation questions?
• Understanding the Purpose of Evaluation Questions for M&E
• How do you develop questions for your evaluation?
• Crafting Appropriate Questions for the Evaluation
• 3 Types of Evaluation Questions
• Write evaluation questions with your stakeholders
• Evaluation questions for the main types of evaluation
• Process evaluation Questions
• Outcome evaluation (or impact evaluation) Questions
• Economic evaluation (cost-effectiveness analysis and cost-benefit analysis) Questions
• Appropriateness, effectiveness, and efficiency
• Analyzing the Results of Your Evaluation Questions

What are evaluation questions?

The evaluation questions are the high-level questions that an evaluation is designed to answer.It is important to distinguish between evaluation questions and interview questions. Evaluation questions are the high-level questions that an evaluation is intended to answer, while interviews or questionnaires ask specific questions.

Evaluation questions are a key component of the monitoring and evaluation process. They are used to assess the progress and performance of a project, program, or policy, and to identify areas for improvement. Evaluation questions can be qualitative or quantitative in nature and should be designed to measure the effectiveness of the intervention and its impact on the target population.

Evaluation questions should be specific, measurable, achievable, relevant, and timely. They should also be framed in a way that allows for comparison of pre-and post-intervention data. By using evaluation questions, organizations can ensure that the monitoring and evaluation process is comprehensive and effective.

In addition to helping focus your evaluation, evaluation questions should be created so that they reflect not only the purpose of the evaluation but also the priorities and needs of the stakeholders involved in the evaluation.

Understanding the Purpose of Evaluation Questions for M&E

Evaluating data collected through M&E processes is an important part of understanding and measuring the success of a project or program. When forming evaluation questions, it’s important to consider the scope of the project, the intended objectives and outcomes, and any resources available for research or data collection. Evaluation questions should also be structured to allow for a simple yes or no response, or a multiple-choice format if appropriate. The answers to these questions can then be interpreted and used to inform decisions on how to refine or improve projects in the future.

Evaluation questions should be designed to help assess the effectiveness of a program by capturing feedback on whether it is achieving its desired outcomes, identify potential areas for improvement and guide decision-making moving forward Also, evaluation questions should be planned in advance to make sure that they accurately reflect the program’s goals and help to determine the success of any changes implemented. They should be designed to capture feedback on whether the program is achieving its desired outcomes, identify areas of improvement and provide insights that help inform the decision-making process. Evaluation questions are an important tool for understanding the effectiveness of a program’s design and implementation.

How do you develop questions for your evaluation?

When it comes to developing questions for your evaluation, it is important to focus on the specific objectives you are trying to measure. Every question should be designed to assess the success of the project or program.

Start by brainstorming a list of potential questions that you think would be relevant to the evaluation. Once you have a list of potential questions, review each one and decide if it is relevant and appropriate. Consider the type of response you are looking for when crafting each question. Make sure to keep the questions as clear and concise as possible.

Finally, test the questions with a small sample of people to ensure that the questions are being understood correctly. By taking the time to develop meaningful questions for your evaluation, you will be able to obtain accurate and valuable. Here are some steps to point out:

• Write evaluation questions with your stakeholders. Ensure that your logic model has been reviewed by key stakeholders and involves key stakeholders
• Brainstorm Evaluation Questions
• Classify your questions into different categories
• Determine what questions should be prioritized for evaluation

Crafting Appropriate Questions for the Evaluation

When crafting appropriate evaluation questions for an assessment, it’s important to consider the purpose of the evaluation and the type of data you want to collect. Questions should be clear, direct, and easy to understand while also being applicable to the desired outcome. Depending on the type of evaluation you are conducting, consider if you should ask open-ended or closed-ended questions. Open-ended questions can provide more detailed and nuanced answers, whereas closed-ended questions can provide quantitative data that can be easier to analyze. Additionally, it is important to consider if the evaluation questions should be tailored to different groups within the population being evaluated so that results are more reflective of specific segments of the population. By taking these factors into consideration when crafting evaluation questions, you can ensure that you are able to gain meaningful insight from your assessment.

For example, when evaluating a training program, open-ended questions can be used to capture qualitative feedback from participants, while quantitative questions should be used to collect direct assessments of learning or performance measures. Again when evaluating a training program, it is important to consider both qualitative and quantitative assessment. Utilizing open-ended questions can help capture the participant’s subjective experience, while quantitative questions can be used to measure direct outcomes of the training. This dual approach to evaluation will provide the most accurate and comprehensive assessment of the program.

An evaluation process should also consider if the questions are relevant to the topic being evaluated, whether they are clear and unbiased in their terminology, and if they are structured in such a way that answers can be accurately collected and analyzed. Taking into account these factors will allow for more successful evaluation questions that yield reliable results.

3 Types of Evaluation Questions

Descriptive questions. Represent “what is”

Examples – Descriptive Questions

• What are the primary activities of the program
• What do stakeholder groups see as the goals of the program?
• Where has the program been implemented?
• Who received what services?

Normative questions – ” Comparisons of “what is” to “what should be”

Examples – Normative Questions

• To what extent was the budget spent efficiently?
• Did the project spend as much as was budgeted?
• To what extent was the program gender equitable?
• To what extent was the target of vaccinating 90% of the nation’s children met?

Cause and Effect questions – Identify if results have been achieved due to the intervention

Seek to determine what difference the intervention made • Eliminate all other possible explanations • Ask if the desired results have been achieved AND whether it is the intervention that has caused results • Suggest before & after and with & without comparisons • Impact Evaluations focus on cause and effect questions.

Examples – Cause and Effect Questions

• As a result of the job training program, do participants have higher-paying jobs than they otherwise would have?
• Did the three-country partnership strategy preserve the biodiversity of the affected area while sustaining livelihoods?
• Did the increased tax on gasoline improve air quality?

Write evaluation questions with your stakeholders

It’s important to involve stakeholders in the development of evaluation questions to ensure that their perspectives and priorities are incorporated into the evaluation design. Stakeholders may include program staff, participants, funders, community members, and other relevant stakeholders.

Here are some examples of evaluation questions that could be developed with stakeholders:

1. Did the program achieve its intended outcomes?
2. How well did the program meet the needs of its target population?
3. What were the strengths and weaknesses of the program design?
4. How effective were the program’s strategies and activities in achieving its objectives?
5. How did the program impact the community or larger system in which it operates?
6. To what extent did the program address issues of equity and inclusion?
7. What were the key barriers or challenges to program implementation and how were they addressed?
8. How well was the program monitored and evaluated throughout its implementation?
9. What lessons were learned from program implementation that could inform future programs or initiatives?
10. Were the program’s resources allocated effectively and efficiently to achieve its objectives?

Evaluation questions for the main types of evaluation

Process evaluation questions

Process evaluation, or how the program addresses the problem, what it does, what the program services are and how the program operates. Process evaluation questions focus on how a program is working, program performance, and involve extensive monitoring. Similarly, formative evaluation questions look at whether program activities occur according to plan or the project is achieving its goals while it is underway. Some sample questions are:

Some sample questions are:

1. How is the program being implemented? Is the program being implemented correctly?
2. What are the underlying assumptions of the project/ program?
3. Are objectives met? If so, how? If not, why not?
4. Are activities conducted with the target population?
5. How appropriate are the processes compared with quality standards?
6. Are there other populations the program should be working with?
7. Is the target population adequately reached by and involved in activities?
8. Are participants being reached as intended?
9. How does the target population interact with the program?
10. What do they think of the services? How satisfied are clients?
11. How is the project functioning from administrative, organizational, and/or personnel perspectives?
12. What has been done in an innovative way?

Outcome evaluation (or impact evaluation) questions

4.Impact/ outcome evaluation, or how does the program reach its outcomes or impact? The evaluation questions may also be used in summative evaluations which focus on what happened after the program or project completed, i.e., were goals achieved? And what can be learned? Some sample questions are:

1. What are the outputs, outcomes, objectives, and goals of the project?
2. Are outcomes, objectives, and goals achieved?
3. Are the project/program services/activities beneficial to the target population?
4. Do they have negative effects? e. Is the target population affected by the project/ program equitably or according to the evaluation plan?
5. Is the problem that the project/ program intends to address alleviated?

How well did the program work?

Did the program produce or contribute to the intended outcomes in the short, medium and long term?

For whom, in what ways and in what circ*mstances? What unintended outcomes (positive and negative) were produced?

To what extent can changes be attributed to the program?

What were the particular features of the program and context that made a difference?

What was the influence of other factors?

Economic evaluation (cost-effectiveness analysis and cost-benefit analysis) questions

Assessment of efficiency, or how cost-effective is the program. Sample questions are:

1. Is the cost of the services or activities reasonable in relation to the benefits?
2. Are there alternative approaches that could have the same outcomes with less cost?*

Through asking questions, M&E practitioners can identify what their project specifically should address. According to Owen and Rogers (1999), there are three levels of evaluation questions at this stage in project planning:

1. Policy level – how does, or could, the evaluation impact relevant policy?
2. Program level (regional, large scale, “Big P”) – how does, or could, the evaluation effect program changes?
3. Project level (local, activity based, “little p”) – how does, or could, the evaluation effect project or local changes?**

The best questions must be developed with stakeholders in the evaluation, including program staff, sponsors and funders, local and regional decision-makers within and outside the program, and community representatives, when the community in which the evaluation or project will be carried out has already been identified. These consultations may be informal conversations, reviewing grant requirements and terms of reference (documentation review), or semi-structured individual and/or group interviewing. The evaluator consults with all accessible stakeholders to develop specific questions that the evaluation will seek to answer. According to Rossi, Freeman, and Lipsey (1999), evaluation questions must be:

• Reasonable and appropriate, or realistic in the given project or program.
• Answerable, similar to the reasonableness of a question, good evaluation questions must be able to be answered to some degree of certainty. If questions are too vague or broad, or require data that is unavailable or unobservable, they are not answerable.
• Based on program goals and objectives.

Once we have developed the larger question of the project, M&E practitioners begin to consider what data they need to answer the question, using a Theory of Change and aLogFrame.

• What has been the ratio of costs to benefits?

• What is the most cost-effective option?

• Has the intervention been cost-effective (compared to alternatives)?

• Is the program the best use of resources?

Appropriateness, effectiveness, and efficiency

Three broad categories of key evaluation questions are often used to assess whether the program is appropriate, effective and efficient .

Organising key evaluation questions under these categories, allows an assessment of the degree to which a particular program in particular circ*mstances is appropriate, effective and efficient. Suitable questions under these categories will vary with the different types of evaluation (process, outcome or economic).

Appropriateness

• To what extent does the program address an identified need?

• How well does the program align with government and agency priorities?

• Does the program represent a legitimate role for government?

Effectiveness

• To what extent is the program achieving the intended outcomes, in the short, medium and long term?

• To what extent is the program producing worthwhile results (outputs, outcomes) and/or meeting each of its objectives?

Efficiency

• Do the outcomes of the program represent value for money?

• To what extent is the relationship between inputs and outputs timely, cost-effective and to expected standards?

Analyzing the Results of Your Evaluation Questions

A key component of evaluating the results of your evaluation questions is to review the responses you received to see what patterns emerge. Doing so can provide valuable insights such as areas of strength, weaknesses, and potential improvement opportunities. To ensure you are collecting the most useful data possible, it’s important to ask questions that are well-crafted, relevant, and focused on the end result you’re hoping to achieve. Additionally, providing a variety of question formats can be beneficial in ensuring you capture responses from participants in different ways. Finally, having a clear plan for how you will review and analyze the responses you receive is key to ensuring your evaluation questions are effective.

Studying these patterns that are revealed in the responses can help inform a better understanding of your evaluation questions and their effectiveness in obtaining useful feedback from your respondents Meanwhile, it is important to consider the effects of the evaluation questions and analyze the patterns that are shown in the responses. This can help to inform a better understanding of the evaluation questions and their effectiveness in obtaining useful feedback from respondents. By studying these patterns, businesses can ensure they are obtaining meaningful feedback which will help them to improve their services over time.

Next topics on this discourse:

• Understanding the Importance of Evaluation Questions in Program Evaluation
• Types of Evaluation Questions and How to Choose Them
• Best Practices for Designing Effective Evaluation Questions
• Methods for Collecting Data from Evaluation Questions
• Analyzing and Interpreting Data from Evaluation Questions
• Using Evaluation Questions to Improve Program Outcomes
• Common Challenges in Designing and Using Evaluation Questions
• Examples of Effective Evaluation Questions in Program Evaluation