• April 13, 2022
• 2

Part 1 — How do you know the bread is any good?

This blog is part 1 of a two-part series, introduced here.

---

What do baking bread and program evaluation have in common? More than you might think. During the pandemic, we started a new family tradition, baking homemade bread on Sundays. One Sunday, I pulled out the recipe and ingredients like I had done dozens of times before. I read through the recipe, mixed the ingredients, and put the bread in the Dutch oven. However, much to my surprise, instead of the delicious tender bread my family had grown to expect, all I had to offer them was a dense, flat, slightly burnt loaf. I immediately set to work analyzing where I went wrong. I checked the ingredients in the recipe and walked back through the steps I had taken. Based on the less than impressive outcome, I hypothesized and subsequently confirmed that I was missing an ingredient to the whole process, the yeast. Now that I had identified the errors in the baking process, I followed the same steps, ensuring that I used the right ingredients, resulting in the expected outcome, delicious fresh bread. However, examining the outcomes of a social program or treatment intervention may not be as apparent as tasting burnt bread.  

Although seemingly complicated on the surface, program evaluation is very similar to the scientific method you have been engaging in throughout your undergraduate career. The scientific method requires us to ask a question, form a hypothesis, test that hypothesis, come to a conclusion, and finally evaluate the conclusion. Social research methods follow the same steps but focus on answering questions that pertain to individuals and society. Program evaluation is when we implement social research methods to analyze the effectiveness of a program. (Rossi et al., 2019). Program evaluation allows us to apply research for a specific purpose and answer the question, “How do you know you are doing any good at all?” Using program evaluation, we can analyze several aspects by conducting focused evaluations like process and outcome evaluations. Program evaluations can identify ways to improve a program, support the program’s effectiveness or ineffectiveness, and provide more information about a program’s implications. 

Program evaluation can be broken into two parts process and outcome. Process evaluation is when we evaluate to determine if a program is being implemented as designed. However, outcome evaluation evaluates whether the program achieved its intended goals. If we revisit the bread-making example, seeing the burnt bread was the outcome evaluation, but reviewing the steps and ingredients was the process evaluation. It is essential to conduct both process and outcome evaluations when implementing social services and behavioral interventions because the information they provide is complementary and gives a complete picture of a program’s overall effectiveness and health. For example, a process evaluation can monitor implementation and help offer alerts to problems that might negatively impact the outcome. Outcome evaluations can provide the data to determine the overall program effectiveness. If the outcome evaluation proves that the program did not meet its intended goals, a process evaluation can help identify implementation issues. Alternatively, if the outcome evaluation reveals positive outcomes, the process evaluation provides information for program replication. Had I been proactively engaging in program evaluation, I could have adjusted to avoid the bad outcome before it was too late. However, in social science and behavioral interventions, the result can be bad outcomes for families.   

While the results of my burnt bread were disappointing, no one was harmed (except maybe my ego). However, the consequences could be devastating when we deliver social services or behavioral interventions that result in unintended outcomes. Children could stay in unsafe homes and continue to be abused, and parents might not get the help they need to reunify with their children; veterans could experience additional trauma while seeking care.  

Using program evaluation, we can analyze several aspects by conducting focused evaluations like process and outcome evaluation that all go hand in hand to provide a comprehensive picture of a program’s overall function and health. Program evaluations can identify ways to improve a program, provide support for the program’s effectiveness or ineffectiveness, and provide more information about a program’s implications.  

Another critical aspect of program evaluation is that it provides external validity and allows us to distribute our findings to funders and stakeholders with greater confidence (Moye, 2016). Social service programs need to demonstrate positive outcomes, and external validity is important because funders require it. Funding for programs is data-driven and will require evidence. As demand for social services increases, securing ongoing funding has grown more complicated. Grant funding to address the grand social challenges has become more competitive, and accountability measures are stricter. With financial demand often outweighing the supply, social workers must continue demonstrating their effectiveness. It is essential to articulate the need for social service programs and interventions to stakeholders, which is done through evidence. What theory and design is the program grounded? What are the program’s processes, impact, and outcomes? Program evaluations can produce practice-based evidence that leads to evidence-based practices. (Fynn et al., 2020).  

Program evaluation is not just limited to social services programs in the macro setting focusing on system-level interventions and mezzo settings that focus on the community-level interventions. Program evaluation can be used on the micro-level to determine if clinical interventions are serving their intended purpose. Program evaluation should matter to all academic scholars because it allows us to determine if the programs, we are providing are doing any good all. Because at the end of the day, nobody wants to eat burnt bread. 

Check out part 2 of this series here.