Program implementation is a tricky business, and few experienced program managers expect a smooth transition from the planning stage to real world application. The challenging nature of implementation provides evaluators with an opportunity: some of the most useful evaluation focuses on these challenges. We can use implementation evaluation to learn about the contextual factors and on-the-ground practices that explain the differences between program success and failure, between good and great.
Since 2013, LFA has been evaluating the implementation of the Hawaiʻi Community Foundation’s middle school program, Connecting for Success (CFS). The CFS program identifies, tracks, and provides services to cohorts of struggling middle school students to increase their connection to school and to improve their attendance and academic performance. Participating middle schools are implementing innovative strategies to increase school connectedness and improve dropout prevention throughout the Hawaiʻi school system.
In designing an evaluation of CFS implementation, we looked to ideas from the school reform literature (see Bryk et al 2010, Finnigan et al 2009, and Newman et al 2001) to build out the concept of strategic coherence, a concept that captures a particularly important aspect of implementation:
School reforms are strategically coherent when they are logically consistent with other initiatives or programs, and with existing practices. When reforms cohere, the various policies and practices build on, complement, and reinforce one another; without coherence, the various policies and practices contradict one another and contribute to the fractured time and attention of school staff. Separate efforts may be based on conflicting strategies, and contradictory demands may be made of staff.
Strategic coherence has three dimensions:
Quality of Integration Planning. When an organization focuses on integration planning, leadership seeks to ensure logical consistency among multiple initiatives and programs being implemented at one organization. Strategies and practices all orient toward one unified set of goals.
Role clarity. When there is role clarity, the plans for how initiatives or programs align create a strong understanding among staff of how different roles fit together, and how they work toward a unified set of goals.
Authentic integration. Authentic integration goes beyond written declarations of alignment or integration, which can sometimes be only “ceremonial.” When initiatives or programs authentically integrate, the daily activities and routines that staff engage in all drive toward a unified set of goals.
This concept of strategic coherence is especially important in the school context, because schools are frequently awash in the latest reform ideas, and are often implementing several reforms at once. As we listened to teachers and staff at the CFS schools (and at other schools involved in other evaluations), we often heard the term “add-on” to describe various initiatives: “this is just one more thing piled on our plates,” they would say, sometimes with world-weariness or exasperation, “and it doesn’t really fit in with how we’re already doing things.”
Under conditions of “strategic incoherence,” teachers and staff must contend with conflicting messages and contradictory demands. Their time and attention are allocated to separate efforts that push in different directions, and different teams may work at cross purposes. But when a set of initiatives, programs, or practices cohere, planners actively seek to avoid situations in which programs have conflicting goals, strategies, or activities. They set up teams for the people who need to work together. They develop work plans so that an activity for one program can be leveraged for an aligned program. Staff do not feel that their attention is divided among inconsistent tasks. All stakeholders are – metaphorically – pushing in the same direction.
Coherence makes a difference. Our CFS evaluation found that strategic coherence was one of the most meaningful factors relating to the program’s success: all of the schools with an excellent rating had improved performance (three of four); in contrast, six of the seven schools without an excellent rating showed no improved performance.
While strategic coherence has an obvious importance for schools contending with multiple reform efforts, it applies to other organizations as well. Sometimes nonprofits overhaul their work and swap out old programs for new ones, but it is much more common that they layer new programs and practices in with existing ones. In our work, we have seen this evolution in fields as disparate as homelessness services, workforce development, and journalism.
For others in the field who might want to explore the concept of strategic coherence as a critical aspect of high-quality program implementation, we offer a slightly modified version of the tool we developed to measure strategic coherence in CFS. (We modified it to remove its specificity to CFS.) The tool includes items used to measure the three dimensions of strategic coherence, as outlined in the rubric below:
What do you think?
We hope this tool sparks new ideas for evaluating program implementation. How does this approach resonate with your experience implementing or evaluating programs? Are there ways you might modify or improve upon it to provide additional insights to evaluators and practitioners alike?