Theory-based evaluation

Nesman, Batsche & Hernandez (2007)

Theory-based evaluation of a comprehensive Latino education initiative: An interactive evaluation approach

This paper describes a 5 year initiative to develop, implement and evaluate program(s) that would increase Latino student access to Higher Education. Theory of change and logic models were used to guide the program as these have been previously shown to be most effective when trying to create social change within  comprehensive community initiatives. 'Logic models can also serve as a roadmap for implementers to move from ideas to action by putting components together into a visual framework (Hernandez & Hodges, 2003).' (p.268)
A conceptual model was developed which incorporated context, guiding principles, implementation strategies, outcomes and  evaluation and resulted in a vision statement for the program. The paper also describes the interventions which were to be implemented, and goes on to describe the evaluation approach in more detail. They us an embedded case-study design (Yin, 1984) and mixed methods with a developmental approach which allowed for adaptation over time as the project moved through the varying stages of completion. Key questions were developed associated with each goal from the funding agency ie Process, Impact and Sustainability. One of the key findings under process was that the initial plan 'had been overly ambitious and that it would not be possible to accomplish this large number of interventions with the available resources.' (p.272). This resulting in a paring back of outcomes with some initiatives being prioritised and some being dropped altogether. A finding under Impact was that 'although it would be several years before long term outcomes could be effectively measured, the evaluators developed a tracking system to monitor changes in student outcomes each year.' (p.274). With sustainability, it was felt that strategies within institutions were more likely to be sustained than those relying on collaboration and cross-institutional coordination, unless there was ongoing external support. (p.279)
The authors also wrote about lessons learned from this approach. If theory-based evaluation is to be maximised, it does require  training of program participants on logic model development and theory of change approaches early in the process of implementation. This training can lead to the development of interactive and productive relationships between evaluators and implementers. Adopting a developmental approach was also highly beneficial in this project.