An evaluation framework for sustaining the impact of educational development

Hashimoto, Kazuaki, Hitendra Pillay, and Peter Hudson. “An Evaluation Framework for Sustaining the Impact of Educational Development.” Studies In Educational Evaluation 36, no. 3 (2010) 101–110.

The context of this paper is international aid agencies funding of educational development projects in recipient countries and their apparent ineffectiveness. The authors were interested in overcoming donor agencies internal compliance requirements by looking how local evaluation capacity could be developed and also how developments could continue to be sustained after project completion. Although this context is not applicable to the HE sphere, the same could be said of external funding agents vs internal projects.

The authors define process evaluation (quote: DAC Network on Development Evaluation. (2008). Evaluating development cooperation. OECD DAC Network on Development Evaluation. Retrieved January 6, 2010, from http://www.oecd.org/dataoecd/3/56/41069878.pdf.) And state the importance of process evaluation being the involvement of the participants in making decisions on a project such as terminating a project if necessary (p.102.). The authors quote Miyoshi and Stemming (2008) in that most studies on evaluation with participatory approaches are not underpinned by evaluation theories but are method-oriented.

So an Egyptian project was used as a case study (see previous post) and there were two research questions: (1) how can an entire educational development project be evaluated? and (2) how can the capacity development in educational reform be evaluated? Participants included six different groups of stakeholders: funding body, local admin, researchers, teachers, parents and students. The analytic technique used was pattern matching (Yin, 2003, p. 116) to enhance its internal validity. There were three emergent themes to the study, context, outcome and process evaluation.

Outcome evaluation:

• assessing outcomes is necessary for determining the success of an educational reform project.
• Outcome evaluation should include local participant involve- ment for evaluating a project since they are the end users.
• Local stakeholders should not be seen as informants or discussants but rather as evaluators working jointly with aid agencies so they can appreciate the success and failure of achieving the objectives.
• Results supported the use of an external evaluator who in collaboration with the internal evaluators of the project can undertake a macro level evaluation of the project

Context Evaluation

• context evaluation assesses local needs and problems to be addressed, cultural, political and financial issues, assists to design a project and sets objectives before the initiation of an educational project. 
• more local stakeholders such as representatives of local community are needed to join the evaluation to make their voice heard because after all they are the beneficiaries. 
• This engagement of various stakeholders in dialogues throughout the project from the project design phase may enable their opinions and interests to be considered for designing and implementing a more effective project (House & Howe, 2000).

Process evaluation

• There was a need for adopting a systematic participatory evaluation approach involving individuals and groups at the different levels of an educational system, which was central to process evaluation. 
• the linchpin of a sound process evaluation is employing skilled people
• the practice and culture of process evaluation should be nurtured during the life of educational projects and be institutionalized locally. This has the potential to sustain the impact of projects. 

Conclusion - conventional monitoring and evaluation practices do not have the ability to sustain a project beyond its lifetime. And that 'paradigms should shift from outcome-focused evaluation currently dominated by international donor agencies to process evaluation conducted largely by local participants but also supported by donor agencies. ' (p109)

Framework outlined in picture below (from p.108):

Other articles that follows this line of thinking:
 Donnelly, John. Maximising participation in international community-level project evaluation: a strength-based approach. [online]. Evaluation Journal of Australasia, v.10, no.2, 2010: 43-50. Availability:<http://search.informit.com.au/fullText;dn=201104982;res=APAFT> ISSN: 1035-719X. [cited 01 Oct 12].

Challenging times for evaluation of international development assistance

M Nagao - Evaluation Journal of Australasia, 2006 - aes.asn.au