A call for a study of evaluation practice

Improving Evaluation Theory through the Empirical Study of Evaluation Practice

Nick L Smith, 1993

The author states that few studies have been done on the practice of evaluation but that these are necessary in order to develop evaluation theory. The relationship between practice and theory often arises during metaevaluations as these tend to highlight the problem of translating current evaluation theory into acceptable evaluation practice. The author calls for increase in the number of research studies on evaluation practice (as opposed to evaluation of evaluation practice). At the time of writing this paper, Smith explains that ‘much of the current empirical justification of evaluation theories is from self-reported cases of what works’ (p238). He also quotes from Shadish, Cook and Leviton (1991) in this regard. Another 1991 paper by Marvin Alkin which studied the factors that influence theorists to change their conceptions of their own models or approaches is: increased experience in conducting evaluations, and accumulation of personal research on evaluation practice.

Smith explains that studies of evaluation practice are needed in order to know:

·      What works and how to make it happen

·      How to avoid bad practice

·      How local contexts differ and what works across different contexts

·      Where the problems of inadequacies of evaluation practice could be ameliorated by improved evaluation theory

If theories are presented in abstract, conceptual terms rather than in concrete terms based on how they would be applied in practice then we cannot know how practitioners actually articulate or operationalize various models or theories or whether they do so at all. So it becomes unclear what is meant when an evaluator claims to be using a particular theoretical approach. And if theories ‘cannot be uniquely operationalized then empirical tests of their utility become increasingly difficult’ (p240). Furthermore, if alternative theories give rise to similar practices, then theoretical differences may not be practically significant.

Smith discusses the use of contingency theories, an approach considered to be the strongest type of evaluation approach (Shadish, Cook and Leviton, 1991). These theories ‘seek to specify the conditions under which different evaluation practices are effective’ (p240). He then goes on to link this approach with the need for theoretical development alongside studies of practice. He calls for a continuation of public reflection by evaluation practitioners alongside more independent empirical studies of evaluation practice by evaluation researchers.

This article is some 20 years old now - need to check more recent articles from same author to see what has been done. Also check for articles citing this one.