A call for a study of evaluation practice

Improving Evaluation Theory through the Empirical Study of Evaluation Practice

Nick L Smith, 1993

The author states that few studies have been done on the practice of evaluation but that these are necessary in order to develop evaluation theory. The relationship between practice and theory often arises during metaevaluations as these tend to highlight the problem of translating current evaluation theory into acceptable evaluation practice. The author calls for increase in the number of research studies on evaluation practice (as opposed to evaluation of evaluation practice). At the time of writing this paper, Smith explains that ‘much of the current empirical justification of evaluation theories is from self-reported cases of what works’ (p238). He also quotes from Shadish, Cook and Leviton (1991) in this regard. Another 1991 paper by Marvin Alkin which studied the factors that influence theorists to change their conceptions of their own models or approaches is: increased experience in conducting evaluations, and accumulation of personal research on evaluation practice.

Smith explains that studies of evaluation practice are needed in order to know:

·      What works and how to make it happen

·      How to avoid bad practice

·      How local contexts differ and what works across different contexts

·      Where the problems of inadequacies of evaluation practice could be ameliorated by improved evaluation theory

If theories are presented in abstract, conceptual terms rather than in concrete terms based on how they would be applied in practice then we cannot know how practitioners actually articulate or operationalize various models or theories or whether they do so at all. So it becomes unclear what is meant when an evaluator claims to be using a particular theoretical approach. And if theories ‘cannot be uniquely operationalized then empirical tests of their utility become increasingly difficult’ (p240). Furthermore, if alternative theories give rise to similar practices, then theoretical differences may not be practically significant.

Smith discusses the use of contingency theories, an approach considered to be the strongest type of evaluation approach (Shadish, Cook and Leviton, 1991). These theories ‘seek to specify the conditions under which different evaluation practices are effective’ (p240). He then goes on to link this approach with the need for theoretical development alongside studies of practice. He calls for a continuation of public reflection by evaluation practitioners alongside more independent empirical studies of evaluation practice by evaluation researchers.

This article is some 20 years old now - need to check more recent articles from same author to see what has been done. Also check for articles citing this one.

Evaluation theory vs practice

What Guides Evaluation? A study of how evaluation practice maps on to evaluation theory.

Christina Christie 2003

This study came in response to repeated calls from theorists for more empirical knowledge of evaluation which would in turn help explain the nature of evaluation practice. A study with similar aims was carried out in 1987 by Shadish and Epstein however their study makes use of a survey instrument designed by the researchers. The Christie study realizes the fact that ‘theoretical orientation often cannot be accurately assessed through direct questioning because evaluation practitioners usually are not proficient in theory (Smith 1993), and so are unable to identify their particular theoretical orientation’ (P.9). A case study approach to observe behaviour of evaluation practitioners is the usual place to go however this offers depth but cannot cover breadth of understanding of how evaluators use theory to guide their work. This study is unique in that it uses eight distinguished evaluation theorists with a broad array of perspectives to construct the survey instrument. These theorists are Boruch, Chen, Cousins, Eisner, Fetterman, House, Patton and Stufflebeam.

The conceptual framework is built on the work of Alkin and House (1992) using their taxonomy of three dimensions: methods, values and use. Each dimension has a continuum that further defines it. For methods, its quantitative to qualitative; for values its unitary to pluralist (criteria used when making evaluative judgments); and for use its from enlightenment (academic) to instrumental (service-oriented). Each theorist was asked to submit one statement ‘demonstrating the practical application associated with his theoretical standpoint related to each of the three dimensions’. P11. They were also invited to submit up to six additional statements and in total, the final instrument contained 38 items related to evaluation practice and was piloted with 5 practicing evaluators.

The participants in this study were from 2 groups. The theorists were asked to complete the survey instrument and then 138 evaluators working on a statewide Californian educational program called Healthy Start. Many of these were not evaluators but school and program administrators and so represent a cross section of how evaluations are being conducted today. This group was also subdivided for reporting of results into internal and external evaluators. The collection of demographic data produced some interesting findings. The majority of evaluators were female, white and over 40. In terms of education, 75% of the external evaluators were PhD qualified but this aligns with the years of experience and self-rating of their evaluation knowledge and skills.

And a very interesting finding - only  a small proportion of evaluators in this study sample were using an explicit theoretical framework to guide their practice. More on this later.

The analytic procedure used multidimensional scaling (MDS) whereby observed similarities (or dissimilarities) are represented spatially as a geometric configuration of points on a map. More specifically, this study used classical multidimensional unfolding (CMDU) which is an individual-differences analysis that portrays differences in preference, perception, thinking or behaviour and can be use when studying differences in subjects in relationship to one another or to stimuli (p14). Furthermore, to interpret the CMDU results in this study, ALSCAL (Alternating Least-Square Scaling Algorithm) was used to produce two dimensions: scope of stakeholder involvement and method proclivity.

The first dimension ranged from users being simply an audience for the evaluation findings, to users being involved in the evaluation at all stages from start to finish. The second dimension, method proclivity is the extent to which the evaluation is guided by a prescribed technical approach. One end of this dimension would be characterized as partiality to a particular methodology that has as a feature predetermined research steps. Boruch anchored this end of the dimension as the experimental research design is at the centre of his approach to evaluation. The other end of this dimension represents partiality to framing evaluations by something other than a particular methodology with predetermined research steps. Patton for example falls to this end by use of his utilization-focused evaluation which is flexible in its nature calling for creative adaption during its problem solving approach.

So in this way, plotting the theorists’ responses on the two-dimensional axes helped to name and clarify the dimensions. The next step was to map the evaluators practice onto the dimensions in order to compare how practitioners rated against theorists. Evaluators were divided into two groups, internal and external due to noted professional characteristics. Results indicated that generally, stakeholders have a more limited role in evaluations conducted by internal evaluators than those conducted by external evaluators. In addition, internal evaluators are more partial to methodologies with predetermined research steps than are external evaluators. This analysis depicts only a broad depiction of their practice. By investigating placement in each quadrant of the map, a more comprehensive understanding is produced.

In general, external evaluators practice was more like the theorists. More specifically, they were most closely aligned with the theorist Cousins. Furthermore although they are concerned with stakeholder involvement, they are partial to their methods and conduct evaluations accordingly. Internal evaluators were distributed evenly between the theorists which reflect the diversity in their practices and implies that we cannot generalize about their categorization into any one genre of theoretical approaches.

Christie goes on to discuss the theorists results and concludes that through this study it has become evident that even though theories may share big picture goals, they don’t necessarily share the same theoretical tenets or similar practical frameworks. In addition, ‘the prescribed practices of a theory are not necessarily best reflected in its name or description’ (p.29). Her other major point was that ‘despite some theoretical concerns related to stakeholder involvement, all of the theorists in this study do involve stakeholders at least minimally’ (p.30). However many theorists have not chosen to incorporate changing notions of such involvement because of ‘a common perception that… it is understood to be a part of the evaluation process, no matter one’s theoretical approach’ (p.31).

In relation to practicing evaluators, they are often intimately involved in the program and therefore assume they understand how other stakeholders think and feel about the program and hence don’t tend to involve them a great deal. Politics may play out more heavily with internal evaluators and may influence their decision not only on stakeholder involvement but also to their emphasis on prescribed methods. In terms of evaluator bias, the study found that internal evaluators may be aware of the importance of objectivity and tend towards more quantitative methods to increase the credibility of their findings. External evaluators on the other hand may be influenced by their colleagues’ perception of their methods used, with the thought that criticism could jeopardize potential for future work. Therefore both types of evaluators employ method-driven frameworks influenced by the perception that the results are more defensible. And finally, this study shows that theory is not requisite to evaluation practice, in fact evaluators adopt only select portions of a given theory. Even ‘those who did claim to use a particular theory did not correspond empirically with the practices of the identified theorist’ (p33). Therefore Christie concludes that ‘the gap between the common evaluator and the notions of evaluation practice put forth by academic theorists has yet to be bridged.’ (p34).

Case Study

Studying cases allows for obtaining an in-depth understanding (through explaining, exploring, and describing) of complex social phenomena, while retaining the holistic and meaningful characteristics of real-life events (Yin 1994).

KOHLBACHER, F.. The Use of Qualitative Content Analysis in Case Study Research. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, North America, 7, jan. 2006. Available at: <http://www.qualitative-research.net/index.php/fqs/article/view/75/153>. Date accessed: 01 Oct. 2012.
Abstract: This paper aims at exploring and discussing the possibilities of applying qualitative content analysis as a (text) interpretation method in case study research. First, case study research as a research strategy within qualitative social research is briefly presented. Then, a basic introduction to (qualitative) content analysis as an interpretation method for qualitative interviews and other data material is given. Finally the use of qualitative content analysis for developing case studies is examined and evaluated. The author argues in favor of both case study research as a research strategy and qualitative content analysis as a method of examination of data material and seeks to encourage the integration of qualitative content analysis into the data analysis in case study research.

Thematic Analysis

Braun and Clarke (2006), offer a complete and in-depth paper on what it is, guidelines to using it and pitfalls to avoid. They state that it can be considered a method in its own right - contrary to other authors stating it is not. It is compatible with constructionist paradigms and they stress its flexible nature in its use which can sometimes cause it to be framed by a realist/experimental method (though they don't particularly go along with this).
Importance is placed on explaining 'how' analysis is conducted (as its often omitted) as well as describing what and why they are dong it that way. Terminology is defined - Data corpus, set, item and extract. [all interviews, answers or themes, one interview and quotes].
" Thematic analysis is a method for identifying, analysing and reporting patterns (themes) within data." p.79
we shouldn't say that themes 'emerge' because actually that implies that they reside in the data - in actual fact they reside in the heads of the researcher who plays an active role in identifying, selecting and reporting them.(Ely et al., 1997 and Taylor & Ussher, 2001)
How do we say a theme exists? There are no hard and fast rules - though prevalence is important. A theme could be prevalent in one data item or across the whole corpus. And it may (or may not) be present in every data set or it may be present to only a small extent.
You should think about how a theme captures something of importance to the overall research question(s). This may make it 'Key'. The the question lies in how to report it ie 'most participants' or 'some..' or 'a number of...' etc.
You need to ask yourself whether you want to provide a rich thematic description of the whole data set or do you want to provide a detailed account of just one aspect of the data.
Next, will you provide an inductive analysis - whereby you link the themes to the data ie from specific questions, or a theoretical analysis - whereby your research questions evolve from the themes.
Semantic vs latent themes ie surface level descriptive or more deeper analysis of the underlying causes, assumptions and conceptualizations - this leads towards a more constructivist approach
Back to the paradigm wars - if a constructivist paradigm is used, then the themes are built from sociocultural contexts which enable individual accounts. In comparison the realism framework allows a more simple explanation to develop since meaning, experience and language are unidirectional. ie the language is used to describe experience and provide meaning.

The paper then goes on to describe 6 steps in thematic analysis - to be used flexibly and as a guide.
1. Familiarize yourself with your data: jot down notes for coding schemas that you will come back to in subsequent phases.
2. Generate initial codes. Work systematically through the data set  and identify interesting aspects in the data items that may form the basis of repeated patterns.
3. Search for themes: sort the potential codes into themes - broader analysis of whole data set. Use concept mapping
4. Review themes. this is a two step process - level 1 consists of looking at the extracts for each theme and deciding if they really fit that theme, and are coherent. if not, reassess the theme and perhaps discard extracts if necessary. Level 2 of this stage will depend on your theoretical approach and requires revisiting the whole data set and consider the validity of each of the themes.
5. Define and name the themes. Look at the data extracts for each theme and organise into a coherent account with an accompanying narrative. Look for sub-themes within a theme and finally, use clear (short) names for the themes so that the reader understands quickly what it means.
6. Produce the report. More than just describe the data, tell the story by using the extracts to make an argument in relation to the research questions.

Some questions to ask towards the end fo the analysis:
what does this theme mean?
What are the assumptions underpinning it?
what are the implications of this theme?
what conditions are likely to have given rise to it?
why do people talk about this thing in this particular way?
what is the overall story the different themes reveal about the topic?

Potential pitfalls are described:
1. no real analysis is done and the analysis is just a sting of extracts with little analytic narrative.
2. uses the data collection questions as the themes.
3. no coherence around an idea or concept in all aspects of the theme.
4. no consideration of alternative explanations or variations of the data
5. mismatch between the interpretation of the data and the theoretical framework.

evaluation or research

Leading on from last weeks ramblings on the differences between research and evaluation, Owen (2006 p.64) says that the inclusion of the planning and communicating stages is what differentiates evaluation from social research. The 'middle section' is the research which uses similar ranges of data collection and analysis techniques.

I want to concentrate on this difference. Try and prove that concentrating on the first and 3rd sections of the pie (picture) will help improve evaluation practices. ie the interactive instrument helps with these sections. and help with including time to do them.

Need more readings around time taken for each of these sections. Need to continue reading Owen.

Update: Nov 2011

Alkin & Taaut, 2003 write that the goal of research is generizable knowledge but the purpose of evaluation is context-specific (p.3). They Quote Cronbach & Suppes (1969) and conclusion-oriented research vs. decision-oriented evaluation.

update March 2012

Alkin (2011 - Evaluation Essentials form A to Z) states on p.8 that research seeks conclusions and evaluation leads to decisions. researchers ask their own questions in order to seek conclusions they can use to add to the knowledge bank. Evaluation answers questions that are important to a particular person - the stakeholder or client, say.

Alkin talks briefly about the definition of evaluation (which are goal orientated, around merit and worth) but he directs the readers focus to the processes which allow one to reach the point of being ready to judge merit and worth.

Update August 2012

reading Mackenzie, N. M. & Ling, L. M. (2009). The research journey: A Lonely Planet approach. Issues In Educational Research, 19(1), 48-60. http://www.iier.org.au/iier19/mackenzie.html

quote Mertens (Mertens, D. M. (2005). Research methods in education and psychology: Integrating diversity with quantitative and qualitative approaches (2nd ed.). Thousand Oaks, CA: SAGE.) though i can't give a page number of this as its in html (open access)


"This highlights the decisions we take as researchers when we aim for convergent outcomes or divergent outcomes. If our research is so prescribed and directed as to push us towards particular desired outcomes, it is convergent and in fact, may potentially not be research at all. Mertens (2005) makes a distinction between research and other forms of activity such as evaluation.

The relationship between research and evaluation is not simplistic. Much of evaluation can look remarkably like research and vice versa. Both make use of systemic inquiry methods to collect, analyse, interpret and use data to understand, describe, predict, control or empower. Evaluation is more typically associated with the need for information for decision making in a specific setting, and research is more typically associated with generating new knowledge that can be transferred to other settings. (p.2)

In fact, much of the prescribed and funded so-called research we undertake for convergent outcomes which fit the agenda of funding bodies is probably more akin to evaluation than research. Research which does not work towards pre-determined or prescribed outcomes and thus can produce divergent outcomes more in the spirit of what we understand as true research."

More mq projects

I revisited the MQ Grants 'previous winners' webpage to find it updated with many more final reports. There are now 29 in total from across the competitive and strategic grants categories.

I've been thinking about how to best make use of these reports. I could compare the final report with the initial application (were available, and when not, I could approach the awardee and ask them for a copy). This may highlight changes, things they said they would do but haven't.

I've been grappling with the 'criteria' - should i use the excellent questions provided by Chesterton and Cummings and supplement with Datta etc? Stufflebeam has an excellent Metaevaluation checklist but it is too detailed with 300 items to be checked. If we could use this it would provide excellent quantitative data. But I thik we cannot.

The main issue for me at the moment is my first question is 'what evaluation forms and approaches were used in this project?' And I have a feeling my answer for majority is 'none'. Maybe its about terminology. Some things that are covered by the word evaluation are things like data collection - isn't this research though? Now I'm struggling with the difference between research and evaluation.....
some more searches required me thinks.

An article by Julia Coffman, Consultant, HFRP based on Scriven's work:

http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/reflecting-on-the-past-and-future-of-evaluation/michael-scriven-on-the-differences-between-evaluation-and-social-science-research

How are evaluation and social science research different?Evaluation determines the merit, worth, or value of things. The evaluation process identifies relevant values or standards that apply to what is being evaluated, performs empirical investigation using techniques from the social sciences, and then integrates conclusions with the standards into an overall evaluation or set of evaluations (Scriven, 1991).
Social science research, by contrast, does not aim for or achieve evaluative conclusions. It is restricted to empirical (rather than evaluative) research, and bases its conclusions only on factual results—that is, observed, measured, or calculated data. Social science research does not establish standards or values and then integrate them with factual results to reach evaluative conclusions. In fact, the dominant social science doctrine for many decades prided itself on being value free. So for the moment, social science research excludes evaluation.¹However, in deference to social science research, it must be stressed again that without using social science methods, little evaluation can be done. One cannot say, however, that evaluation is the application of social science methods to solve social problems. It is much more than that.