What
Guides Evaluation? A study of how evaluation practice maps on to evaluation
theory.
Christina
Christie 2003
This study came in response to repeated
calls from theorists for more empirical knowledge of evaluation which would in
turn help explain the nature of evaluation practice. A study with similar aims
was carried out in 1987 by Shadish and Epstein however their study makes use of
a survey instrument designed by the researchers. The Christie study realizes
the fact that ‘theoretical orientation often cannot be accurately assessed
through direct questioning because evaluation practitioners usually are not
proficient in theory (Smith 1993), and so are unable to identify their
particular theoretical orientation’ (P.9). A case study approach to observe
behaviour of evaluation practitioners is the usual place to go however this
offers depth but cannot cover breadth of understanding of how evaluators use
theory to guide their work. This study is unique in that it uses eight
distinguished evaluation theorists with a broad array of perspectives to
construct the survey instrument. These theorists are Boruch, Chen, Cousins,
Eisner, Fetterman, House, Patton and Stufflebeam.
The conceptual framework is built on the work
of Alkin and House (1992) using their taxonomy of three dimensions: methods,
values and use. Each dimension has a continuum that further defines it. For
methods, its quantitative to qualitative; for values its unitary to pluralist
(criteria used when making evaluative judgments); and for use its from
enlightenment (academic) to instrumental (service-oriented). Each theorist was
asked to submit one statement ‘demonstrating the practical application
associated with his theoretical standpoint related to each of the three
dimensions’. P11. They were also invited to submit up to six additional
statements and in total, the final instrument contained 38 items related to
evaluation practice and was piloted with 5 practicing evaluators.
The participants in this study were from 2
groups. The theorists were asked to complete the survey instrument and then 138
evaluators working on a statewide Californian educational program called
Healthy Start. Many of these were not evaluators but school and program
administrators and so represent a cross section of how evaluations are being
conducted today. This group was also subdivided for reporting of results into
internal and external evaluators. The collection of demographic data produced
some interesting findings. The majority of evaluators were female, white and
over 40. In terms of education, 75% of the external evaluators were PhD
qualified but this aligns with the years of experience and self-rating of their
evaluation knowledge and skills.
And a very interesting finding - only a small proportion of evaluators in this
study sample were using an explicit theoretical framework to guide their
practice. More on this later.
The analytic procedure used
multidimensional scaling (MDS) whereby observed similarities (or dissimilarities)
are represented spatially as a geometric configuration of points on a map. More
specifically, this study used classical multidimensional unfolding (CMDU) which
is an individual-differences analysis that portrays differences in preference,
perception, thinking or behaviour and can be use when studying differences in
subjects in relationship to one another or to stimuli (p14). Furthermore, to
interpret the CMDU results in this study, ALSCAL (Alternating Least-Square Scaling
Algorithm) was used to produce two dimensions: scope of stakeholder involvement
and method proclivity.
The first dimension ranged from users being
simply an audience for the evaluation findings, to users being involved in the
evaluation at all stages from start to finish. The second dimension, method
proclivity is the extent to which the evaluation is guided by a prescribed
technical approach. One end of this dimension would be characterized as
partiality to a particular methodology that has as a feature predetermined
research steps. Boruch anchored this end of the dimension as the experimental
research design is at the centre of his approach to evaluation. The other end
of this dimension represents partiality to framing evaluations by something
other than a particular methodology with predetermined research steps. Patton
for example falls to this end by use of his utilization-focused evaluation
which is flexible in its nature calling for creative adaption during its
problem solving approach.
So in this way, plotting the theorists’
responses on the two-dimensional axes helped to name and clarify the
dimensions. The next step was to map the evaluators practice onto the
dimensions in order to compare how practitioners rated against theorists.
Evaluators were divided into two groups, internal and external due to noted
professional characteristics. Results indicated that generally, stakeholders
have a more limited role in evaluations conducted by internal evaluators than
those conducted by external evaluators. In addition, internal evaluators are
more partial to methodologies with predetermined research steps than are
external evaluators. This analysis depicts only a broad depiction of their
practice. By investigating placement in each quadrant of the map, a more
comprehensive understanding is produced.
In general, external evaluators practice
was more like the theorists. More specifically, they were most closely aligned
with the theorist Cousins. Furthermore although they are concerned with
stakeholder involvement, they are partial to their methods and conduct
evaluations accordingly. Internal evaluators were distributed evenly between
the theorists which reflect the diversity in their practices and implies that
we cannot generalize about their categorization into any one genre of theoretical
approaches.
Christie goes on to discuss the theorists
results and concludes that through this study it has become evident that even though
theories may share big picture goals, they don’t necessarily share the same
theoretical tenets or similar practical frameworks. In addition, ‘the
prescribed practices of a theory are not necessarily best reflected in its name
or description’ (p.29). Her other major point was that ‘despite some
theoretical concerns related to stakeholder involvement, all of the theorists
in this study do involve stakeholders at least minimally’ (p.30). However many
theorists have not chosen to incorporate changing notions of such involvement
because of ‘a common perception that… it is understood to be a part of the
evaluation process, no matter one’s theoretical approach’ (p.31).
In relation to practicing evaluators, they
are often intimately involved in the program and therefore assume they
understand how other stakeholders think and feel about the program and hence
don’t tend to involve them a great deal. Politics may play out more heavily
with internal evaluators and may influence their decision not only on
stakeholder involvement but also to their emphasis on prescribed methods. In
terms of evaluator bias, the study found that internal evaluators may be aware
of the importance of objectivity and tend towards more quantitative methods to
increase the credibility of their findings. External evaluators on the other
hand may be influenced by their colleagues’ perception of their methods used,
with the thought that criticism could jeopardize potential for future work.
Therefore both types of evaluators employ method-driven frameworks influenced
by the perception that the results are more defensible. And finally, this study
shows that theory is not requisite to evaluation practice, in fact evaluators
adopt only select portions of a given theory. Even ‘those who did claim to use
a particular theory did not correspond empirically with the practices of the
identified theorist’ (p33). Therefore Christie concludes that ‘the gap between
the common evaluator and the notions of evaluation practice put forth by
academic theorists has yet to be bridged.’ (p34).
