Well its half way through my 2nd full year of PhD and still finding my way. This year feels like it has mostly been writing and rewriting. Phase 1 findings have been 90% written up into a conference paper that was originally intended for the Australasian Evaluation Society Conference in Brisbane. It was accepted but then funding from the department was not forthcoming. So a last minute change of plans required a rewrite of the abstract and have recently received acceptance to The 5th Asian Conference on Education. It's in Osaka but I will hopefully present virtually (funding pending). My dilemma now is how to reduce what is about an 8 and a half thousand word paper into just 5000 for the conference. Also how do I ensure the paper is selected for the journal as a publication is really my intended outcome for participating in this conference.
My literature review has also been submitted to the Journal Studies in Educational Evaluation. This was peer reviewed and received considerable constructive feedback. However the editor did invite resubmission so that has been done and I'm now awaiting the second round of feedback (or rejection).
So it is time to return to data collection for phase 2. I'm transcribing interviews from phase 2 and need to soon turn to analysis and writing. But first need to do wrap up interviews for phase 2 case studies.
What have I learned from all this writing? That improvements can always be made; that people see and read different meanings than you sometimes intend; that writing short (5000 word) articles is difficult; that I am able to write.
Also, I need to do some more work on theory. I have briefly touched on the use of pragmatism or realism but it might be interesting to try and write either a short paper or definitely more for the chapter in the thesis.
And finally whilst considering how to cut down paper on phase 1, we have discussed the possibility of removing the Leximancer information and either trying to publish something on the methodology, using the results tables as examples or putting that information into the thesis.
Tuesday, August 13, 2013
A call for a study of evaluation practice
Improving Evaluation Theory through the
Empirical Study of Evaluation Practice
Nick L Smith, 1993
The author states that few studies have
been done on the practice of evaluation but that these are necessary in order
to develop evaluation theory. The relationship between practice and theory
often arises during metaevaluations as these tend to highlight the problem of
translating current evaluation theory into acceptable evaluation practice. The
author calls for increase in the number of research studies on evaluation
practice (as opposed to evaluation of evaluation practice). At the time of
writing this paper, Smith explains that ‘much of the current empirical
justification of evaluation theories is from self-reported cases of what works’
(p238). He also quotes from Shadish, Cook and Leviton (1991) in this regard.
Another 1991 paper by Marvin Alkin which studied the factors that influence
theorists to change their conceptions of their own models or approaches is:
increased experience in conducting evaluations, and accumulation of personal
research on evaluation practice.
Smith explains that studies of evaluation
practice are needed in order to know:
·
What works and how to make it
happen
·
How to avoid bad practice
·
How local contexts differ and
what works across different contexts
·
Where the problems of
inadequacies of evaluation practice could be ameliorated by improved evaluation
theory
If theories are presented in abstract,
conceptual terms rather than in concrete terms based on how they would be
applied in practice then we cannot know how practitioners actually articulate
or operationalize various models or theories or whether they do so at all. So
it becomes unclear what is meant when an evaluator claims to be using a
particular theoretical approach. And if theories ‘cannot be uniquely
operationalized then empirical tests of their utility become increasingly
difficult’ (p240). Furthermore, if alternative theories give rise to similar
practices, then theoretical differences may not be practically significant.
Smith discusses the use of contingency
theories, an approach considered to be the strongest type of evaluation
approach (Shadish, Cook and Leviton, 1991). These theories ‘seek to specify the
conditions under which different evaluation practices are effective’ (p240). He
then goes on to link this approach with the need for theoretical development
alongside studies of practice. He calls for a continuation of public reflection
by evaluation practitioners alongside more independent empirical studies of
evaluation practice by evaluation researchers.
This article is some 20 years old now - need to check more recent articles from same author to see what has been done. Also check for articles citing this one.
Subscribe to:
Posts (Atom)