case study method - draft

What data will be collected and how?

Invitations to participate in this study were sent to the awardees of all Learning and Teaching grants in 2012. There were three respondents who agreed to become a case study. The projects were from the School of Education; the Faculties of Human Sciences and Arts; and the Office for First Year Experience (FYE). Each project had a project leader and a project manager. Both were briefed on the case study protocol (see Appendix A). This included a minimum of three interviews with the project manager over the duration of the project (beginning, middle and end) and an interview with the project leader. These interviews were recorded for transcription, then transcripts were sent to the participants for review. Access to documentation including grant applications, minutes of any project meetings and final reports was also part of the case study protocol. There was also a request to attend any project meetings as a participant observer. This was only possible in one project.

How will the data be analysed?

Data generated in case study research can become unwieldy...(because?). The amount collected for this study was kept to a minimum by following just three projects. Each project was analysed separately and results compared. Content analysis of the interview and project documentation data was carried out using the manual extraction of themes (Krippendorf, date).

limitations to the methodology:
generalising results from just three cases can be problematic. However results could then be used in a pilot or wider study.

Introduction - case study

30 minute writing exercise:

In the higher education sector, learning and teaching research projects constitute one avenue of funding. These projects are either internally or externally funded by various private and governmental bodies (reference). With the move towards greater accountability of public funding and the drive to increase quality education, the necessity to incorporate evaluation measures is growing. Not only to include evaluation but to build evaluation into the project life-cycle both systematically and rigorously. There is a wide body of literature on the evaluation of learning and teaching and the utilization of evaluation results to improve the student experience (references here).
A recent review of the learning and teaching project evaluation literature indicated that there is little evidence of the scholarship....
But there is little to no empirical research on the evaluation of learning and teaching projects in the higher education sector and how the benefits can be realised in practice. One recent research study conducted by the authors has begun to invesigate the factors that inhibit the use of evalaution practices in the sector nad found that project leaders perceptions are an influencing factor. However that paper called for further study into this area, more specifically to observe a learning and teaching project to examine how the perceptions play out over time and investigate what can be done in practice to support the effective use of evaluation for wider scale benefit to the organisation / institution and ultimately the student experience.
This study aims to investigate the evaluative measures used during three internally funded learning and teaching projects, and explore how the project leaders' perceptions of evaluation affected their praxis. Furthermore, this study will consider what can be done to overcome barriers to successful evaluation implementation and therefore enhance this praxis.

writing sandwich

 The following was part of writers retreat:


area/discipline: education
topic: learning and teaching projects
sub topic: evaluation of such
2nd sub topic: higher education sector

working title:
a case study of the evaluation of learning and teaching projects in higher education: practices and perceptions

Problem:

In the higher education sector, grants can be obtained from internal and external sources to fund research into learning and teaching. The internal grants tend to be of around 12-18 month duration with specific guidelines as to how the projects should be conducted and results disseminated. Evaluation of these research projects should enable the research to become more rigorous and to supply evidence that the research is meeting its aims. However there some evidence from the literature that this evaluation is not taking place. One such study has investigated a number of completed internally funded learning and teaching projects and shown that evaluation is influenced by the perceptions of the project leaders.

Aim:
The aim of this study is to follow three project leaders as they conduct their research projects and invistigate how the evaluation unfolds. To identify factors that are at play which influence the evaluation. Observe what and how they evaluate and...


WOW factor of my project:
If money is to be spent wisely on research in learning and teaching, results and findings  of such research projects need to make an impact. We need to reflect and learn from such findings and incorporate them into the development cycle. This will ultimately result in better student learning. The findings from this study can be used to educate researchers about the value of evaluation and get them to see it as a learning mechanism rather than a punitive one. We also need to find a way to incorporate evaluation findings into the next round of projects such that they too can make an impact. So in a way to generalise findings so that a wider audience can benefit from the project.

Brown's 8 questions

Planning the next paper (4):

1. Who are the intended readers: grant bodies/grant application reviewers/ grant applicants
2. What did you do: Three recipients of locally funded learning and teaching project grants were 'followed' as they conducted their research projects over 12-18 months. There were three interviews with the project managers (beginning middle and end of project). And one interview with the project leaders (the person who applied for the grant and had the initial idea). Other project documentation was also used as evidence / data and project meetings were attended were relevant. The questions which guided the data collection were developed from the literature.
3. Why did you do it: To collect evidence as it was being generated rather than retrospectively when people have to 'recall' information which may be influenced by their perception of evaluation and other unknown factors. To see how perceptions changed over the course of the project and what influencing factors caused these changes.
4. what happened: all three had different experiences and perceptions of evaluation to begin with. This appeared to dictate how they conducted the evaluations and what they evaluated and what they did with any evaluation data.
5. What do the results mean in theory: don't know at this stage....
6. What do the results mean in practice: production of guidelines for project evaluation for/from the grant funding body. Development of a framework to guide project evaluation could arise from the findings.
7. What is the key benefit for readers: the grant bodies (ie application reviewers) could use the findings to guide their application and review processes. Academics and project leaders could use this framework to support them in their learning through the project and enable the institution to benefit more widely from the projects. 
8. What remains unsolved: How this framework could best be implemented, feedback from use of the framework.

progress check

Well its half way through my 2nd full year of PhD and still finding my way. This year feels like it has mostly been writing and rewriting. Phase 1 findings have been 90% written up into a conference paper that was originally intended for the Australasian Evaluation Society Conference in Brisbane. It was accepted but then funding from the department was not forthcoming. So a last minute change of plans required a rewrite of the abstract and have recently received acceptance to The 5th Asian Conference on Education. It's in Osaka but I will hopefully present virtually (funding pending). My dilemma now is how to reduce what is about an 8 and a half thousand word paper into just 5000 for the conference. Also how do I ensure the paper is selected for the journal as a publication is really my intended outcome for participating in this conference.

My literature review has also been submitted to the Journal Studies in Educational Evaluation. This was peer reviewed and received considerable constructive feedback. However the editor did invite resubmission so that has been done and I'm now awaiting the second round of feedback (or rejection).

So it is time to return to data collection for phase 2. I'm transcribing interviews from phase 2 and need to soon turn to analysis and writing. But first need to do wrap up interviews for phase 2 case studies.

What have I learned from all this writing? That improvements can always be made; that people see and read different meanings than you sometimes intend; that writing short (5000 word) articles is difficult; that I am able to write.

Also, I need to do some more work on theory. I have briefly touched on the use of pragmatism or realism but it might be interesting to try and write either a short paper or definitely more for the chapter in the thesis.

And finally whilst considering how to cut down paper on phase 1, we have discussed the possibility of removing the Leximancer information and either trying to publish something on the methodology, using the results tables as examples or  putting that information into the thesis.

A call for a study of evaluation practice

Improving Evaluation Theory through the Empirical Study of Evaluation Practice

Nick L Smith, 1993

The author states that few studies have been done on the practice of evaluation but that these are necessary in order to develop evaluation theory. The relationship between practice and theory often arises during metaevaluations as these tend to highlight the problem of translating current evaluation theory into acceptable evaluation practice. The author calls for increase in the number of research studies on evaluation practice (as opposed to evaluation of evaluation practice). At the time of writing this paper, Smith explains that ‘much of the current empirical justification of evaluation theories is from self-reported cases of what works’ (p238). He also quotes from Shadish, Cook and Leviton (1991) in this regard. Another 1991 paper by Marvin Alkin which studied the factors that influence theorists to change their conceptions of their own models or approaches is: increased experience in conducting evaluations, and accumulation of personal research on evaluation practice.

Smith explains that studies of evaluation practice are needed in order to know:

·      What works and how to make it happen

·      How to avoid bad practice

·      How local contexts differ and what works across different contexts

·      Where the problems of inadequacies of evaluation practice could be ameliorated by improved evaluation theory

If theories are presented in abstract, conceptual terms rather than in concrete terms based on how they would be applied in practice then we cannot know how practitioners actually articulate or operationalize various models or theories or whether they do so at all. So it becomes unclear what is meant when an evaluator claims to be using a particular theoretical approach. And if theories ‘cannot be uniquely operationalized then empirical tests of their utility become increasingly difficult’ (p240). Furthermore, if alternative theories give rise to similar practices, then theoretical differences may not be practically significant.

Smith discusses the use of contingency theories, an approach considered to be the strongest type of evaluation approach (Shadish, Cook and Leviton, 1991). These theories ‘seek to specify the conditions under which different evaluation practices are effective’ (p240). He then goes on to link this approach with the need for theoretical development alongside studies of practice. He calls for a continuation of public reflection by evaluation practitioners alongside more independent empirical studies of evaluation practice by evaluation researchers.

This article is some 20 years old now - need to check more recent articles from same author to see what has been done. Also check for articles citing this one.

Evaluation theory vs practice

What Guides Evaluation? A study of how evaluation practice maps on to evaluation theory.

Christina Christie 2003

This study came in response to repeated calls from theorists for more empirical knowledge of evaluation which would in turn help explain the nature of evaluation practice. A study with similar aims was carried out in 1987 by Shadish and Epstein however their study makes use of a survey instrument designed by the researchers. The Christie study realizes the fact that ‘theoretical orientation often cannot be accurately assessed through direct questioning because evaluation practitioners usually are not proficient in theory (Smith 1993), and so are unable to identify their particular theoretical orientation’ (P.9). A case study approach to observe behaviour of evaluation practitioners is the usual place to go however this offers depth but cannot cover breadth of understanding of how evaluators use theory to guide their work. This study is unique in that it uses eight distinguished evaluation theorists with a broad array of perspectives to construct the survey instrument. These theorists are Boruch, Chen, Cousins, Eisner, Fetterman, House, Patton and Stufflebeam.

The conceptual framework is built on the work of Alkin and House (1992) using their taxonomy of three dimensions: methods, values and use. Each dimension has a continuum that further defines it. For methods, its quantitative to qualitative; for values its unitary to pluralist (criteria used when making evaluative judgments); and for use its from enlightenment (academic) to instrumental (service-oriented). Each theorist was asked to submit one statement ‘demonstrating the practical application associated with his theoretical standpoint related to each of the three dimensions’. P11. They were also invited to submit up to six additional statements and in total, the final instrument contained 38 items related to evaluation practice and was piloted with 5 practicing evaluators.

The participants in this study were from 2 groups. The theorists were asked to complete the survey instrument and then 138 evaluators working on a statewide Californian educational program called Healthy Start. Many of these were not evaluators but school and program administrators and so represent a cross section of how evaluations are being conducted today. This group was also subdivided for reporting of results into internal and external evaluators. The collection of demographic data produced some interesting findings. The majority of evaluators were female, white and over 40. In terms of education, 75% of the external evaluators were PhD qualified but this aligns with the years of experience and self-rating of their evaluation knowledge and skills.

And a very interesting finding - only  a small proportion of evaluators in this study sample were using an explicit theoretical framework to guide their practice. More on this later.

The analytic procedure used multidimensional scaling (MDS) whereby observed similarities (or dissimilarities) are represented spatially as a geometric configuration of points on a map. More specifically, this study used classical multidimensional unfolding (CMDU) which is an individual-differences analysis that portrays differences in preference, perception, thinking or behaviour and can be use when studying differences in subjects in relationship to one another or to stimuli (p14). Furthermore, to interpret the CMDU results in this study, ALSCAL (Alternating Least-Square Scaling Algorithm) was used to produce two dimensions: scope of stakeholder involvement and method proclivity.

The first dimension ranged from users being simply an audience for the evaluation findings, to users being involved in the evaluation at all stages from start to finish. The second dimension, method proclivity is the extent to which the evaluation is guided by a prescribed technical approach. One end of this dimension would be characterized as partiality to a particular methodology that has as a feature predetermined research steps. Boruch anchored this end of the dimension as the experimental research design is at the centre of his approach to evaluation. The other end of this dimension represents partiality to framing evaluations by something other than a particular methodology with predetermined research steps. Patton for example falls to this end by use of his utilization-focused evaluation which is flexible in its nature calling for creative adaption during its problem solving approach.

So in this way, plotting the theorists’ responses on the two-dimensional axes helped to name and clarify the dimensions. The next step was to map the evaluators practice onto the dimensions in order to compare how practitioners rated against theorists. Evaluators were divided into two groups, internal and external due to noted professional characteristics. Results indicated that generally, stakeholders have a more limited role in evaluations conducted by internal evaluators than those conducted by external evaluators. In addition, internal evaluators are more partial to methodologies with predetermined research steps than are external evaluators. This analysis depicts only a broad depiction of their practice. By investigating placement in each quadrant of the map, a more comprehensive understanding is produced.

In general, external evaluators practice was more like the theorists. More specifically, they were most closely aligned with the theorist Cousins. Furthermore although they are concerned with stakeholder involvement, they are partial to their methods and conduct evaluations accordingly. Internal evaluators were distributed evenly between the theorists which reflect the diversity in their practices and implies that we cannot generalize about their categorization into any one genre of theoretical approaches.

Christie goes on to discuss the theorists results and concludes that through this study it has become evident that even though theories may share big picture goals, they don’t necessarily share the same theoretical tenets or similar practical frameworks. In addition, ‘the prescribed practices of a theory are not necessarily best reflected in its name or description’ (p.29). Her other major point was that ‘despite some theoretical concerns related to stakeholder involvement, all of the theorists in this study do involve stakeholders at least minimally’ (p.30). However many theorists have not chosen to incorporate changing notions of such involvement because of ‘a common perception that… it is understood to be a part of the evaluation process, no matter one’s theoretical approach’ (p.31).

In relation to practicing evaluators, they are often intimately involved in the program and therefore assume they understand how other stakeholders think and feel about the program and hence don’t tend to involve them a great deal. Politics may play out more heavily with internal evaluators and may influence their decision not only on stakeholder involvement but also to their emphasis on prescribed methods. In terms of evaluator bias, the study found that internal evaluators may be aware of the importance of objectivity and tend towards more quantitative methods to increase the credibility of their findings. External evaluators on the other hand may be influenced by their colleagues’ perception of their methods used, with the thought that criticism could jeopardize potential for future work. Therefore both types of evaluators employ method-driven frameworks influenced by the perception that the results are more defensible. And finally, this study shows that theory is not requisite to evaluation practice, in fact evaluators adopt only select portions of a given theory. Even ‘those who did claim to use a particular theory did not correspond empirically with the practices of the identified theorist’ (p33). Therefore Christie concludes that ‘the gap between the common evaluator and the notions of evaluation practice put forth by academic theorists has yet to be bridged.’ (p34).