My Case Study Protocol (draft)

Questions directing this phase of the research:
Why was a particular evaluation approach chosen and how successful was it?
How can we overcome barriers to successful evaluation praxis?
How does perception influence evaluative practice?

Overview
It has been shown that evaluation of locally funded learning and teaching projects is not reported in the literature (reference - lit review). The studies that are reported have been analysed and applied to the context of HE to find that xxxxx
Three cases will be studied to further explore these findings. Each case is an internally funded learning and teaching project, running for 12 months. These projects came from a successful application for a grant. One was taken from each of the currently running grant programs at one Australian university, the teaching innovation and scholarship; the competitive grants and the priority grants [is this true?].  Project 1 comes from the department of Education and is titled 'MOOCS'; Project 2 is a joint research study between the Faculty of Arts and the Faculty of Human Sciences, investigating feedback methods; Project 3 is from the Office of Social Inclusion (?) and is investigating the embedded mentor program.

why did we select these? They were only ones that volunteered... Wanted to follow along with them as they progress - find out what they were doing in terms of evaluation.

Procedures
General plan:

• Receive Ethics clearance.
• Meet with the project team (if there is one). Show the list of questions which will be used as part of the data gathering instrument. Answer any of their questions about the study.
• First 'interview' for follow up on answers plus some background to project. Take information and consent forms. Record interview.
• Attend all of their project meetings (where possible) or presentations or focus grups etc and take notes which I will use in my reflections. Act as participant-as-observer.
• Meet two more times for 'interview', once after the progress report is due and again at the end of the project - perhaps after the final report is submitted. Each time, there will be a set of questions to be answered. Obtain further clearance from Ethics for these questions.
• Review other documentation including minutes of meetings, reports and application etc.

Questions
were there any identified barriers  and what did they do to overcome these (project leaders)
what approaches did they choose and why (project manager)
what was written about evaluation in the application (documents) and how does this compare to what actually happened (self reflections and interview data)
how did their individual experience with evaluation affect the way evaluation was conducted? (self reflection)
how were the projects similar and different in respect to their chosen approach and their identified barriers and their methods for overcoming barriers.
What does the literature say about such barriers (refer to lit review)
How important is the evaluation component in the selection of successful grant applications? (grant selection committee)

Report

At this stage I'm hoping to publish the findings from this case study in a journal or conference publication. Although having read Yin he suggests it is difficult to get this done, I will nontheless give it a go. I'm thinking about the corroboration of evidence and also the investigation of alternative propositions. If I could present something at a seminar or similar and get people to share like experiences/findings, this would be a great way to validate the findings of the study.

So who is the audience for this study? Who would benefit from the findings? I would say it is future project / grant holders. A list of guidelines to help with L&T project evaluation would certainly be helpful, aimed at how to overcome barriers etc. In addition, if we want to deal with the perception angle - then publishing findings in this area is one way to gather input.

Collecting Case Study Evidence

There are six major sources of evidence discussed in this chapter of Yin (2009). 

Documentation. Such as letters, emails, diaries, agendas, meeting minutes, proposals, progress reports, formal studies or evaluations of the same case and news clippings or other relevant media articles. Documents cannot be accepted as literal recordings of events but their value is in corroborating and augmenting  evidence from other sources. Inferences can be made but must be backed up by observed data, remembering that documents are written for a specific audience and purpose.

Archival records.  Such as census or statistical data, service records, budgets or personnel records, maps and charts, survey data about a particular group. Again these would have been produced for a specific purpose and audience so this must be taken into account when determining the usefulness of such data.

Interviews. These should take the form of guided conversations rather than structured queries. Yin stresses the importance of not asking interviewees the 'why' questions of your study directly but asking 'how'. This has a much softer, less accusatory tone. There are different types of interview, the in-depth and the focused interview. The former asks participants about the facts of the matter as well as their opinion of events. This obviously needs to be corroborated by other sources and also it is wise to search of contrary evidence to strengthen these findings. In the focused interview, you follow a certain set of questions derived from the case study protocol. Care must be taken not to ask leading questions. A third type of interview consists of a more structured questions along the lines of a survey. For all types of interview though, care must be taken with the fact that responses are subject to problems of bias, poor recall and poor or inaccurate articulation. As such, all interview data should be corroborated with other sources.

Direct Observation. This provides opportunities for gathering additional data about the topic under study. multiple observers can increase the reliability of this method though not always a practical option.

Participant-observation. Similar to the previous method, but with the advantage of a more in-depth understanding of the case due to taking on a role within the scenario. There are problems associated with this method however and these are related to potential biases that may occur in getting too involved. There is also the situation where the time to participate leaves little time to observe.

Physical artifact. These can be collected or observed and could include such items as a tool, a work of art, a printout of computer statistics such as usage etc. They do have less potential relevance in most typical case studies but can be used as supplementary evidence sources.

During collection of data from any of these six methods, there are three principles which can maximise the benefits of the data collection phase by dealing with the problem of construct validity and reliability of the case study evidence.

1. Use multiple sources of evidence. This has the advantage of using converging lines of enquiry, a process of triangulation and corroboration. Construct validity is addressed since 'multiple sources of evidence provide multiple measures of the same phenomenon'.(p117)

2. Create a case study database. This concerns how we organise and document the data collected and if done well (systematically) can increase the reliability of the case study. The aim is to allow independent investigation of the data should anyone want to query the findings or final case study report. There are four possible components to a database. 

• case study notes - these may come from interviews, observations, document analysis etc but they must be organised in such a way that they can be easily accessed later on by external parties (and yourself).
• case study documents - this can be weildly  but annotated bibliography of such documents can be useful. another method is to cross-reference documents in interview notes (say). As previously these must be organised for easy access.
• tabular materials - this could be a collection of quantitative results such as survey, observational counts or archival data.
• narratives - this is the case study investigators open-ended answers to the case study protocol's questions. Each answer represents an attempt to integrate the available evidence and to converge upon the facts of the matter or the tentative interpretation.

3. Maintain a chain of evidence. This will allow an external observer to follow the derivation of any evidence from the initial research questions to ultimate case study conclusions and thus increase the reliability of the case study. The chain runs from questions to protocol to evidence to database to final report, each linked forward and backwards (see p123 for a diagram).

Case study data collection

Yin (2009) identifies a number of skills required for a good case study researcher. One must be:

• a good listener
• able to ask good questions -
• adaptive and flexible
• have a firm grasp of the issues being studies and 
• unbiased by preconceived notions

A good way to approach this last point, is to report preliminary findings (to a conference or colleagues) and look for alternative explanations and suggestions. If these can be refuted then the likelihood of bias will have been reduced. 

In the case whereby a team is to conduct the case study research, Yin recommends training for the group as this can uncover a number of problems. Flaws with the case study design or research questions; incompatibilities within the investigating team; any impractical time deadlines or expectations of resources. Positive features may also be uncovered at a training session such as team members' complimentary skills.

Case study protocol. As mentioned before this is a major way of increasing the reliability of case study research. Recommened sections of this protocol include: an overview of the project; field procedures; case study questions; a guide for the case study report.

The overview includes background information such as the context and perspective. It also covers the substantive issues being covered, which would include the rationale for selecting the cases, any propositions or hypotheses being studied and the broader theoretical or policy relevance to the inquiry. Finally, the overview would include any relevant readings relating to the issues.

Field procedures refers to being prepared for the unexpected. Since the case study depends heavily on the interviewee, the researchers must be prepared to be flexible and responsive to the interviewee, must bring along all equipment to the field location and be prepared. This includes receiving ethics clearance where appropriate and providing information and consent forms for signing, from participants.

Case study questions. The questions in a case study protocol are different from those in a survey because they aim to reflect the actual line of enquiry and not a respondents point of view. The protocols questions are specific reminders of what information needs to be collected, and why so their purpose is to keep the researcher on track. There are different levels of questions ranging from the interview questions, to the questions to be asked of the case, to the questions to be asked of nthe pattern of findings across multiple cases, plus questions to be asked of literature or other published data related to the study and finally, possibly, to normative questions about policy documentation and conclusions. Yin uses a nice description for distinguishing particularly between the first two levels of questions in that 'The verbal line of inquiry is different from the mental line of enquiry' (p87).
Great care must be taken not to confuse the data collection source with the unit of analysis.

In my study, I am interested in an individual as the unit of analysis. However information should also be sourced (and questions asked) from the organisation - from reports, other employees and managers etc. I'm now thinking it may be a good idea to interview the project leaders not just the project managers. ie Matt, Mitch and Justin. And so need to come up with a list of questions to gather data from these sources. Also, I'm not crystal clear about my unit of analysis. I'm thinking it is the individual project but maybe I should be looking at the person as they are the ones exhibiting the behaviour, attitudes and perceptions about evaluation....

The final element is the guide for the case study report. One must think ahead to the audience for the report (as in a good evaluation) before data collection begins but also the format and outline of what will be included. Unlike other research methods where the sequence of events in the research are linear, a case study approach requires this out-of-step- method so that all possibilities for data collection are considered and reconsidered throughout the study, allowing for redesign as we go along. The other consideration for the report is how to include the large amount of documentary evidence that may form part of the data collection phase. This can be included as an annotated bibliography thus directing readers to locations if they require further evidence. One final note rom Yin here is that other research methods are often dictated by journal requirements but as case-study research is less likely to make it to a journal publication, the researcher is able to be more flexible in their approach to the method.

Yin completes this chapter with a short discussion on how to screen the candidates for a case study. Reasons for selection should be included in the protocol. Pilot case studies are also recommended whereby the pilot test offers the opportunity for formative feedback on the research questions and even some conceptual clarification for the research design.