Conferences 2013

{requirements - needs to have evaluation as one of the themes and must be in Australia (preferably Sydney)}

Possible contenders:

Australasian Higher Education Evaluation Forum (AHEEF) 2013 - host is Uni TAS but not advertised any info as yet.
- has refereed papers - 2.5 days.

Australasian Evaluation Society - Brisbane 2-6 september (Micheal Scriven is Keynoting)
- only require presentation proposal (due 21 March) 450 words. Presenters encouraged to submit paper to Evaluation Journal of Australasia (refereed).

To Do list for March 2013

Phase 2

1. Write up my case study protocol
2. organise catch ups with the three projects - preferably to attend their project meetings
3. send transcripts to PMs for their feedback
4. write new questions to ask the project leaders and get ethics clearance.

Phase 1

revisit the article reporting on the results - change style from a report to a story

Lit Review

1. Finalise this paper and submit to journal 
2. build chapter 2 of thesis with content deleted from that paper

Other

1. Find a conference for this year to share phase 1 findings
2. apply for funding - FoHS
3. submit conference application
4. submit update budget for the year - to include Nvivo training

case study design

Returning to Yin (2009) to review the details of Case Study research.

Many social scientists believe that case studies are only appropriate for the exploratory phase of an investigation. Yin describes how this is not the case. There are three conditions to be asked about the use of the case study. i) the form of the research question - case studies answer the how and why questions; ii) case studies do not require control of behavioural events and iii) the case study focuses on contemporary events.
The case study relies on two sources of evidence, direct observation of the events and interviews of the persons involved in the events.
Case studies like experiments are generalizable to theoretical propositions and not to populations or universes. Case studies can offer important evidence to complement experiments since experiments are limited in their ability to explain how or why a treatment necessarily worked.
Yin uses two features of a definition, one is about the scope and the other is based on the data collection and data analysis strategies.
He also discusses the critiques against case study research. He insists that case studies can and do use a mix of qualitative and quantitative evidence. They also have a distinctive place in evaluation research.

There are 5 components to a good research design:
1. the research questions - how or why is most common in case study research
2. the propositions (if relevant) - an exploratory study may not have these
3. the unit(s) of analysis - previous literature is often a guide for defining this
4. the logic linking the data to the propositions - eg pattern matching, logic models etc
5. the criteria for interpreting the findings - this is not usually done via statistical methods but often uses rival explanations for findings  to clarify current findings - this needs to be anticipated though to ensure correct data collection

Theory development is the ultimate aim of the design process. This may be in the form of developing or testing a theory. So 'the complete research design embodies a "theory" of what is being studied'. p.36

Now the question is, what is MY theoretical framework or philosophy underpinning my study? Or at least in phase 2 where I am using a case study approach.

Looking back to my proposal, I have stated the use of emergent realism, or pragmatism as a theoretical paradigm for my study since emergent realists do not insist that theirs is the only form of evaluative enquiry, on the contrary this paradigm encourages other forms and approaches to evaluation (Mark, Henry, & Julnes, 1998). Proponents of a pragmatic approach to mixed-methods designs state the importance of practicality, contextual responsiveness and consequentiality as important factors for success (p.44, Datta, 1997). Since the aim of this study is to investigate the most efficient evaluation strategies, as well as barriers to their success, utilising the theoretical framework of pragmatism, provides the benefit of enabling a consideration of what has previously worked (and what has not).

With any research design, we need to judge its quality, there are four common criteria for doing this. Yin explains how each criterion can be applied to the case study method.

• Construct validity - use multiple sources of evidence, establish chain of evidence or have key informants review case study draft. This is done at the data collection phase or in the third criterion, at the composition phase.
• Internal validity - pattern matching, explanation building, address rival explanations, or use logic models. These are all done at the data analysis phase.
• External validity - use theory (single case studies) or replication (multiple case studies). This is done at the research design phase .
• Reliability - use case study protocol or develop a case study database. These are done at the data collection phase.

Internal validity is mainly a concern for explanatory case studies as it becomes difficult to make inferences. With external validity care must be taken not to not to fall prey to the criticism that case studies cannot be generalised to a larger audience. This happens because there are no statistical generalizations being made but rather analytic generalisation. In terms of reliability, we must be sure to document procedures carefully.

The next step for me is to re-examine the different types of design (single vs. multiple and holistic vs. embedded). I had gone with the most complex of the matrix of 4, the multiple embedded with multiple units of analysis. Let's now break this down and explain what it means. I'll then define each of these elements for my study.

There are five main rationales used for using a single case study. A critical case, an extreme or unique case, a representative or typical case, a revelatory case or a longitudinal case ie looking at how things change over time. Within the single case option, the case can either be holistic or embedded. A common problem however of the holistic type of design may be that it lacks sufficient clear measures or data. In the embedded design, subunits of analyses are developed to focus the case study. These can stop the case slipping from its original intent.

Multiple case designs consist of more than one case within a study. This is the design I have chosen for stage 2. The evidence from multiple cases is considered more compelling leading to a more robust study. A mistake ofen made is to consider multiple cases as you would multiple respondents in a survey (which is sampling logic). This is not the case, instead each case should be 'replicated' in its design such that each can predict similar results (literal replication) or contrasting results for anticipated reasons (theoretical replication). So case studies are not the place for testing the presence of phenomena.

Again we turn to the option of holistic or embedded. When an embedded design is used, each study may  include the collection and analysis of quantitative data such as a survey. The choice between these two designs rests upon the type of phenomenon being studied and the research questions.

Looking back to my proposal, I stated the following research questions for phase 2:
-->

1. What is understood by evaluation?
2. What can be done to overcome barriers to successful project evaluation praxis
3. How successful was the modified approach to evaluation?
4. How can an interactive evaluation instrument benefit the process of evaluation of learning and teaching projects (if at all)?

 I now need to rethink these in light of the fact that case studies are designed to answer the how and why questions. I think i have begun to answer the first question in my lit review and phase 1. Having identified the barriers in phase 1, i now want to test them in this phase. So perhaps I need to rethink question 2. How can we overcome.... Question three perhaps should ask about their approach as each of the three cases is using a different evaluation approach. So why did you chose that approach and how successful was it. At this stage I think Q4 could be redundant. If the need for such an instrument emerges from the studies then that can become the question in the next phase.