analysis notes - phase 2

Starting a second cycle of coding on the first of three sets of data. Initial coding interspersed with some InVivo Coding and Versus coding was used in first cycle.

Codes were transposed into a spreadsheet and then colour coded using a focused coding (Charmaz, 2006) approach.

The categories thus far are:

• People (who are connected with a project such as steering group, audience etc)
• changing nature of projects; contextual factors
• project management information
• issues or challenges
• tame taken or timing
• types of evaluation or evaluand
• perceptions, affective language, emotions, conceptions
• communications
• quality

Another option for 2nd cycle coding is Elaborative coding (Auerbach & Silverstein, 2003). In this approach, findings from previous research can be supported, strengthened, modified or disconfirmed (Saldana 2013). If I use this, I can work with the first phase findings which unveiled 4 themes from across the 15 completed L&T projects: conflation between research and evaluation; capability building in evaluation; resources (time and money) and an action approach to evaluation.

At this point I will start with Focused coding and perhaps simultaneously note if any other previous themes appear in the current data set.

Once I complete this with the first set of data I can separately work on the 2nd and 3rd data sets (projects/ case studies). Then using Case Study methods I can compare and contrast  findings from each case.

research questions - phase 2

Some reminders for this phase:

Keep revisiting the questions that drive this part of the study:

• Why was a particular evaluation approach chosen and how successful was it?
• (ALT) why did you choose not to evaluate?
• How can we overcome barriers to successful evaluation praxis?
• How does perception influence evaluative practice?

Case studies help to answer these how and why questions.

These are really the Level 2 questions of the case study (as opposed to the Level 1 questions which are those asked in the interviews). The verbal line of enquiry (level 1) is different from the mental line of enquiry (level 2). (Yin, 2009, p 87).

Level 3 questions are those asked about the pattern of findings across multiple cases. In my study, we want to find out whether there were similar issues encountered with the evaluation and how such issues were dealt with.

Level 4 questions - about the entire study. How do findings relate to other literature or to say phase 1 of my study.


Case study evidence can (and should) come from multiple sources. These may include documents, interviews, participant observation, direct observation, physical artifacts and archival records. In this study the first four types were used.

Update Aug 2014

Where did the year go?? I was on a roll back in April, investigating types of coding and starting work thorugh the transcripts. Then suddently its the middle of August and I've completely forgotten where I am at!
I'll start by going back to the coding bible (Saldana) re reading my scribbles and trying to pick up where I left off.

I haven't been completely redundant. I've been scratching my head for a long time considering what to do about my rejection of paper 3 from the Journal of Further and Higher Education. This was the paper that reported outcomes from phase one of the project. I attempted to ask for a fourth reviewer because I felt that one of the reviewers had not read the paper or rather had jumped to inaccurate conclusions about what I was trying to do. Anyway that led nowhere so have now resubmitted to the International Journal of Educational Management. That was about a month ago so will sit tight and wait to hear.

In the meantime I have been trialling the modified Chesterton & Cummings Evaluation Framework with my OLT project for which I am an external evaluator. There has been good feedback from the team about how this has really helped them keep their focus and keep the project on track. We are also in discussion on writing a paper together to describe how the evaluation has worked for them.

I had a workshop abstract accepted at the Australasian Evaluation Society's annual conference in Darwin. However they were unable to attract enough participants so I am now reconsidering my attendance. It would have been a great opportunity to trial the framework and application with a wider audience.

And finally I have submitted an abstract to the AAIR (Australasian Association for Institutional Research) Forum 2014. This year they have partnered up with the Australasian Higher Education Evaluation Forum(AHEEF). I was particularly interested in two of the Themes: 

• Closing the Loop – Putting Evidence into Practice
• Influencing Change through Information and Evaluation

So am just waiting to hear back to see whether I have been successful.

sharing project info

This is an observation that comes from my experience of working on projects and interviewing project managers. Often people say they want feedback from stakeholders. Preferably in a formative sense so that feedback can be reviewed and enacted upon within the project if necessary. Often people choose to use a project wiki and publicise this in order to invite comments. I wonder how effective this is. Will/do people really comment and give feedback in such a public space?

Others often quote final reports and journal articles as effective means of dissemination but also as channels for feedback. Such summative methods are unlikely to provide useful feedback that can be incorporated unless the project has a second phase perhaps either through a new grant or other means.

Coding

Just finished reading a great book by Jonny Saldana, The coding manual for qualitative research (2013, SAGE). I've learned a lot of things, here's a summary.

I should have started coding as I was collecting my data rather than waiting till the end of the data collection period which was over 18mths.
The value in analytic memos should not be underestimated. Another way of saying this is reflective comments which I guess is what I did but next time I would formalise these and use them as sources of evidence for coding.

There are so many ways to code! I've tried to identify methods that may be suitable for my study and these include structural, descriptive, In Vivo, values, provisional, hypothesis, landscaping, focused and elaborative coding.

I have begun first cycle coding using Initial Coding (Charmaz, 2006) as a starting point. I'll then look at the codes generated and decide which other method of coding may be more appropriate for the next pass at the data.

During the first cycle coding process I have added many observer comments to the transcripts which I will use as a further data source.

I've so far resisted using CAQDAS or computer aided qualitative data analysis software ie NVivo. Mainly because it does not run on Mac and I don't have a windows computer. I have tried Leximancer to do a first pass on the data without much success. May come back to this and use as a triangulation method. This book has outlined other methods that are manual but make use of excel and word to manipulate and organise data. I'm going to manually write codes on the transcripts and then use these programs to arrange codes and data.

I need to document some of this process that i end up using and include in dissertation as a chapter. Only one or two paras would make it into a journal paper.

When I look at the artifacts such as reports and applications, critically reflect on them as they "reflect the interests and perspectives of their authors" (Hammersley & Atkinson, 2007 as cited in Saldana, 2013, p.54). They can also carry "values" (Hitchcock and Hughes, 1995 as cited in Saldana, 2013, p.54)

other notes:
Simultaneous Coding: "socail interaction does not occur in neat, isolated units" (Glesne, 2011, p.192).
Structural Coding: useful for hyposthesis testing (amongst other applications) - could be phase 2 aim..
Descriptive Coding: summarise in a short sentence or few words the topic of a paragraph of data. Basic method
In Vivo Coding: applicable to action and practitioner research (Coghlan & Branick, 2010; Fox, Martin & Green, 2007; Stringer, 1999)
Process Coding: useful for studies searching for "ongoing/interaction/emotion taken in response to situations or problems often with the purpose of reaching a goal or handling a problem" (Corbin & Strauss, 2008, pp96-97)
Values Coding: reflects participant's values, attitudes and beliefs representing her or his perspectives or world-view. One application is to explore experiences and actions in case studies. Must remember though that "values coding is values laden" (p.114)
Provisional Coding: builds on or corroborates previous research and investigations.
Causation Coding: Attribution refers to reasons or causal explanations. An attribution answers the 'why?' question. We should carefully consider the nuanced differences between a cause and a reason and a motive and to keep our focus primarily on people's intentions, choices, objectives, vakues, perspectives, expectations, needs, desires and agency within their particular contexts and circumstances (Morrison, 2009).

Finally, "coding is not a precise science it is primarily an interpretive act " (p.193)

From codes to themes -

case study evidence from the lit

"Although case studies have been used by anthropologists, psychoanalysts and many others as a method of exploration preliminary to theory development, the characteristics of the method are usually more suited to expansionist than reductionist pursuits. Theory building is the search for essences, pervasive and determining ingredients, and the makings of laws. The case study, however, proliferates rather than narrows. One is left with more to pay attention to rather than less. The case study attends to the idiosyncratic more than to the pervasive. The fact that it has been useful in theory building does not mean that that is its best use.

Its best use appears to me to be for adding to existing experience and humanistic understanding. Its characteristics match the ‘readiness’ people have for added experience. As Von Wright and others stressed, intentionality and empathy are central to the comprehension of social problems, but so also is information that is holistic and episodic. The discourse of persons struggling to increase their understanding of social matters features and solicits these qualities. And these qualities match nicely the characteristics of the case study." 

Stake, R. (2009). 1 THE CASE STUDY METHOD IN SOCIAL INQUIRY. In R. Gomm, M. Hammersley, & P. Foster (Eds.), Case Study Method. (pp. 18-27). London, England: SAGE Publications Ltd. doi: http://dx.doi.org.simsrad.net.ocs.mq.edu.au/10.4135/9780857024367.d5

Analyzing Case Study Data - Yin

The importance of having a clear analytical strategy is highlighted in this chapter of Yin (2009).  It is the least developed and most difficult aspect of conducting a case study and 'much relies on the investigator's own style of rigorous empirical thinking, along with the sufficient presentation of evidence and careful consideration of alternative interpretations' (p.127).
There is a short summary of the use of computer assisted tools but a reminder comes that they are only tools and only assistants, how the investigator manipulates the data is more important. It is recommended to first 'play' with the data and some ways in which to do this are summarised. For examples to create arrays; make a matrix of categories; use flowcharts and graphics; frequency of events; statistically analyze events; organising data chronologically.

All data has a story to tell and the needed analytical strategy will help you craft the story. Yin describes four general strategies.

Relying on theoretical propositions. Those that led to your case study, shaped the research questions and the literature review and therefore led to new hypotheses or propositions.

Developing a case description. Whilst less preferable to the previous strategy, a descriptive framework in which to organise the data can sometimes be useful particularly when the data has been collected without any theoretical propositions being made. Such a framework should have been developed before designing the data collection instruments. A descriptive approach may help to identify any causal links which can then be analysed.

Using both qualitative and quantitative data. There are two reasons why quantitative data may be relevant to your study. a) the data may cover the behaviour or events that your study is trying to explain. b) it may be related to an embedded unit of analysis within the broader study. Using statistical techniques to analyse this quantitative data at the same time as analysing the qualitative data will strengthen your study.

Examining rival explanations. This can be done within all of the previous three strategies and will strengthen the analysis if collection of evidence regarding 'other influences' is carried out. These rivals can be categorised as craft rivals (including null hypothesis, researcher bias and threats to validity) or real-life rivals. The more rivals a study addresses and rejects, the more confidence can be placed on findings in a case study.

The next section outlines five analytic techniques. Each one  needs to be practiced and the case study researcher will develop their own repertoire over time to produce compelling case studies.

Pattern Matching. This can strengthen a study's internal validity by comparing empirically based patterns with predicted ones. The following pattern types can be used:

• nonequivalent dependant variables as a pattern
• rival explanations s patterns
• simpler patterns

No matter the pattern type chosen, the more precise measures you can obtain, the stronger your argument/case study will be.

Explanation building.  When the data is analysed to explain the case. A parallel method for exploratory case studies can be used if the aim is to develop ideas for further study rather than in this case, to make conclusions. To explain something we stipulate a presumed set of causal links as to why or how it happened. Due to the imprecise nature of narratives, 'the better case studies are ones which have reflected some theoretically significant propositions' (p.141). The eventual explanation is likely to be a result of a series of iterations whereby the evidence is examined, theoretical propositions revised and the evidence examined again from a new perspective. As in pattern matching the aim is to show how rival explanations cannot be supported. The author notes however that this approach is 'fraught with dangers' and refers back to earlier advice to strengthen the study such as following a case study protocol, creating a case database and the following of a chain of evidence. If constant referral to the original source of inquiry is made and to the possible alternative explanations, the researcher is less likely to stray along a divergent path, during iterative cycles.

Time-Series Analysis. If there is just one dependent or independent variable we call it a simple time series. A mixed pattern across time can give rise to a complex time series and tracing events over time is known as a chronology. Care must be taken with this type of analysis not to simply describe or observe trends over time (this would be known as a chronicle) but rather to look for causal inferences.

Logic Models. This technique involves matching empirically observed events to theoretically predicted events and is differentiated from pattern matching because of their sequential stages. There are four types, related to the unit of analysis chosen.

• Individual-level 
• Firm or organisational-level (linear sequence)
• Firm or organisational-level (dynamic, multi-directional sequence)
• program-level

In each type, you should define your logic model then 'test' the model by analysing how well your data supports it. (p156).

Cross-Case Synthesis. This techniques treats each case as a separate study. One example is the use of word tables to display data for each case according to some framework. Then cross-case conclusions can be made. However care must be taken in developing 'strong argumentative interpretation' (p160).

Update Jan 2014

Agenda for this month is to update the gantt chart - where are we what are we adjusting/modifying: Acting as a project evaluator I am able to test out my framework - it may be useful to include this as a paper in my study as findings will certainly impact on stage 3. Actually if i think about it, that was what my original prposal suggested - that i use the framework to evaluate 3 local projects. I didn't end up doing that as it wasn't my role. Phase 2 has turned into a triangulation/measure of the findings from phase 1 and the lit review.
Thoughts on triangulation: use of three case studies offering an in depth look at the findings from phase one and the literature review - contextualising what's happening. Deeper richer insights to the findings.

There is a need to engage people and bring them into the process of evaluation - what needs to change? what comes next - this is the contribution to the knowledge.

Also need to think about publications and budget spending for this year: aiming for one new publication ie not including the article that went to JFHE. Need to target a journal for the case study manuscript.