This is an observation that comes from my experience of working on projects and interviewing project managers. Often people say they want feedback from stakeholders. Preferably in a formative sense so that feedback can be reviewed and enacted upon within the project if necessary. Often people choose to use a project wiki and publicise this in order to invite comments. I wonder how effective this is. Will/do people really comment and give feedback in such a public space?
Others often quote final reports and journal articles as effective means of dissemination but also as channels for feedback. Such summative methods are unlikely to provide useful feedback that can be incorporated unless the project has a second phase perhaps either through a new grant or other means.
Friday, April 25, 2014
Thursday, April 24, 2014
Coding
Just finished reading a great book by Jonny Saldana, The coding manual for qualitative research (2013, SAGE). I've learned a lot of things, here's a summary.
I should have started coding as I was collecting my data rather than waiting till the end of the data collection period which was over 18mths.
The value in analytic memos should not be underestimated. Another way of saying this is reflective comments which I guess is what I did but next time I would formalise these and use them as sources of evidence for coding.
There are so many ways to code! I've tried to identify methods that may be suitable for my study and these include structural, descriptive, In Vivo, values, provisional, hypothesis, landscaping, focused and elaborative coding.
I have begun first cycle coding using Initial Coding (Charmaz, 2006) as a starting point. I'll then look at the codes generated and decide which other method of coding may be more appropriate for the next pass at the data.
During the first cycle coding process I have added many observer comments to the transcripts which I will use as a further data source.
I've so far resisted using CAQDAS or computer aided qualitative data analysis software ie NVivo. Mainly because it does not run on Mac and I don't have a windows computer. I have tried Leximancer to do a first pass on the data without much success. May come back to this and use as a triangulation method. This book has outlined other methods that are manual but make use of excel and word to manipulate and organise data. I'm going to manually write codes on the transcripts and then use these programs to arrange codes and data.
I need to document some of this process that i end up using and include in dissertation as a chapter. Only one or two paras would make it into a journal paper.
When I look at the artifacts such as reports and applications, critically reflect on them as they "reflect the interests and perspectives of their authors" (Hammersley & Atkinson, 2007 as cited in Saldana, 2013, p.54). They can also carry "values" (Hitchcock and Hughes, 1995 as cited in Saldana, 2013, p.54)
other notes:
Simultaneous Coding: "socail interaction does not occur in neat, isolated units" (Glesne, 2011, p.192).
Structural Coding: useful for hyposthesis testing (amongst other applications) - could be phase 2 aim..
Descriptive Coding: summarise in a short sentence or few words the topic of a paragraph of data. Basic method
In Vivo Coding: applicable to action and practitioner research (Coghlan & Branick, 2010; Fox, Martin & Green, 2007; Stringer, 1999)
Process Coding: useful for studies searching for "ongoing/interaction/emotion taken in response to situations or problems often with the purpose of reaching a goal or handling a problem" (Corbin & Strauss, 2008, pp96-97)
Values Coding: reflects participant's values, attitudes and beliefs representing her or his perspectives or world-view. One application is to explore experiences and actions in case studies. Must remember though that "values coding is values laden" (p.114)
Provisional Coding: builds on or corroborates previous research and investigations.
Causation Coding: Attribution refers to reasons or causal explanations. An attribution answers the 'why?' question. We should carefully consider the nuanced differences between a cause and a reason and a motive and to keep our focus primarily on people's intentions, choices, objectives, vakues, perspectives, expectations, needs, desires and agency within their particular contexts and circumstances (Morrison, 2009).
Finally, "coding is not a precise science it is primarily an interpretive act " (p.193)
From codes to themes -
I should have started coding as I was collecting my data rather than waiting till the end of the data collection period which was over 18mths.
The value in analytic memos should not be underestimated. Another way of saying this is reflective comments which I guess is what I did but next time I would formalise these and use them as sources of evidence for coding.
There are so many ways to code! I've tried to identify methods that may be suitable for my study and these include structural, descriptive, In Vivo, values, provisional, hypothesis, landscaping, focused and elaborative coding.
I have begun first cycle coding using Initial Coding (Charmaz, 2006) as a starting point. I'll then look at the codes generated and decide which other method of coding may be more appropriate for the next pass at the data.
During the first cycle coding process I have added many observer comments to the transcripts which I will use as a further data source.
I've so far resisted using CAQDAS or computer aided qualitative data analysis software ie NVivo. Mainly because it does not run on Mac and I don't have a windows computer. I have tried Leximancer to do a first pass on the data without much success. May come back to this and use as a triangulation method. This book has outlined other methods that are manual but make use of excel and word to manipulate and organise data. I'm going to manually write codes on the transcripts and then use these programs to arrange codes and data.
I need to document some of this process that i end up using and include in dissertation as a chapter. Only one or two paras would make it into a journal paper.
When I look at the artifacts such as reports and applications, critically reflect on them as they "reflect the interests and perspectives of their authors" (Hammersley & Atkinson, 2007 as cited in Saldana, 2013, p.54). They can also carry "values" (Hitchcock and Hughes, 1995 as cited in Saldana, 2013, p.54)
other notes:
Simultaneous Coding: "socail interaction does not occur in neat, isolated units" (Glesne, 2011, p.192).
Structural Coding: useful for hyposthesis testing (amongst other applications) - could be phase 2 aim..
Descriptive Coding: summarise in a short sentence or few words the topic of a paragraph of data. Basic method
In Vivo Coding: applicable to action and practitioner research (Coghlan & Branick, 2010; Fox, Martin & Green, 2007; Stringer, 1999)
Process Coding: useful for studies searching for "ongoing/interaction/emotion taken in response to situations or problems often with the purpose of reaching a goal or handling a problem" (Corbin & Strauss, 2008, pp96-97)
Values Coding: reflects participant's values, attitudes and beliefs representing her or his perspectives or world-view. One application is to explore experiences and actions in case studies. Must remember though that "values coding is values laden" (p.114)
Provisional Coding: builds on or corroborates previous research and investigations.
Causation Coding: Attribution refers to reasons or causal explanations. An attribution answers the 'why?' question. We should carefully consider the nuanced differences between a cause and a reason and a motive and to keep our focus primarily on people's intentions, choices, objectives, vakues, perspectives, expectations, needs, desires and agency within their particular contexts and circumstances (Morrison, 2009).
Finally, "coding is not a precise science it is primarily an interpretive act " (p.193)
From codes to themes -
Saturday, February 15, 2014
case study evidence from the lit
"Although case studies have been used by anthropologists, psychoanalysts and many others as a method of exploration preliminary to theory development, the characteristics of the method are usually more suited to expansionist than reductionist pursuits. Theory building is the search for essences, pervasive and determining ingredients, and the makings of laws. The case study, however, proliferates rather than narrows. One is left with more to pay attention to rather than less. The case study attends to the idiosyncratic more than to the pervasive. The fact that it has been useful in theory building does not mean that that is its best use.
Its best use appears to me to be for adding to existing experience and humanistic understanding. Its characteristics match the ‘readiness’ people have for added experience. As Von Wright and others stressed, intentionality and empathy are central to the comprehension of social problems, but so also is information that is holistic and episodic. The discourse of persons struggling to increase their understanding of social matters features and solicits these qualities. And these qualities match nicely the characteristics of the case study."
Stake, R. (2009). 1 THE CASE STUDY METHOD IN SOCIAL INQUIRY. In R. Gomm, M. Hammersley, & P. Foster (Eds.), Case Study Method. (pp. 18-27). London, England: SAGE Publications Ltd. doi: http://dx.doi.org.simsrad.net.ocs.mq.edu.au/10.4135/9780857024367.d5
Analyzing Case Study Data - Yin
The importance of having a clear analytical strategy is highlighted in this chapter of Yin (2009). It is the least developed and most difficult aspect of conducting a case study and 'much relies on the investigator's own style of rigorous empirical thinking, along with the sufficient presentation of evidence and careful consideration of alternative interpretations' (p.127).
There is a short summary of the use of computer assisted tools but a reminder comes that they are only tools and only assistants, how the investigator manipulates the data is more important. It is recommended to first 'play' with the data and some ways in which to do this are summarised. For examples to create arrays; make a matrix of categories; use flowcharts and graphics; frequency of events; statistically analyze events; organising data chronologically.
All data has a story to tell and the needed analytical strategy will help you craft the story. Yin describes four general strategies.
Relying on theoretical propositions. Those that led to your case study, shaped the research questions and the literature review and therefore led to new hypotheses or propositions.
Developing a case description. Whilst less preferable to the previous strategy, a descriptive framework in which to organise the data can sometimes be useful particularly when the data has been collected without any theoretical propositions being made. Such a framework should have been developed before designing the data collection instruments. A descriptive approach may help to identify any causal links which can then be analysed.
Using both qualitative and quantitative data. There are two reasons why quantitative data may be relevant to your study. a) the data may cover the behaviour or events that your study is trying to explain. b) it may be related to an embedded unit of analysis within the broader study. Using statistical techniques to analyse this quantitative data at the same time as analysing the qualitative data will strengthen your study.
Examining rival explanations. This can be done within all of the previous three strategies and will strengthen the analysis if collection of evidence regarding 'other influences' is carried out. These rivals can be categorised as craft rivals (including null hypothesis, researcher bias and threats to validity) or real-life rivals. The more rivals a study addresses and rejects, the more confidence can be placed on findings in a case study.
The next section outlines five analytic techniques. Each one needs to be practiced and the case study researcher will develop their own repertoire over time to produce compelling case studies.
Pattern Matching. This can strengthen a study's internal validity by comparing empirically based patterns with predicted ones. The following pattern types can be used:
No matter the pattern type chosen, the more precise measures you can obtain, the stronger your argument/case study will be.
Explanation building. When the data is analysed to explain the case. A parallel method for exploratory case studies can be used if the aim is to develop ideas for further study rather than in this case, to make conclusions. To explain something we stipulate a presumed set of causal links as to why or how it happened. Due to the imprecise nature of narratives, 'the better case studies are ones which have reflected some theoretically significant propositions' (p.141). The eventual explanation is likely to be a result of a series of iterations whereby the evidence is examined, theoretical propositions revised and the evidence examined again from a new perspective. As in pattern matching the aim is to show how rival explanations cannot be supported. The author notes however that this approach is 'fraught with dangers' and refers back to earlier advice to strengthen the study such as following a case study protocol, creating a case database and the following of a chain of evidence. If constant referral to the original source of inquiry is made and to the possible alternative explanations, the researcher is less likely to stray along a divergent path, during iterative cycles.
Time-Series Analysis. If there is just one dependent or independent variable we call it a simple time series. A mixed pattern across time can give rise to a complex time series and tracing events over time is known as a chronology. Care must be taken with this type of analysis not to simply describe or observe trends over time (this would be known as a chronicle) but rather to look for causal inferences.
Logic Models. This technique involves matching empirically observed events to theoretically predicted events and is differentiated from pattern matching because of their sequential stages. There are four types, related to the unit of analysis chosen.
In each type, you should define your logic model then 'test' the model by analysing how well your data supports it. (p156).
Cross-Case Synthesis. This techniques treats each case as a separate study. One example is the use of word tables to display data for each case according to some framework. Then cross-case conclusions can be made. However care must be taken in developing 'strong argumentative interpretation' (p160).
There is a short summary of the use of computer assisted tools but a reminder comes that they are only tools and only assistants, how the investigator manipulates the data is more important. It is recommended to first 'play' with the data and some ways in which to do this are summarised. For examples to create arrays; make a matrix of categories; use flowcharts and graphics; frequency of events; statistically analyze events; organising data chronologically.
All data has a story to tell and the needed analytical strategy will help you craft the story. Yin describes four general strategies.
Relying on theoretical propositions. Those that led to your case study, shaped the research questions and the literature review and therefore led to new hypotheses or propositions.
Developing a case description. Whilst less preferable to the previous strategy, a descriptive framework in which to organise the data can sometimes be useful particularly when the data has been collected without any theoretical propositions being made. Such a framework should have been developed before designing the data collection instruments. A descriptive approach may help to identify any causal links which can then be analysed.
Using both qualitative and quantitative data. There are two reasons why quantitative data may be relevant to your study. a) the data may cover the behaviour or events that your study is trying to explain. b) it may be related to an embedded unit of analysis within the broader study. Using statistical techniques to analyse this quantitative data at the same time as analysing the qualitative data will strengthen your study.
Examining rival explanations. This can be done within all of the previous three strategies and will strengthen the analysis if collection of evidence regarding 'other influences' is carried out. These rivals can be categorised as craft rivals (including null hypothesis, researcher bias and threats to validity) or real-life rivals. The more rivals a study addresses and rejects, the more confidence can be placed on findings in a case study.
The next section outlines five analytic techniques. Each one needs to be practiced and the case study researcher will develop their own repertoire over time to produce compelling case studies.
Pattern Matching. This can strengthen a study's internal validity by comparing empirically based patterns with predicted ones. The following pattern types can be used:
- nonequivalent dependant variables as a pattern
- rival explanations s patterns
- simpler patterns
No matter the pattern type chosen, the more precise measures you can obtain, the stronger your argument/case study will be.
Explanation building. When the data is analysed to explain the case. A parallel method for exploratory case studies can be used if the aim is to develop ideas for further study rather than in this case, to make conclusions. To explain something we stipulate a presumed set of causal links as to why or how it happened. Due to the imprecise nature of narratives, 'the better case studies are ones which have reflected some theoretically significant propositions' (p.141). The eventual explanation is likely to be a result of a series of iterations whereby the evidence is examined, theoretical propositions revised and the evidence examined again from a new perspective. As in pattern matching the aim is to show how rival explanations cannot be supported. The author notes however that this approach is 'fraught with dangers' and refers back to earlier advice to strengthen the study such as following a case study protocol, creating a case database and the following of a chain of evidence. If constant referral to the original source of inquiry is made and to the possible alternative explanations, the researcher is less likely to stray along a divergent path, during iterative cycles.
Time-Series Analysis. If there is just one dependent or independent variable we call it a simple time series. A mixed pattern across time can give rise to a complex time series and tracing events over time is known as a chronology. Care must be taken with this type of analysis not to simply describe or observe trends over time (this would be known as a chronicle) but rather to look for causal inferences.
Logic Models. This technique involves matching empirically observed events to theoretically predicted events and is differentiated from pattern matching because of their sequential stages. There are four types, related to the unit of analysis chosen.
- Individual-level
- Firm or organisational-level (linear sequence)
- Firm or organisational-level (dynamic, multi-directional sequence)
- program-level
In each type, you should define your logic model then 'test' the model by analysing how well your data supports it. (p156).
Cross-Case Synthesis. This techniques treats each case as a separate study. One example is the use of word tables to display data for each case according to some framework. Then cross-case conclusions can be made. However care must be taken in developing 'strong argumentative interpretation' (p160).
Tuesday, February 4, 2014
Update Jan 2014
Agenda for this month is to update the gantt chart - where are we what are we adjusting/modifying: Acting as a project evaluator I am able to test out my framework - it may be useful to include this as a paper in my study as findings will certainly impact on stage 3. Actually if i think about it, that was what my original prposal suggested - that i use the framework to evaluate 3 local projects. I didn't end up doing that as it wasn't my role. Phase 2 has turned into a triangulation/measure of the findings from phase 1 and the lit review.
Thoughts on triangulation: use of three case studies offering an in depth look at the findings from phase one and the literature review - contextualising what's happening. Deeper richer insights to the findings.
There is a need to engage people and bring them into the process of evaluation - what needs to change? what comes next - this is the contribution to the knowledge.
Also need to think about publications and budget spending for this year: aiming for one new publication ie not including the article that went to JFHE. Need to target a journal for the case study manuscript.
Thoughts on triangulation: use of three case studies offering an in depth look at the findings from phase one and the literature review - contextualising what's happening. Deeper richer insights to the findings.
There is a need to engage people and bring them into the process of evaluation - what needs to change? what comes next - this is the contribution to the knowledge.
Also need to think about publications and budget spending for this year: aiming for one new publication ie not including the article that went to JFHE. Need to target a journal for the case study manuscript.
Thursday, October 24, 2013
case study method - draft
What data will be collected and how?
Invitations to participate in this study were sent to the awardees of all Learning and Teaching grants in 2012. There were three respondents who agreed to become a case study. The projects were from the School of Education; the Faculties of Human Sciences and Arts; and the Office for First Year Experience (FYE). Each project had a project leader and a project manager. Both were briefed on the case study protocol (see Appendix A). This included a minimum of three interviews with the project manager over the duration of the project (beginning, middle and end) and an interview with the project leader. These interviews were recorded for transcription, then transcripts were sent to the participants for review. Access to documentation including grant applications, minutes of any project meetings and final reports was also part of the case study protocol. There was also a request to attend any project meetings as a participant observer. This was only possible in one project.
How will the data be analysed?
Data generated in case study research can become unwieldy...(because?). The amount collected for this study was kept to a minimum by following just three projects. Each project was analysed separately and results compared. Content analysis of the interview and project documentation data was carried out using the manual extraction of themes (Krippendorf, date).
limitations to the methodology:
generalising results from just three cases can be problematic. However results could then be used in a pilot or wider study.
Invitations to participate in this study were sent to the awardees of all Learning and Teaching grants in 2012. There were three respondents who agreed to become a case study. The projects were from the School of Education; the Faculties of Human Sciences and Arts; and the Office for First Year Experience (FYE). Each project had a project leader and a project manager. Both were briefed on the case study protocol (see Appendix A). This included a minimum of three interviews with the project manager over the duration of the project (beginning, middle and end) and an interview with the project leader. These interviews were recorded for transcription, then transcripts were sent to the participants for review. Access to documentation including grant applications, minutes of any project meetings and final reports was also part of the case study protocol. There was also a request to attend any project meetings as a participant observer. This was only possible in one project.
How will the data be analysed?
Data generated in case study research can become unwieldy...(because?). The amount collected for this study was kept to a minimum by following just three projects. Each project was analysed separately and results compared. Content analysis of the interview and project documentation data was carried out using the manual extraction of themes (Krippendorf, date).
limitations to the methodology:
generalising results from just three cases can be problematic. However results could then be used in a pilot or wider study.
Introduction - case study
30 minute writing exercise:
In the higher education sector, learning and teaching research projects constitute one avenue of funding. These projects are either internally or externally funded by various private and governmental bodies (reference). With the move towards greater accountability of public funding and the drive to increase quality education, the necessity to incorporate evaluation measures is growing. Not only to include evaluation but to build evaluation into the project life-cycle both systematically and rigorously. There is a wide body of literature on the evaluation of learning and teaching and the utilization of evaluation results to improve the student experience (references here).
A recent review of the learning and teaching project evaluation literature indicated that there is little evidence of the scholarship....
But there is little to no empirical research on the evaluation of learning and teaching projects in the higher education sector and how the benefits can be realised in practice. One recent research study conducted by the authors has begun to invesigate the factors that inhibit the use of evalaution practices in the sector nad found that project leaders perceptions are an influencing factor. However that paper called for further study into this area, more specifically to observe a learning and teaching project to examine how the perceptions play out over time and investigate what can be done in practice to support the effective use of evaluation for wider scale benefit to the organisation / institution and ultimately the student experience.
This study aims to investigate the evaluative measures used during three internally funded learning and teaching projects, and explore how the project leaders' perceptions of evaluation affected their praxis. Furthermore, this study will consider what can be done to overcome barriers to successful evaluation implementation and therefore enhance this praxis.
In the higher education sector, learning and teaching research projects constitute one avenue of funding. These projects are either internally or externally funded by various private and governmental bodies (reference). With the move towards greater accountability of public funding and the drive to increase quality education, the necessity to incorporate evaluation measures is growing. Not only to include evaluation but to build evaluation into the project life-cycle both systematically and rigorously. There is a wide body of literature on the evaluation of learning and teaching and the utilization of evaluation results to improve the student experience (references here).
A recent review of the learning and teaching project evaluation literature indicated that there is little evidence of the scholarship....
But there is little to no empirical research on the evaluation of learning and teaching projects in the higher education sector and how the benefits can be realised in practice. One recent research study conducted by the authors has begun to invesigate the factors that inhibit the use of evalaution practices in the sector nad found that project leaders perceptions are an influencing factor. However that paper called for further study into this area, more specifically to observe a learning and teaching project to examine how the perceptions play out over time and investigate what can be done in practice to support the effective use of evaluation for wider scale benefit to the organisation / institution and ultimately the student experience.
This study aims to investigate the evaluative measures used during three internally funded learning and teaching projects, and explore how the project leaders' perceptions of evaluation affected their praxis. Furthermore, this study will consider what can be done to overcome barriers to successful evaluation implementation and therefore enhance this praxis.
Subscribe to:
Posts (Atom)