My Case Study Protocol (draft)

Questions directing this phase of the research:
Why was a particular evaluation approach chosen and how successful was it?
How can we overcome barriers to successful evaluation praxis?
How does perception influence evaluative practice?

Overview
It has been shown that evaluation of locally funded learning and teaching projects is not reported in the literature (reference - lit review). The studies that are reported have been analysed and applied to the context of HE to find that xxxxx
Three cases will be studied to further explore these findings. Each case is an internally funded learning and teaching project, running for 12 months. These projects came from a successful application for a grant. One was taken from each of the currently running grant programs at one Australian university, the teaching innovation and scholarship; the competitive grants and the priority grants [is this true?].  Project 1 comes from the department of Education and is titled 'MOOCS'; Project 2 is a joint research study between the Faculty of Arts and the Faculty of Human Sciences, investigating feedback methods; Project 3 is from the Office of Social Inclusion (?) and is investigating the embedded mentor program.

why did we select these? They were only ones that volunteered... Wanted to follow along with them as they progress - find out what they were doing in terms of evaluation.

Procedures
General plan:

• Receive Ethics clearance.
• Meet with the project team (if there is one). Show the list of questions which will be used as part of the data gathering instrument. Answer any of their questions about the study.
• First 'interview' for follow up on answers plus some background to project. Take information and consent forms. Record interview.
• Attend all of their project meetings (where possible) or presentations or focus grups etc and take notes which I will use in my reflections. Act as participant-as-observer.
• Meet two more times for 'interview', once after the progress report is due and again at the end of the project - perhaps after the final report is submitted. Each time, there will be a set of questions to be answered. Obtain further clearance from Ethics for these questions.
• Review other documentation including minutes of meetings, reports and application etc.

Questions
were there any identified barriers  and what did they do to overcome these (project leaders)
what approaches did they choose and why (project manager)
what was written about evaluation in the application (documents) and how does this compare to what actually happened (self reflections and interview data)
how did their individual experience with evaluation affect the way evaluation was conducted? (self reflection)
how were the projects similar and different in respect to their chosen approach and their identified barriers and their methods for overcoming barriers.
What does the literature say about such barriers (refer to lit review)
How important is the evaluation component in the selection of successful grant applications? (grant selection committee)

Report

At this stage I'm hoping to publish the findings from this case study in a journal or conference publication. Although having read Yin he suggests it is difficult to get this done, I will nontheless give it a go. I'm thinking about the corroboration of evidence and also the investigation of alternative propositions. If I could present something at a seminar or similar and get people to share like experiences/findings, this would be a great way to validate the findings of the study.

So who is the audience for this study? Who would benefit from the findings? I would say it is future project / grant holders. A list of guidelines to help with L&T project evaluation would certainly be helpful, aimed at how to overcome barriers etc. In addition, if we want to deal with the perception angle - then publishing findings in this area is one way to gather input.

Phase 2

There are so many competing actions needed right now I can't seem to move forward at all.
So this is my attempt to list what needs to be done and so form a plan of action.

1. write paper 1 - lit review identifying gaps and hence need for this project
2. thematic analysis of data from phase 1 - write  a paper on findings - need to adress the research questions identified in proposal
3. use findings from analysis to inform the development of an evaluation framework to be tested in phase 2
4. identify the projects to be used as case study in phase 2
5. write a plan of action for the case study approach - something to give to project leaders
6. have initial meetings with project leaders of two projects. Get ethics info and consent forms ready.

Action plan - phase 2

"The investigator will act in the role of Participant-as-Observer (Gold, 1958) of the evaluation and project process(es), actively participating with the project members and documentation and providing a depth to the research which would not be possible with an observer-only role (Babchuk, 1962)." (proposal, June, 2012)

1. Meet with the project team. Show the list of questions which will be used as part of the data gathering instrument. Answer any of their questions.
2. First 'interview' for follow up on answers. [need to develop some more questions but may be those from the 'notes' column in the table below.]
3. Explain that I will attend all of their project meetings (where possible) and take notes which i will use in my reflections. Act as participant-as-observer.
4. Then I will meet two more times for 'interview', once after the progress report is due and again at the end of the project - perhaps after the final report is submitted. Each time, there will be a set of questions to be answered and then I will follow up on these in more detail in the interviews.

So in total there will be three interview stages but perhaps a number of people will be interviewed.

Questions for first interview - Phase 2. (Time required – approx. 1hr)

Area	Questions	Notes
1. project clarification	What is the nature of the project? What is the focus of the project? What is the scope of the project? What are the intended outcomes? What (if any) are the project outputs? What are the operational processes developed to achieve the outcomes? What is the conceptual and theoretical framework underpinning the project? What is the context of the project? Are their any identified risks? What key values drive the project?	Are there sufficient resources/admin for this project? Are the plans too ambitious? What may happen to delay the project? Has teaching time been factored in?
2. Evaluation purpose and scope	What are you evaluating? Why is the evaluation being done? Are you basing the evaluation on any particular method framework or approach? How will the information be used? Who will evaluate this project? Are they suitably skilled? What value will the evaluation process add to the project?	Do they need training? Do they need some support resources?
3. Project Stakeholders and study audiences	Who are the stakeholders for the project and the audiences for the evaluation information? Stakeholders- Who has an interest or stake in the project and/or its outcomes, and in the evaluation of the project? Audiences - Who will be interested in the results of the study and what types of information do they expect from the evaluation? How should competing interests be prioritised?	Have you asked for feedback on the project? Use these to your advantage – to help guide the project. Be clear to understand the difference between two groups.
4. Key evaluation questions	What are the KEQ? Some examples could be: What processes were planned and what were actually put in place for the project? Were there any variations from the processes that were initially proposed, and if so, why? How might the project be improved? What were the observable short-term outcomes? To what extent have the intended outcomes been achieved? Were there any unintended outcomes? What factors helped and hindered in the achievement of the outcomes? What measures, if any, have been put in place to promote sustainability of the project’s focus and outcomes? What lessons have been learned from this project and how might these be of assistance to other institutions?
5. Data Collection Methods	How will the information be collected and analysed? What/who are the data sources? What types of data are most appropriate? What are the most appropriate methods of data collection? How will the data be analysed and presented in order to address the key evaluation questions? What ethical issues are involved in the evaluation and how will they be addressed?
6. Dissemination of Findings	How will the evaluation findings be disseminated? Who are the audiences for reports on the evaluation and what are their particular needs and interests? What are the functions of reporting? What reporting strategies will be used? When will reporting take place? What kinds of information will be included in evaluation reports?	Are the stakeholders involved in dissemination plans?
7. Evaluation Plan	What does your evaluation time line and activity schedule look like? What measures do you have in place to ensure you don’t run out of time for the evaluation to take place as planned? Who will you ask to review your evaluation plan?	Has time for reflection been built in to the plan?

Q13 - other comments

This was an interesting final question. There was a number of themes that came from each interviewees final responses.

A number of people talked about the need for support and resources on evaluation. Both during the time of application and also during the project. Suggested ideas included templates and guidelines as well as information available of different frameworks and their benefits.

A few people mentioned having time to look back at the project revisit it an look at impact - but I guess that would depend on whether the project produced an 'product' that could be evaluated for impact.

A few people mentioned the importance of incorporating the evaluation into the research cycle. And there was the mention of the importance of receiving constructive feedback from the university. This could be viewed as a need for identifying study audiences and stakeholders which was another theme that came out re the importance if one required some traction in implementing the outcomes of a project.

Another theme was the forms - make evaluation compulsory and give it importance by having a section on the application, but more importantly on the reporting pro-forma.

And finally the theme of networking was also evident. Participants mentioned the benefit of sharing findings with people who were interested ie L&T related research etc.

Q12 - the value of doing evaluation

This questions asked participants 'what value did evaluation add to your project'? Even though there were a number of projects that didn't conduct an evaluation, most people attempted to answer by saying hypothetically what they felt rather than what they observed.
The majority of responses mentioned the learning that takes place when one looks back, reflects. A couple of responses mentioned learning from mistakes and valuing that. Evaluation was also seen as a mechanism for keeping the project focussed.
One participant was concerned about the fine line between feeling like you are being checked on and being supported. Another participant agreed that there has to be some amount of accountability.

Question 11 - challenges to conducting the evaluation

I'm now looking at the question 'were there any challenges to conducting the evaluation?'

An emerging theme that is coming through is that there is a lot of interference from contextual events. Running a project is something that is done at the same time as teaching, research, marking. Then there is the institutional requirements that occur simultaneously - the playing out of major projects and change procedures that also impact on the time available and possibly the outcomes. Now these all impact on how the project is run and not necessarily to the evaluation but it appears as a consequence, evaluation gets sidelined or is often not as rigorous as perhaps they initially hoped.

Some other themes that emerged:

• Time
•  Money
• Administration/resources
• Confusion in answering, talking about the challenges of conducting the research and not the evaluation.

When asked what could be done to overcome some of these issues, the following themes emerged:

• More money
• More time
• Better planning 
• More help with evaluation
• Developmental Evaluation

This last dot point arose because a few people were talking about using evaluation for checks and measures along the way, to refocus. And also to have evaluation incorporated into the approach so it becomes part of what you do and not something extra you have to make time for.

Question 10 - reporting strategies and feedback

The following reporting strategies were used:

15 final report to funding body

6 reports or products to audience

5 interim reports

5 conference presentations

4 reports to stakeholders

2 journal articles

2 reports to department (meetings)

2 faculty L&T

1 minutes of project meetings to stakeholders

1 report to faculty executive 


When asked how their reports were received, 10 out of the 15 participants reported receiving no feedback other than thank-you. This provoked a range of reactions from indifference to relief to annoyance. I think it was clear though that more than a tokenistic feedback is required. Even if it was a follow up by someone to ask about the project - show interest in how it has developed or moved on with.


One participant mentioned that it would be good if they could receive help with publishing in the L&T field rather than their discipline field - which of course they can do easily. Considering a number of people mentioned conference and journal articles then perhaps this is a service the LTC could follow up with?

Question 9 - usability and generalisibility

The first part of this question asked whether the results of the evaluation were usable. The answers were interesting. 11 answered yes, two answered no and there were two projects who had no answer. In effect since only 5 projects did project evaluation (and a further 2 did some form of product evaluation) its interesting that 11 answered yes. This indicates that they were thinking along the lines of whether the project result were usable and not the evaluation results (although this may have been fault of the interviewer by not clarifying well enough). Still, another example of crossover in meaning of research and evaluation.
I think it would take a brave person to say there was nothing usable that came of their project. In fact one participant talked at length about this. That there was no room in the academic world for admitting failure.

So back to the 11 yes answers. The five projects that had evaluated were included in this number. Therefore it is good to see that there was in fact benefit to evaluating the projects. Furthermore, there were two projects who did product evaluation and one of these answered yes to the question but one answered no. This is an interesting case (14). The product was evaluation but eventually no one ended up implementing the recommendations for and resources produced in this project. reasons quoted included 'no time', 'no one was interested'. This leads to the question - about stakeholders being consulted with at time of application. When we look at the answer to this question, for this project we find that the project leader did not consult and in fact was not sure about what exactly a stakeholder is.

As for the part on Generalisability, there were a range of answers. Some stated categorically no, but others, when pressed reflected and stated various levels of yes from some to a lot. It was clear though that few had thought about this question, and obviously not reported on it. The other theme that emerges is the difference between content and process. Some were stuck in thinking of their project as a discipline specific thing that couldn't possibly be transposed but others mentioned that their process could be followed by anyone doing similar evaluation and projects, though interestingly few had actually explained the process in the report so how could another follow? Perhaps this level of details could be found in research publications.

Question 8 - review of evaluation plan

This set of questions asked whether the evaluation plan was reviewed and if so what was the benefit of this review.
From the 6 projects who previously said they had a plan, only 3 of them said they had it reviewed. However I have to say that these answers were a little misguided. What they really meant was that they got feedback on their project at some stage. One mentioned the steering group, another mentioned student and colleague's feedback on the product and the third mentioned fedback from the application review on who should be consulted during implementation of the project.
So in effect non of the evaluation plans was really reviewed before submission and just one mentioned formative checking in with the steering group.

Question 7 - Key evaluation questions

I think this set of questions - 'Were Key Evaluation questions stated?' and if yes could they be answered adequately, did not produce any fruitful data. Out of the 15 projects, 13 said no and for the two who said yes, there was no evidence of these questions in the aplication or the report.

This could suggest that there is no clear conception of what evaluation is. Though most people interviewed could talk about the topic, when put on the spot with this question, the lack of evidence points clearly towards the fact that evaluation is not well done or at least not clearly understood or maybe that there are different perceptions of evaluation. In other words there is no 'one size fits all'.
Interesting quote " although not stated, it is understood (implicit)". But is it??


Other reasons for there being no key evaluation questions stated could be due to the inherent nature of action research and developmental evaluation whereby the questions are developed as you go.

Q6 - did the evaluation plan go to plan

This question asked whether an evaluation plan was written and presented in the application or the report. 9 interviewees said no and six said yes. Interesting that there were 7 people who originally stated that they evaluated (2 of which were evaluations of a product rather than the project) and yet only 6 had a plan. In fact two of the 6 yes's wrote a plan in their application but in actual fact they didn't end up evaluating. So to compare this:

	Evaluated the project?	Had an evaluation plan?
1	N	N
2	N	Y (in application)
3	N	Y (in application)
4	Y	N
5	Y	N
6	Y	Y
7	Y	N
8	N	N
9	N (evaluated the product)	Y
10	Y	Y
11	N	N
12	N	N
13	N (evaluated the product)	Y
14	N	N
15	N	N

Looking at the comments people made about whether the evaluation plan went to plan, there were two major themes running through the answers. The main one was that most people were thinking about the project – did IT run as planned, and not the evaluation (in some instances it could have been how I phrased the question). There appeared to be no linkage between having an evaluation plan and checking if it ran as planned ie no formative mechanisms or critical reflection checks along the way. The second theme was time. Most of those who talked about why the plan did not run as planned indicated that due to factors beyond their control or unplanned circumstances, the time it took to do the project/study meant either there was no time to do the evaluation or the project just ran out of time in general.
This could indicate that more time has to be built in to project planning for unexpected happenings that are bound to crop up.

Q5 - Stakeholders (primary and secondary)

This question asked whether stakeholder and study audiences were identified (and hence consulted - though this wasn't asked but implied) and furthermore whether it was both primary and secondary stakeholders.
The aim of this set of questions was to see whether there was any understanding of the terminology and also the relevance of identifying these groups.
There is a good paper on working with Evaluation Stakeholders (Bryson, Patton & Bowman, 2011) which offers a rationale, step-wise approach and toolkit. They define stakeholders as 'individuals, groups or organisations that can affect or are affected by an evaluation process and/ or its findings. (p.1)

In Stufflebeam's 1974 Seminal paper on Meta-Evaluation (republished in 2011), one of his eleven essential criteria for technical adequacy of evaluations, is Relevance: 'This criterion concerns whether the findings respond to the purposes of the evaluation. What are the audiences? What information do they need? To what extent is the evaluation design responsive to the stated purposes of the study and the information requests made by the audiences? The concern for relevance is crucial if the findings are to have more than academic appeal and if they are to be used by the intended audiences. Application of the criterion of relevance requires that the evaluation audiences and purposes be specified. Such specifications essentially result in the questions to be answered. Relevance is determined by comparing each datum to be gathered with the questions to be answered.'

Themes from the interviews:

• Six projects mentioned students
• Some confusion about this topic
• Not thought of or implicitly stated in the application.

Q4 purpose and scope of evaluation

There were two themes that came out of this question. Most people did not detail the purpose and scope of the evaluation in the application or final report. (2 out of 15 - 27%). Most of these people explained why they didn't and that was because they felt it wasn't required, there was no importance laid on this. However many people [how many?] said they did scope the evaluation, (though there was no evidence in the report) which perhaps indicates that there was some confusion over what this question meant.

The follow up to this question asked how the information from the evaluation would be (planned) or was used. The aim was to see if there was any correlation to those that scoped the evaluation and how the evaluation results were actually used. Themes from the responses:

• To produce guidelines and / or recommendations
• Publications and conferences
• Unsure
• Feed into a larger grant

Question 3 - frameworks and shortcomings

These two questions ask which approach, framework or method was used to evaluate. There is limited data since only 7 of the 15 projects really did any kind of evaluation.

From these 7, 3 didn’t use any particular framework (could infer that they didn’t know any names of frameworks or didn't appreciate the benefit of using such).

The other 4 named frameworks – two created their own (but these were not rigorous or tested)

one cited CICTO which is used for software evaluation (not project eval)

and one cited PAR which is not an evaluation approach but a research approach – could say that this is developmental evaluation and could also infer that there is some confusion over what is difference between evaluation and research?


When asked about shortcomings of frameworks, the following themes were evident in answers:

1. External eyes would be helpful 
2. Formative and summative is imperative
3. Don’t be too ambitious – time is limited
4. Need a good project manager

Q2 - skills of the evaluator

A range of evaluators were identified though mostly it was team approach. There were three themes that came out when asked whether the evaluator had sufficient skills in evaluation and whether they were able to be objective.

1. evaluation skills were lacking in the team
2. external pair of eyes may have been beneficial
3. hard to be truly objective 

This leads to findings from a later question that more support resources around evaluation are needed.

Question 1 - did you evaluate

I first looked at the question B1 - If no evaluation was carried out, why not?  From the 15 interviews, only 5 did project evaluation (or said/thought they did). From the 10 nos, I would say four of those did product evaluation in some format and one did some informal evaluation as part of an action research project which by its nature has reflection and redesign built in.
The five who said yes they did evaluation, three did evaluation as part of an action research approach and two did product evaluation.
To summarise:

Key:

Prod/Prog – evaluated a product or program which was the outcome of the project

AR/DE – used an action research approach so did a kind of developmental evaluation

	Did you evaluate?	type
1	No
2	No
3	No
4	Yes	DE
5	Yes	prog
6	Yes	AR
7	Yes	PAR
8	No	prod
9	No	prog
10	Yes	prog
11	No
12	No
13	No	Prod
14	No	Prog
15	No

There were three main themes that came through in the answers:

1. Different concepts of what is meant by evaluation – confusion over product, process and outcomes. Not a universal understanding of what evaluation is
2. Action research and developmental evaluation – informally evaluating whilst not actually writing formal reports etc
3. Lack of money and time to do evaluation