writing sandwich

 The following was part of writers retreat:


area/discipline: education
topic: learning and teaching projects
sub topic: evaluation of such
2nd sub topic: higher education sector

working title:
a case study of the evaluation of learning and teaching projects in higher education: practices and perceptions

Problem:

In the higher education sector, grants can be obtained from internal and external sources to fund research into learning and teaching. The internal grants tend to be of around 12-18 month duration with specific guidelines as to how the projects should be conducted and results disseminated. Evaluation of these research projects should enable the research to become more rigorous and to supply evidence that the research is meeting its aims. However there some evidence from the literature that this evaluation is not taking place. One such study has investigated a number of completed internally funded learning and teaching projects and shown that evaluation is influenced by the perceptions of the project leaders.

Aim:
The aim of this study is to follow three project leaders as they conduct their research projects and invistigate how the evaluation unfolds. To identify factors that are at play which influence the evaluation. Observe what and how they evaluate and...


WOW factor of my project:
If money is to be spent wisely on research in learning and teaching, results and findings  of such research projects need to make an impact. We need to reflect and learn from such findings and incorporate them into the development cycle. This will ultimately result in better student learning. The findings from this study can be used to educate researchers about the value of evaluation and get them to see it as a learning mechanism rather than a punitive one. We also need to find a way to incorporate evaluation findings into the next round of projects such that they too can make an impact. So in a way to generalise findings so that a wider audience can benefit from the project.

Brown's 8 questions

Planning the next paper (4):

1. Who are the intended readers: grant bodies/grant application reviewers/ grant applicants
2. What did you do: Three recipients of locally funded learning and teaching project grants were 'followed' as they conducted their research projects over 12-18 months. There were three interviews with the project managers (beginning middle and end of project). And one interview with the project leaders (the person who applied for the grant and had the initial idea). Other project documentation was also used as evidence / data and project meetings were attended were relevant. The questions which guided the data collection were developed from the literature.
3. Why did you do it: To collect evidence as it was being generated rather than retrospectively when people have to 'recall' information which may be influenced by their perception of evaluation and other unknown factors. To see how perceptions changed over the course of the project and what influencing factors caused these changes.
4. what happened: all three had different experiences and perceptions of evaluation to begin with. This appeared to dictate how they conducted the evaluations and what they evaluated and what they did with any evaluation data.
5. What do the results mean in theory: don't know at this stage....
6. What do the results mean in practice: production of guidelines for project evaluation for/from the grant funding body. Development of a framework to guide project evaluation could arise from the findings.
7. What is the key benefit for readers: the grant bodies (ie application reviewers) could use the findings to guide their application and review processes. Academics and project leaders could use this framework to support them in their learning through the project and enable the institution to benefit more widely from the projects. 
8. What remains unsolved: How this framework could best be implemented, feedback from use of the framework.

Phase 2

There are so many competing actions needed right now I can't seem to move forward at all.
So this is my attempt to list what needs to be done and so form a plan of action.

1. write paper 1 - lit review identifying gaps and hence need for this project
2. thematic analysis of data from phase 1 - write  a paper on findings - need to adress the research questions identified in proposal
3. use findings from analysis to inform the development of an evaluation framework to be tested in phase 2
4. identify the projects to be used as case study in phase 2
5. write a plan of action for the case study approach - something to give to project leaders
6. have initial meetings with project leaders of two projects. Get ethics info and consent forms ready.

Action plan - phase 2

"The investigator will act in the role of Participant-as-Observer (Gold, 1958) of the evaluation and project process(es), actively participating with the project members and documentation and providing a depth to the research which would not be possible with an observer-only role (Babchuk, 1962)." (proposal, June, 2012)

1. Meet with the project team. Show the list of questions which will be used as part of the data gathering instrument. Answer any of their questions.
2. First 'interview' for follow up on answers. [need to develop some more questions but may be those from the 'notes' column in the table below.]
3. Explain that I will attend all of their project meetings (where possible) and take notes which i will use in my reflections. Act as participant-as-observer.
4. Then I will meet two more times for 'interview', once after the progress report is due and again at the end of the project - perhaps after the final report is submitted. Each time, there will be a set of questions to be answered and then I will follow up on these in more detail in the interviews.

So in total there will be three interview stages but perhaps a number of people will be interviewed.

Questions for first interview - Phase 2. (Time required – approx. 1hr)

Area	Questions	Notes
1. project clarification	What is the nature of the project? What is the focus of the project? What is the scope of the project? What are the intended outcomes? What (if any) are the project outputs? What are the operational processes developed to achieve the outcomes? What is the conceptual and theoretical framework underpinning the project? What is the context of the project? Are their any identified risks? What key values drive the project?	Are there sufficient resources/admin for this project? Are the plans too ambitious? What may happen to delay the project? Has teaching time been factored in?
2. Evaluation purpose and scope	What are you evaluating? Why is the evaluation being done? Are you basing the evaluation on any particular method framework or approach? How will the information be used? Who will evaluate this project? Are they suitably skilled? What value will the evaluation process add to the project?	Do they need training? Do they need some support resources?
3. Project Stakeholders and study audiences	Who are the stakeholders for the project and the audiences for the evaluation information? Stakeholders- Who has an interest or stake in the project and/or its outcomes, and in the evaluation of the project? Audiences - Who will be interested in the results of the study and what types of information do they expect from the evaluation? How should competing interests be prioritised?	Have you asked for feedback on the project? Use these to your advantage – to help guide the project. Be clear to understand the difference between two groups.
4. Key evaluation questions	What are the KEQ? Some examples could be: What processes were planned and what were actually put in place for the project? Were there any variations from the processes that were initially proposed, and if so, why? How might the project be improved? What were the observable short-term outcomes? To what extent have the intended outcomes been achieved? Were there any unintended outcomes? What factors helped and hindered in the achievement of the outcomes? What measures, if any, have been put in place to promote sustainability of the project’s focus and outcomes? What lessons have been learned from this project and how might these be of assistance to other institutions?
5. Data Collection Methods	How will the information be collected and analysed? What/who are the data sources? What types of data are most appropriate? What are the most appropriate methods of data collection? How will the data be analysed and presented in order to address the key evaluation questions? What ethical issues are involved in the evaluation and how will they be addressed?
6. Dissemination of Findings	How will the evaluation findings be disseminated? Who are the audiences for reports on the evaluation and what are their particular needs and interests? What are the functions of reporting? What reporting strategies will be used? When will reporting take place? What kinds of information will be included in evaluation reports?	Are the stakeholders involved in dissemination plans?
7. Evaluation Plan	What does your evaluation time line and activity schedule look like? What measures do you have in place to ensure you don’t run out of time for the evaluation to take place as planned? Who will you ask to review your evaluation plan?	Has time for reflection been built in to the plan?

Q6 - did the evaluation plan go to plan

This question asked whether an evaluation plan was written and presented in the application or the report. 9 interviewees said no and six said yes. Interesting that there were 7 people who originally stated that they evaluated (2 of which were evaluations of a product rather than the project) and yet only 6 had a plan. In fact two of the 6 yes's wrote a plan in their application but in actual fact they didn't end up evaluating. So to compare this:

	Evaluated the project?	Had an evaluation plan?
1	N	N
2	N	Y (in application)
3	N	Y (in application)
4	Y	N
5	Y	N
6	Y	Y
7	Y	N
8	N	N
9	N (evaluated the product)	Y
10	Y	Y
11	N	N
12	N	N
13	N (evaluated the product)	Y
14	N	N
15	N	N

Looking at the comments people made about whether the evaluation plan went to plan, there were two major themes running through the answers. The main one was that most people were thinking about the project – did IT run as planned, and not the evaluation (in some instances it could have been how I phrased the question). There appeared to be no linkage between having an evaluation plan and checking if it ran as planned ie no formative mechanisms or critical reflection checks along the way. The second theme was time. Most of those who talked about why the plan did not run as planned indicated that due to factors beyond their control or unplanned circumstances, the time it took to do the project/study meant either there was no time to do the evaluation or the project just ran out of time in general.
This could indicate that more time has to be built in to project planning for unexpected happenings that are bound to crop up.