Introduction - case study

30 minute writing exercise:

In the higher education sector, learning and teaching research projects constitute one avenue of funding. These projects are either internally or externally funded by various private and governmental bodies (reference). With the move towards greater accountability of public funding and the drive to increase quality education, the necessity to incorporate evaluation measures is growing. Not only to include evaluation but to build evaluation into the project life-cycle both systematically and rigorously. There is a wide body of literature on the evaluation of learning and teaching and the utilization of evaluation results to improve the student experience (references here).
A recent review of the learning and teaching project evaluation literature indicated that there is little evidence of the scholarship....
But there is little to no empirical research on the evaluation of learning and teaching projects in the higher education sector and how the benefits can be realised in practice. One recent research study conducted by the authors has begun to invesigate the factors that inhibit the use of evalaution practices in the sector nad found that project leaders perceptions are an influencing factor. However that paper called for further study into this area, more specifically to observe a learning and teaching project to examine how the perceptions play out over time and investigate what can be done in practice to support the effective use of evaluation for wider scale benefit to the organisation / institution and ultimately the student experience.
This study aims to investigate the evaluative measures used during three internally funded learning and teaching projects, and explore how the project leaders' perceptions of evaluation affected their praxis. Furthermore, this study will consider what can be done to overcome barriers to successful evaluation implementation and therefore enhance this praxis.

writing sandwich

 The following was part of writers retreat:


area/discipline: education
topic: learning and teaching projects
sub topic: evaluation of such
2nd sub topic: higher education sector

working title:
a case study of the evaluation of learning and teaching projects in higher education: practices and perceptions

Problem:

In the higher education sector, grants can be obtained from internal and external sources to fund research into learning and teaching. The internal grants tend to be of around 12-18 month duration with specific guidelines as to how the projects should be conducted and results disseminated. Evaluation of these research projects should enable the research to become more rigorous and to supply evidence that the research is meeting its aims. However there some evidence from the literature that this evaluation is not taking place. One such study has investigated a number of completed internally funded learning and teaching projects and shown that evaluation is influenced by the perceptions of the project leaders.

Aim:
The aim of this study is to follow three project leaders as they conduct their research projects and invistigate how the evaluation unfolds. To identify factors that are at play which influence the evaluation. Observe what and how they evaluate and...


WOW factor of my project:
If money is to be spent wisely on research in learning and teaching, results and findings  of such research projects need to make an impact. We need to reflect and learn from such findings and incorporate them into the development cycle. This will ultimately result in better student learning. The findings from this study can be used to educate researchers about the value of evaluation and get them to see it as a learning mechanism rather than a punitive one. We also need to find a way to incorporate evaluation findings into the next round of projects such that they too can make an impact. So in a way to generalise findings so that a wider audience can benefit from the project.

Brown's 8 questions

Planning the next paper (4):

1. Who are the intended readers: grant bodies/grant application reviewers/ grant applicants
2. What did you do: Three recipients of locally funded learning and teaching project grants were 'followed' as they conducted their research projects over 12-18 months. There were three interviews with the project managers (beginning middle and end of project). And one interview with the project leaders (the person who applied for the grant and had the initial idea). Other project documentation was also used as evidence / data and project meetings were attended were relevant. The questions which guided the data collection were developed from the literature.
3. Why did you do it: To collect evidence as it was being generated rather than retrospectively when people have to 'recall' information which may be influenced by their perception of evaluation and other unknown factors. To see how perceptions changed over the course of the project and what influencing factors caused these changes.
4. what happened: all three had different experiences and perceptions of evaluation to begin with. This appeared to dictate how they conducted the evaluations and what they evaluated and what they did with any evaluation data.
5. What do the results mean in theory: don't know at this stage....
6. What do the results mean in practice: production of guidelines for project evaluation for/from the grant funding body. Development of a framework to guide project evaluation could arise from the findings.
7. What is the key benefit for readers: the grant bodies (ie application reviewers) could use the findings to guide their application and review processes. Academics and project leaders could use this framework to support them in their learning through the project and enable the institution to benefit more widely from the projects. 
8. What remains unsolved: How this framework could best be implemented, feedback from use of the framework.

progress check

Well its half way through my 2nd full year of PhD and still finding my way. This year feels like it has mostly been writing and rewriting. Phase 1 findings have been 90% written up into a conference paper that was originally intended for the Australasian Evaluation Society Conference in Brisbane. It was accepted but then funding from the department was not forthcoming. So a last minute change of plans required a rewrite of the abstract and have recently received acceptance to The 5th Asian Conference on Education. It's in Osaka but I will hopefully present virtually (funding pending). My dilemma now is how to reduce what is about an 8 and a half thousand word paper into just 5000 for the conference. Also how do I ensure the paper is selected for the journal as a publication is really my intended outcome for participating in this conference.

My literature review has also been submitted to the Journal Studies in Educational Evaluation. This was peer reviewed and received considerable constructive feedback. However the editor did invite resubmission so that has been done and I'm now awaiting the second round of feedback (or rejection).

So it is time to return to data collection for phase 2. I'm transcribing interviews from phase 2 and need to soon turn to analysis and writing. But first need to do wrap up interviews for phase 2 case studies.

What have I learned from all this writing? That improvements can always be made; that people see and read different meanings than you sometimes intend; that writing short (5000 word) articles is difficult; that I am able to write.

Also, I need to do some more work on theory. I have briefly touched on the use of pragmatism or realism but it might be interesting to try and write either a short paper or definitely more for the chapter in the thesis.

And finally whilst considering how to cut down paper on phase 1, we have discussed the possibility of removing the Leximancer information and either trying to publish something on the methodology, using the results tables as examples or  putting that information into the thesis.

writing and writing and writing

Peer review is a funny old thing. My experience of publishing to date has not changed. Writing about something,  anything, is very subjective. You do it in a way that makes sense to yourself. But will it make sense to others? People have their own preferred style and if yours happens to align with your reviewers then a much smoother journey towards publication will be had.

I'm currently 'resting' my lit review and critique of the project evaluation literature. I've spent too many hours crafting and reading and rewriting. I would love to get it published (wouldn't anyone) but am considering just including it as part of the thesis only.

We had a discussion about this, this week ie whether to continue with Thesis by Publication or to revert to plain old thesis. My thoughts are that publishing matters. If i take time to research and find out information (and write it down) then I want to share that while it is current. As I'm planning to take a long while to complete my PhD, then in 6 or more years time, a lot of this information will be superseded. So I think for now I am sticking with it this way (by publication) and will continue to write and be reviewed.

The write/right journal

I'm going to try and publish my literature review but the question i'm now finding hard to answer is which journal. The suggestion was to target one of those journals mentioned in the paper. So for each of these I'm checking the journal requirements, particularly the length of papers accepted since this one is particularly long. Summaries of journals are on this page. Studies in evaluation seems like a good possibility, they have really long articles (over 9000 words) and 4 of the articles in my paper came from this journal. Another possibility is Assessment and Evaluation in Higher Education but they limit articles to between 3000 -5000 words. I'm also interested in the journal of multidisciplinary evaluation which has long articles - this is open source and states they have a quick turn around for feedback. They suggest a limit of 10-12 pages though they do accept longer articles.

Writing

I'm currently writing up my lit review, which is taking longer than I expected. Initially I had planned to conduct a review of the literature as i progressed through the PhD. However we have discussed wring this up as a paper defining why a study of the nature I proposed for my PhD is needed. ie a rational if you like.
One of my issues with this is that there will be more literature as I go and so this can't be one defining document. Marina suggested I publish the table in which I'm synthesising findings in the papers, on a public website so it can be constantly updated. I like this idea.
On a writing retreat this week I showed my writing group the efforts so far and I have to say I was a bit disappointed with the feedback at first. The general opinion was that I shouldn't try and publish this lit review as most journals would not be interested in it as after all who am I to tell the world..? They suggested I keep writing it up but that it becomes a chapter in the Thesis rather than a journal article. That's actually fine and I'll discuss this with my supervisors next week.

So where I am at right now is trying to weave the themes that have come from my searching, with the relevant items from the articles and then craft a suitable reflection/discussion and conclusion.
Themes I'm working with are:

Formative vs. summative evaluation

Building capacity in evaluation skills

Non-completion of project elements (such as evaluation, reports and the project itself)

Non-usage of evaluation findings

Importance of stakeholder involvement

Inaccurate initial expectations (could be linked to #2)

Importance of planning and defining clear evaluation criteria

Evaluation approaches, models or frameworks