Analysis of Phase One

The research questions for this phase are:

1. What evaluation forms and approaches have been used in Macquarie funded learning and teaching projects? [easy to answer from the interview data.]
2. What are the issues and challenges in evaluating learning and teaching projects?
   
   [This was the original Q11 in the interviews. Briefly, items would include:
   lack of skills - guidelines and support/resources needed
   initial plans too ambitious
   insufficient use of Stakeholders
   insufficient money/budget - to pay for extra help or input when needed i.e. admin support
   no feedback at any stage in the project
   lack of time
   • to plan
   • for reflection/learning
   • too busy with teaching and other demands]
3. What is understood by evaluation? [can look at the misuse or confusion in terminology.
   For example Evaluation vs. Research and Evaluation vs. Project in terms of both planning and results. There is also some misinterpretation between evaluation and feedback. Also look at Q12 from the interviews.]
4. How does perception influence how evaluation is carried out in practice? Or, What influences how evaluation is carried out in practice....... [for this i think i need to go through the data and pull out all examples where perception is discussed, albeit implicitly most times. Also need to look for other papers that have done this and see how they have done it. Need to refer back to the theoretical approach of the project as well as the realism paradigm]

Question 11 - challenges to conducting the evaluation

I'm now looking at the question 'were there any challenges to conducting the evaluation?'

An emerging theme that is coming through is that there is a lot of interference from contextual events. Running a project is something that is done at the same time as teaching, research, marking. Then there is the institutional requirements that occur simultaneously - the playing out of major projects and change procedures that also impact on the time available and possibly the outcomes. Now these all impact on how the project is run and not necessarily to the evaluation but it appears as a consequence, evaluation gets sidelined or is often not as rigorous as perhaps they initially hoped.

Some other themes that emerged:

• Time
•  Money
• Administration/resources
• Confusion in answering, talking about the challenges of conducting the research and not the evaluation.

When asked what could be done to overcome some of these issues, the following themes emerged:

• More money
• More time
• Better planning 
• More help with evaluation
• Developmental Evaluation

This last dot point arose because a few people were talking about using evaluation for checks and measures along the way, to refocus. And also to have evaluation incorporated into the approach so it becomes part of what you do and not something extra you have to make time for.

Question 1 - did you evaluate

I first looked at the question B1 - If no evaluation was carried out, why not?  From the 15 interviews, only 5 did project evaluation (or said/thought they did). From the 10 nos, I would say four of those did product evaluation in some format and one did some informal evaluation as part of an action research project which by its nature has reflection and redesign built in.
The five who said yes they did evaluation, three did evaluation as part of an action research approach and two did product evaluation.
To summarise:

Key:

Prod/Prog – evaluated a product or program which was the outcome of the project

AR/DE – used an action research approach so did a kind of developmental evaluation

	Did you evaluate?	type
1	No
2	No
3	No
4	Yes	DE
5	Yes	prog
6	Yes	AR
7	Yes	PAR
8	No	prod
9	No	prog
10	Yes	prog
11	No
12	No
13	No	Prod
14	No	Prog
15	No

There were three main themes that came through in the answers:

1. Different concepts of what is meant by evaluation – confusion over product, process and outcomes. Not a universal understanding of what evaluation is
2. Action research and developmental evaluation – informally evaluating whilst not actually writing formal reports etc
3. Lack of money and time to do evaluation