Analysis of Phase One

The research questions for this phase are:

1. What evaluation forms and approaches have been used in Macquarie funded learning and teaching projects? [easy to answer from the interview data.]
2. What are the issues and challenges in evaluating learning and teaching projects?
   
   [This was the original Q11 in the interviews. Briefly, items would include:
   lack of skills - guidelines and support/resources needed
   initial plans too ambitious
   insufficient use of Stakeholders
   insufficient money/budget - to pay for extra help or input when needed i.e. admin support
   no feedback at any stage in the project
   lack of time
   • to plan
   • for reflection/learning
   • too busy with teaching and other demands]
3. What is understood by evaluation? [can look at the misuse or confusion in terminology.
   For example Evaluation vs. Research and Evaluation vs. Project in terms of both planning and results. There is also some misinterpretation between evaluation and feedback. Also look at Q12 from the interviews.]
4. How does perception influence how evaluation is carried out in practice? Or, What influences how evaluation is carried out in practice....... [for this i think i need to go through the data and pull out all examples where perception is discussed, albeit implicitly most times. Also need to look for other papers that have done this and see how they have done it. Need to refer back to the theoretical approach of the project as well as the realism paradigm]

Question 7 - Key evaluation questions

I think this set of questions - 'Were Key Evaluation questions stated?' and if yes could they be answered adequately, did not produce any fruitful data. Out of the 15 projects, 13 said no and for the two who said yes, there was no evidence of these questions in the aplication or the report.

This could suggest that there is no clear conception of what evaluation is. Though most people interviewed could talk about the topic, when put on the spot with this question, the lack of evidence points clearly towards the fact that evaluation is not well done or at least not clearly understood or maybe that there are different perceptions of evaluation. In other words there is no 'one size fits all'.
Interesting quote " although not stated, it is understood (implicit)". But is it??


Other reasons for there being no key evaluation questions stated could be due to the inherent nature of action research and developmental evaluation whereby the questions are developed as you go.

Question 3 - frameworks and shortcomings

These two questions ask which approach, framework or method was used to evaluate. There is limited data since only 7 of the 15 projects really did any kind of evaluation.

From these 7, 3 didn’t use any particular framework (could infer that they didn’t know any names of frameworks or didn't appreciate the benefit of using such).

The other 4 named frameworks – two created their own (but these were not rigorous or tested)

one cited CICTO which is used for software evaluation (not project eval)

and one cited PAR which is not an evaluation approach but a research approach – could say that this is developmental evaluation and could also infer that there is some confusion over what is difference between evaluation and research?


When asked about shortcomings of frameworks, the following themes were evident in answers:

1. External eyes would be helpful 
2. Formative and summative is imperative
3. Don’t be too ambitious – time is limited
4. Need a good project manager

Question 1 - did you evaluate

I first looked at the question B1 - If no evaluation was carried out, why not?  From the 15 interviews, only 5 did project evaluation (or said/thought they did). From the 10 nos, I would say four of those did product evaluation in some format and one did some informal evaluation as part of an action research project which by its nature has reflection and redesign built in.
The five who said yes they did evaluation, three did evaluation as part of an action research approach and two did product evaluation.
To summarise:

Key:

Prod/Prog – evaluated a product or program which was the outcome of the project

AR/DE – used an action research approach so did a kind of developmental evaluation

	Did you evaluate?	type
1	No
2	No
3	No
4	Yes	DE
5	Yes	prog
6	Yes	AR
7	Yes	PAR
8	No	prod
9	No	prog
10	Yes	prog
11	No
12	No
13	No	Prod
14	No	Prog
15	No

There were three main themes that came through in the answers:

1. Different concepts of what is meant by evaluation – confusion over product, process and outcomes. Not a universal understanding of what evaluation is
2. Action research and developmental evaluation – informally evaluating whilst not actually writing formal reports etc
3. Lack of money and time to do evaluation

evaluation or research

Leading on from last weeks ramblings on the differences between research and evaluation, Owen (2006 p.64) says that the inclusion of the planning and communicating stages is what differentiates evaluation from social research. The 'middle section' is the research which uses similar ranges of data collection and analysis techniques.

I want to concentrate on this difference. Try and prove that concentrating on the first and 3rd sections of the pie (picture) will help improve evaluation practices. ie the interactive instrument helps with these sections. and help with including time to do them.

Need more readings around time taken for each of these sections. Need to continue reading Owen.

Update: Nov 2011

Alkin & Taaut, 2003 write that the goal of research is generizable knowledge but the purpose of evaluation is context-specific (p.3). They Quote Cronbach & Suppes (1969) and conclusion-oriented research vs. decision-oriented evaluation.

update March 2012

Alkin (2011 - Evaluation Essentials form A to Z) states on p.8 that research seeks conclusions and evaluation leads to decisions. researchers ask their own questions in order to seek conclusions they can use to add to the knowledge bank. Evaluation answers questions that are important to a particular person - the stakeholder or client, say.

Alkin talks briefly about the definition of evaluation (which are goal orientated, around merit and worth) but he directs the readers focus to the processes which allow one to reach the point of being ready to judge merit and worth.

Update August 2012

reading Mackenzie, N. M. & Ling, L. M. (2009). The research journey: A Lonely Planet approach. Issues In Educational Research, 19(1), 48-60. http://www.iier.org.au/iier19/mackenzie.html

quote Mertens (Mertens, D. M. (2005). Research methods in education and psychology: Integrating diversity with quantitative and qualitative approaches (2nd ed.). Thousand Oaks, CA: SAGE.) though i can't give a page number of this as its in html (open access)


"This highlights the decisions we take as researchers when we aim for convergent outcomes or divergent outcomes. If our research is so prescribed and directed as to push us towards particular desired outcomes, it is convergent and in fact, may potentially not be research at all. Mertens (2005) makes a distinction between research and other forms of activity such as evaluation.

The relationship between research and evaluation is not simplistic. Much of evaluation can look remarkably like research and vice versa. Both make use of systemic inquiry methods to collect, analyse, interpret and use data to understand, describe, predict, control or empower. Evaluation is more typically associated with the need for information for decision making in a specific setting, and research is more typically associated with generating new knowledge that can be transferred to other settings. (p.2)

In fact, much of the prescribed and funded so-called research we undertake for convergent outcomes which fit the agenda of funding bodies is probably more akin to evaluation than research. Research which does not work towards pre-determined or prescribed outcomes and thus can produce divergent outcomes more in the spirit of what we understand as true research."

some words

empirical : derived from or relating to experiment and observation rather than theory
epistemology: The branch of philosophy that studies the nature of knowledge, its presuppositions and foundations, and its extent and validity.