Question 3 - frameworks and shortcomings

These two questions ask which approach, framework or method was used to evaluate. There is limited data since only 7 of the 15 projects really did any kind of evaluation.

From these 7, 3 didn’t use any particular framework (could infer that they didn’t know any names of frameworks or didn't appreciate the benefit of using such).

The other 4 named frameworks – two created their own (but these were not rigorous or tested)

one cited CICTO which is used for software evaluation (not project eval)

and one cited PAR which is not an evaluation approach but a research approach – could say that this is developmental evaluation and could also infer that there is some confusion over what is difference between evaluation and research?


When asked about shortcomings of frameworks, the following themes were evident in answers:

1. External eyes would be helpful 
2. Formative and summative is imperative
3. Don’t be too ambitious – time is limited
4. Need a good project manager

Q2 - skills of the evaluator

A range of evaluators were identified though mostly it was team approach. There were three themes that came out when asked whether the evaluator had sufficient skills in evaluation and whether they were able to be objective.

1. evaluation skills were lacking in the team
2. external pair of eyes may have been beneficial
3. hard to be truly objective 

This leads to findings from a later question that more support resources around evaluation are needed.

Question 1 - did you evaluate

I first looked at the question B1 - If no evaluation was carried out, why not?  From the 15 interviews, only 5 did project evaluation (or said/thought they did). From the 10 nos, I would say four of those did product evaluation in some format and one did some informal evaluation as part of an action research project which by its nature has reflection and redesign built in.
The five who said yes they did evaluation, three did evaluation as part of an action research approach and two did product evaluation.
To summarise:

Key:

Prod/Prog – evaluated a product or program which was the outcome of the project

AR/DE – used an action research approach so did a kind of developmental evaluation

	Did you evaluate?	type
1	No
2	No
3	No
4	Yes	DE
5	Yes	prog
6	Yes	AR
7	Yes	PAR
8	No	prod
9	No	prog
10	Yes	prog
11	No
12	No
13	No	Prod
14	No	Prog
15	No

There were three main themes that came through in the answers:

1. Different concepts of what is meant by evaluation – confusion over product, process and outcomes. Not a universal understanding of what evaluation is
2. Action research and developmental evaluation – informally evaluating whilst not actually writing formal reports etc
3. Lack of money and time to do evaluation

Trawling the Data

I've been playing with Leximancer for a few weeks now. To begin, I just loaded all of the transcripts into it to get a very complex themes and concept map. There was nothing strikingly interesting from that because there was lots of irrelevant information (ie included all of the questions too).

So now I am trying to extract the answers to each of the questions and anlyse them separately. This is proving really difficult because I was not consistent in my questioning. I often let the interviewee ramble off topic and did not get all of the questions answered adequately.

However going back through the data time and time again is good as it gives me a feel for what is being said. I'm not confident how to use Leximancer well, so i have supplemented this with two things. Firstly i have uploaded the transcript into Wordle to see what words come about. And secondly i have gone through the summaries i made of the interviews and looked to see what my themes were. By triangulating this evidence i hope to come up with something relevant.


The real learning in this is for me though. My questioning technique was really not consistent and next time i will read all questions even if i am repeating and say them clearly and not try and answer for the interviewee!

Visit to the OLT

Today we visited the Office for Learning and Teaching (OLT) in Sydney. We met with Siobhan Lenihan to discuss plans for my PhD project and to get feedback and ideas. Took the following notes:

ALTC changed policy to evaluate projects in 2009 - would be interesting to look at the difference before and after this policy change - what was done differently. Apparently they went from requiring summative evaluation to both formative and summative mechanisms.

contact C&C to find their results of their interviews - should be willing to share.

is there scope to find out what other HE ares doing? Look at the websites of QUT, Griffith, Monash and possibly Deakin

longer term impact - is there scope to do a follow up on projects, down the line to report on change and impact. I have founf this an interesting outcome of may of my interviews - people have expressed a desire to talk about their findings etc and have been pleased that there has been an interest in their work.


**James Dalziel has a teaching fellowship his evaluator is Grainne Conole, she is reviewing the outcomes at each stage of the followship**

visit OLT website and request read some of the evaluation reports - these can be sent.

networks national that could be used for dissemination opportunities - contact Melissa???

Promoting Excellence networks (PEN) - ask if they have knowledge about internal grant processes etc State-based grps with a uni rep on each

Lesley Wilcoxsen - Sally Kift took over the project "the whole of university experience"

Siobhan to send me - (Look out for) the upcoming report: Carmel McNought - Hong Kong Uni
large first year stem classes to do team work Glen Lawrie - project leader

the C&C framework has been tweaked slightly - have checked the changes - very minimal. Wondering if I should follow up on their recommended references/links and summarise some of these. They look like an exhaustive collection of evaluation materials.

Looking through a different lens

I decided this week that it would be a good idea to also interview the selection panel members who sit on the application review committee for the internal grants at MQ. This would give the perspective of what's required etc and fins out what value they put on evaluation.
Have written amendment to the ethics application and reworded the questions to suit. Will discus with Marina.

a meta-evaluation of a different kind

Means, Toyama, Murphy, Bakia, Jones (2010) Evaluation of Evidence-Based Pratices in online Learning: A Meta-Analysis and review of Online Learning Studies

I’m interested in this article for the methodology of how they went about their meta-evaluation (although I have to say the content is really interesting too and leads itself to the FLaMe program).

They specify the type of studies they were evaluating, these studies must have stringent designs (random-assignment or controlled quasi-experimental) and examine the effects for objective measures (of student learning). In relation to my project, I have not really specified the sample such as this and have used a sample of convenience – will this hold up to rigorous testing?

Furthermore I am not looking at quantitative studies. Evaluations tend to be more qualitative in their nature.

there was a main finding from the literature review - few rigorous studies in the area of interest had been published. In fact non had been found in the years of search and so the search was expanded by 2 years.

There were key findings from the meta-analysis and then there were findings from the narrative review. This review came from analysing the studies that were not able to be included in the meta-analysis due to the fact that they did not have a particular control condition. (p.xii)

Interesting comments on the potential for bias stemming from studies' authors dual roles as experimenters and instructors

The lit review and Meta-analysis were guided by four research questions. Context for the meta-analysis was given, explaining who commissioned the study (stakeholders) and the overall goal of the study. Then they described a conceptual framework for the topic [online learning] which included three key components.

Methodology

First define the topic [online learning], state what is included and what is not. Use this to define the categories to be searched for (three in this study).
List data sources and search strategies. Give years from and to and mention dates of different say for thesis etc.
List the databases that were used and the list of keywords (in the appendix).
Additional search activities - included a review of articles cited in recent meta-analyses and narrative syntheses of similar topics.
Key journals were identified and abstracts from each of these in a given period were manually reviewed.
A Google Scholar search engine was used with a series of keywords and any article abstracts retrieved were reviewed to ensure no duplication from pervious searches.

Scrrening was then carried out in 2 stages. Firstly abstracts were reviewed, giving studies the 'benefit of the doubt' (p.11) ensuring they met the inclusion criteria. Statement of number of papers included and number rejected (as percentages) with reasons for rejection.

Full text screen - the next pass had two stages, studies had to meet content relevance criteria (listed) as well as basic Quality (method) criteria (also listed.). A table of the primary reasons for exclusion was included detailing numbers for each reason and percentages. (p.13)

The coding of the study features was then detailed, regarding study features and study quality, then a paragraph explaining how the interrater reliability was checked.  Finally data analysis was explained, but this is completely statistical since the chosen studies were all quantitative, and a meta-analysis software was used for computations.