Q6 - did the evaluation plan go to plan

This question asked whether an evaluation plan was written and presented in the application or the report. 9 interviewees said no and six said yes. Interesting that there were 7 people who originally stated that they evaluated (2 of which were evaluations of a product rather than the project) and yet only 6 had a plan. In fact two of the 6 yes's wrote a plan in their application but in actual fact they didn't end up evaluating. So to compare this:

	Evaluated the project?	Had an evaluation plan?
1	N	N
2	N	Y (in application)
3	N	Y (in application)
4	Y	N
5	Y	N
6	Y	Y
7	Y	N
8	N	N
9	N (evaluated the product)	Y
10	Y	Y
11	N	N
12	N	N
13	N (evaluated the product)	Y
14	N	N
15	N	N

Looking at the comments people made about whether the evaluation plan went to plan, there were two major themes running through the answers. The main one was that most people were thinking about the project – did IT run as planned, and not the evaluation (in some instances it could have been how I phrased the question). There appeared to be no linkage between having an evaluation plan and checking if it ran as planned ie no formative mechanisms or critical reflection checks along the way. The second theme was time. Most of those who talked about why the plan did not run as planned indicated that due to factors beyond their control or unplanned circumstances, the time it took to do the project/study meant either there was no time to do the evaluation or the project just ran out of time in general.
This could indicate that more time has to be built in to project planning for unexpected happenings that are bound to crop up.

Q5 - Stakeholders (primary and secondary)

This question asked whether stakeholder and study audiences were identified (and hence consulted - though this wasn't asked but implied) and furthermore whether it was both primary and secondary stakeholders.
The aim of this set of questions was to see whether there was any understanding of the terminology and also the relevance of identifying these groups.
There is a good paper on working with Evaluation Stakeholders (Bryson, Patton & Bowman, 2011) which offers a rationale, step-wise approach and toolkit. They define stakeholders as 'individuals, groups or organisations that can affect or are affected by an evaluation process and/ or its findings. (p.1)

In Stufflebeam's 1974 Seminal paper on Meta-Evaluation (republished in 2011), one of his eleven essential criteria for technical adequacy of evaluations, is Relevance: 'This criterion concerns whether the findings respond to the purposes of the evaluation. What are the audiences? What information do they need? To what extent is the evaluation design responsive to the stated purposes of the study and the information requests made by the audiences? The concern for relevance is crucial if the findings are to have more than academic appeal and if they are to be used by the intended audiences. Application of the criterion of relevance requires that the evaluation audiences and purposes be specified. Such specifications essentially result in the questions to be answered. Relevance is determined by comparing each datum to be gathered with the questions to be answered.'

Themes from the interviews:

• Six projects mentioned students
• Some confusion about this topic
• Not thought of or implicitly stated in the application.

Q4 purpose and scope of evaluation

There were two themes that came out of this question. Most people did not detail the purpose and scope of the evaluation in the application or final report. (2 out of 15 - 27%). Most of these people explained why they didn't and that was because they felt it wasn't required, there was no importance laid on this. However many people [how many?] said they did scope the evaluation, (though there was no evidence in the report) which perhaps indicates that there was some confusion over what this question meant.

The follow up to this question asked how the information from the evaluation would be (planned) or was used. The aim was to see if there was any correlation to those that scoped the evaluation and how the evaluation results were actually used. Themes from the responses:

• To produce guidelines and / or recommendations
• Publications and conferences
• Unsure
• Feed into a larger grant

Question 3 - frameworks and shortcomings

These two questions ask which approach, framework or method was used to evaluate. There is limited data since only 7 of the 15 projects really did any kind of evaluation.

From these 7, 3 didn’t use any particular framework (could infer that they didn’t know any names of frameworks or didn't appreciate the benefit of using such).

The other 4 named frameworks – two created their own (but these were not rigorous or tested)

one cited CICTO which is used for software evaluation (not project eval)

and one cited PAR which is not an evaluation approach but a research approach – could say that this is developmental evaluation and could also infer that there is some confusion over what is difference between evaluation and research?


When asked about shortcomings of frameworks, the following themes were evident in answers:

1. External eyes would be helpful 
2. Formative and summative is imperative
3. Don’t be too ambitious – time is limited
4. Need a good project manager

Q2 - skills of the evaluator

A range of evaluators were identified though mostly it was team approach. There were three themes that came out when asked whether the evaluator had sufficient skills in evaluation and whether they were able to be objective.

1. evaluation skills were lacking in the team
2. external pair of eyes may have been beneficial
3. hard to be truly objective 

This leads to findings from a later question that more support resources around evaluation are needed.

Question 1 - did you evaluate

I first looked at the question B1 - If no evaluation was carried out, why not?  From the 15 interviews, only 5 did project evaluation (or said/thought they did). From the 10 nos, I would say four of those did product evaluation in some format and one did some informal evaluation as part of an action research project which by its nature has reflection and redesign built in.
The five who said yes they did evaluation, three did evaluation as part of an action research approach and two did product evaluation.
To summarise:

Key:

Prod/Prog – evaluated a product or program which was the outcome of the project

AR/DE – used an action research approach so did a kind of developmental evaluation

	Did you evaluate?	type
1	No
2	No
3	No
4	Yes	DE
5	Yes	prog
6	Yes	AR
7	Yes	PAR
8	No	prod
9	No	prog
10	Yes	prog
11	No
12	No
13	No	Prod
14	No	Prog
15	No

There were three main themes that came through in the answers:

1. Different concepts of what is meant by evaluation – confusion over product, process and outcomes. Not a universal understanding of what evaluation is
2. Action research and developmental evaluation – informally evaluating whilst not actually writing formal reports etc
3. Lack of money and time to do evaluation

Trawling the Data

I've been playing with Leximancer for a few weeks now. To begin, I just loaded all of the transcripts into it to get a very complex themes and concept map. There was nothing strikingly interesting from that because there was lots of irrelevant information (ie included all of the questions too).

So now I am trying to extract the answers to each of the questions and anlyse them separately. This is proving really difficult because I was not consistent in my questioning. I often let the interviewee ramble off topic and did not get all of the questions answered adequately.

However going back through the data time and time again is good as it gives me a feel for what is being said. I'm not confident how to use Leximancer well, so i have supplemented this with two things. Firstly i have uploaded the transcript into Wordle to see what words come about. And secondly i have gone through the summaries i made of the interviews and looked to see what my themes were. By triangulating this evidence i hope to come up with something relevant.


The real learning in this is for me though. My questioning technique was really not consistent and next time i will read all questions even if i am repeating and say them clearly and not try and answer for the interviewee!