Effective Dissemination

Deborah Southwell, Deanne Gannaway, Janice Orrell, Denise Chalmers & Catherine Abraham (2010): Strategies for effective dissemination of the outcomes of teaching and learning projects, Journal of Higher Education Policy and Management, 32:1, 55-67http://dx.doi.org/10.1080/13600800903440550

Came across this article and found some useful information in it, and its related ALTC (2005) report with same name. The report has been reviewed in 2011 and is now in the OLT repository, here. 

Summary:

The paper looks at a range of Australian and International funding bodies who support L&T projects and asks the question 'How can a national institute for Learning and teaching in higher education maximise the likelihood of achieving large-scale change in teaching and learning across the Australian higher education sector, especially thorough its grants program and resource repository and clearing house?'(p.58)

Now whilst this covers national schemes, there are some important findings that could be applied to smaller scale projects and schemes such as the one I'm looking at and there are also some interesting findings on evaluation. Furthermore the ALTC report identifies some other international funding bodies which i could follow up on in terms of evaluation requirements etc.

Questions from the guiding framework of the project team are included and a couple would be relevant to my study:

What are the influences on and factors affecting a teacher's decision to make use of a project pr process that is being scaled up? What local and external factors facilitate or create barriers affecting the teachers decisions? What is the relationship between the development of local capacity and the quality of external reform structures? (p.59)

The projects which were selected came from a range of locations ut focussed more on projects that sought to change teaching and learning processes, practices and beliefs rather than on projects that focussed on developing products. (p60.)

Some Findings:

Two items of note were:

1. Initiation of an innovation - Intended users of an innovation need to be engaged very early int he planning stages of the innovation in an endeavour to ensure adoption and take up of ideas later on.(p.61) This can be extrapolated to my project in terms of getting stakeholders involved from the early stages and to getting uptake of evaluation findings.

2. Implementation, embedding and upscaling of innovation - one influence on this comes from the personal conception of teaching (p.62). I'm suggesting that the same is true for evaluation. Also, Academics with teaching qualifications appear to be more open to investigation of alternate curriculum and teaching approaches. (Lueddeke, 2003). It would be interesting to test and see if this is the case with evaluation ie does the qualified teacher value evaluation mechanisms more and therefore employ them in the projects in comparison with academics without teaching quals. and if this is the case is that because they have been 'taught' evaluation methods and skills and therefore their conception is different?

Conditions for successful dissemination:

1. effective, multi-level leadership & management
2. climate of readiness for change
3. availability of resources
4. comprehensive systems in institutions and funding bodies
5. funding design

This last point is of interest - it mentions that expectations about the approaches to projects and activities that ought to be adopted are taken from the funding design. so i could say that if there is no  real expectation of evaluation requirements then of course the project applicants are not going to be that stringent. - can link to this in the analysis of phase one.

One final mention in the conclusion (p.65) says 'An important aspect of this study was to identify abd to recommend that learning and teaching grant recipients must be supported and provided with access to experts in educational innovation and evaluation.'

The 2005 Carrick report goes into much more detail and evaluation is mentioned throughout. It also gives recommended strategies for each of the 5 'conditions', at the national, institution and discipline level. when i read through look like they were all implemented at MQ  ie recommending standard requirements for application, including description of which evaluation strategies will be used. 

FOr item 3, the report recommends providing project teams with access to specialist expertise - this could be for evaluation or at the least evaluation resources. Findings on p.55 state that 'those responsible for the project may require assistance in designing an appropriate evaluation process'.also info on this on page 45 (emerging themes)

For item 4 it mentions that support for quality processes, particularly monitoring and evaluation ought to be supplied. Also that evaluation is reported within an evaluation framework. Also, on p.58 there are findings that state that institutions that allocated funders AFTER the projects were finished were evaluated well and regularly and were eventually embedded within an institution. 'Generally, however, experiences quoted int he literature and in case studies evidenced poor quality of evaluation if done at all.' It then went on to explain that frequently dissemination across institutions occurred before it was apparent that there was any value or improvement instudent learning and therefore impact analysis was vital.

The 2011 review of this project didn't seem to add much more in regards to Evaluation other than a recommendation that external evaluation reports be provided publicly. There was a reference to Dow (2008) which i should follow up and could provide some useful data.
An evaluation of the Australia Learning and Teaching Council 2005-2008. Sydney: Australian Learning and Teaching Council.

What can be evaluated?

The following excerpt is taken from: Johannessen, E.M. (2001). Guidelines for evaluation of education projects in emergency situations. The Norwegian Refugee Council. http://toolkit.ineesite.org/toolkit/INEEcms/uploads/1039/Guidelines_for_Evaluation_of_Educ_Projects.PDF

5 WHAT CAN BE EVALUATED?
It is common in many evaluations to focus on the OECD criteria: relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability. Relevance refers to whether the project addresses the needs of the beneficiaries, efficiency to how productively the resources have been used while effectiveness relates to whether the objectives of the project were achieved. Impact is the long-term effect  while sustainability relates to the maintenance of the changes after the project or program has been terminated (Dale 1998, Lewin 1993, Samset 1993).   Sida adds "lessons learned" to the criteria which refer to important general experience the evaluation has yielded. All or some of the criteria are commonly addressed in evaluations of education projects although the terms may be used differently.
Although the concepts seem relevant and reasonable, there are several problems associated
with them. First of all, they need to be defined and specified in terms of identifiable activities when the project is being planned. This is often not the case. Secondly, the concepts "relevance", "impact" and "sustainability" may be difficult to investigate. The project may have been going on for a too short period and/or there is not enough time available to find answers to this type of questions.  Another objection is that the OECD concepts are related to final products and not to description of the processes that have taken place. No projects/programs, whether in research or evaluations, develop neatly in a linear way. The description of the project and the steps we plan in the beginning are always transformed as they are being implemented. Major decisions regarding adjustments and turning points should be described and justified and be part of the evaluation.

I think it will be useful in the next paper when I'm analysing the themes and relating them back to the literature.

The following excerpt is from p12. The author uses the terminology 'Terms of Reference' (ToR) to describe to evaluation plan. This is in line with other articles and will fit into the paper on lit review/gaps in the lit.

Often the terms of reference are too vast and ambitious taken the time and resources into
consideration. Sometimes questions and topics are suggested that are not possible to answer, at least not within the limited time available. As a result the evaluators know already before they start working that it is not feasible to follow the ToR

And again on p18:

8 WHEN TO EVALUATE
Ideally the planning of an evaluation should start parallel with the activities/project. But more
often the opposite is the case; the coordinators start too think about evaluation when the
project  is concluded or about to finish.

More thoughts on phase 1 interviews

I've been reading through each of the interviews and marking up the themes that I have and I've come across another theme that is partially related to evaluation but also related to grants and projects.

It seems that more than a few people I interviewed complained about the fact that even though their project created or produced some wonderful outputs, not everyone was willing to take up these new ideas.  For example redesign of some units into online units with chunking up of content meant that there were now more opportunities to interact with the material but the students felt this was too much work and the tutors felt this was too much work (marking). This kind of leads then to the point that if you don't fully involve your stakeholders then the project outcomes can't really meet their needs.

On the other hand another participant talked about this and said that there has to be a balance because when you try to please all of the stakeholders in this way you end up with a substandard product ie you 'dumb it down' to keep everyone happy and some of those novel and 'out there' ideas are lost in translation. This participant said that often it is a case of good timing. If no one takes up your new idea/product as you intended then it may because they are just not ready yet.

That then in turn leads to the conclusion i keep coming back to that evaluation has to be done later down the track (as well as formative and summative in the terms of the project). Often it is too early to say whether this project has been successful or not. You can say whether you met your outcomes or not but you cannot say whether those outcomes have had impact. Neither do people report what didn't work or rather why they may not be being taken up.

I think perhaps also there is some confusion between steering groups and stakeholders. A steering group may well reign you in, but stakeholders will be sure to tell you what they want and why/how they want it. Its one thing to listen to their needs then made an informed decision on how you are going to design your product (say). It is another to present it to a steering group because then you pretty much have to make the changes they request.

Analysis of Phase One

The research questions for this phase are:

1. What evaluation forms and approaches have been used in Macquarie funded learning and teaching projects? [easy to answer from the interview data.]
2. What are the issues and challenges in evaluating learning and teaching projects?
   
   [This was the original Q11 in the interviews. Briefly, items would include:
   lack of skills - guidelines and support/resources needed
   initial plans too ambitious
   insufficient use of Stakeholders
   insufficient money/budget - to pay for extra help or input when needed i.e. admin support
   no feedback at any stage in the project
   lack of time
   • to plan
   • for reflection/learning
   • too busy with teaching and other demands]
3. What is understood by evaluation? [can look at the misuse or confusion in terminology.
   For example Evaluation vs. Research and Evaluation vs. Project in terms of both planning and results. There is also some misinterpretation between evaluation and feedback. Also look at Q12 from the interviews.]
4. How does perception influence how evaluation is carried out in practice? Or, What influences how evaluation is carried out in practice....... [for this i think i need to go through the data and pull out all examples where perception is discussed, albeit implicitly most times. Also need to look for other papers that have done this and see how they have done it. Need to refer back to the theoretical approach of the project as well as the realism paradigm]

Phase 2

There are so many competing actions needed right now I can't seem to move forward at all.
So this is my attempt to list what needs to be done and so form a plan of action.

1. write paper 1 - lit review identifying gaps and hence need for this project
2. thematic analysis of data from phase 1 - write  a paper on findings - need to adress the research questions identified in proposal
3. use findings from analysis to inform the development of an evaluation framework to be tested in phase 2
4. identify the projects to be used as case study in phase 2
5. write a plan of action for the case study approach - something to give to project leaders
6. have initial meetings with project leaders of two projects. Get ethics info and consent forms ready.

Action plan - phase 2

"The investigator will act in the role of Participant-as-Observer (Gold, 1958) of the evaluation and project process(es), actively participating with the project members and documentation and providing a depth to the research which would not be possible with an observer-only role (Babchuk, 1962)." (proposal, June, 2012)

1. Meet with the project team. Show the list of questions which will be used as part of the data gathering instrument. Answer any of their questions.
2. First 'interview' for follow up on answers. [need to develop some more questions but may be those from the 'notes' column in the table below.]
3. Explain that I will attend all of their project meetings (where possible) and take notes which i will use in my reflections. Act as participant-as-observer.
4. Then I will meet two more times for 'interview', once after the progress report is due and again at the end of the project - perhaps after the final report is submitted. Each time, there will be a set of questions to be answered and then I will follow up on these in more detail in the interviews.

So in total there will be three interview stages but perhaps a number of people will be interviewed.

Questions for first interview - Phase 2. (Time required – approx. 1hr)

Area	Questions	Notes
1. project clarification	What is the nature of the project? What is the focus of the project? What is the scope of the project? What are the intended outcomes? What (if any) are the project outputs? What are the operational processes developed to achieve the outcomes? What is the conceptual and theoretical framework underpinning the project? What is the context of the project? Are their any identified risks? What key values drive the project?	Are there sufficient resources/admin for this project? Are the plans too ambitious? What may happen to delay the project? Has teaching time been factored in?
2. Evaluation purpose and scope	What are you evaluating? Why is the evaluation being done? Are you basing the evaluation on any particular method framework or approach? How will the information be used? Who will evaluate this project? Are they suitably skilled? What value will the evaluation process add to the project?	Do they need training? Do they need some support resources?
3. Project Stakeholders and study audiences	Who are the stakeholders for the project and the audiences for the evaluation information? Stakeholders- Who has an interest or stake in the project and/or its outcomes, and in the evaluation of the project? Audiences - Who will be interested in the results of the study and what types of information do they expect from the evaluation? How should competing interests be prioritised?	Have you asked for feedback on the project? Use these to your advantage – to help guide the project. Be clear to understand the difference between two groups.
4. Key evaluation questions	What are the KEQ? Some examples could be: What processes were planned and what were actually put in place for the project? Were there any variations from the processes that were initially proposed, and if so, why? How might the project be improved? What were the observable short-term outcomes? To what extent have the intended outcomes been achieved? Were there any unintended outcomes? What factors helped and hindered in the achievement of the outcomes? What measures, if any, have been put in place to promote sustainability of the project’s focus and outcomes? What lessons have been learned from this project and how might these be of assistance to other institutions?
5. Data Collection Methods	How will the information be collected and analysed? What/who are the data sources? What types of data are most appropriate? What are the most appropriate methods of data collection? How will the data be analysed and presented in order to address the key evaluation questions? What ethical issues are involved in the evaluation and how will they be addressed?
6. Dissemination of Findings	How will the evaluation findings be disseminated? Who are the audiences for reports on the evaluation and what are their particular needs and interests? What are the functions of reporting? What reporting strategies will be used? When will reporting take place? What kinds of information will be included in evaluation reports?	Are the stakeholders involved in dissemination plans?
7. Evaluation Plan	What does your evaluation time line and activity schedule look like? What measures do you have in place to ensure you don’t run out of time for the evaluation to take place as planned? Who will you ask to review your evaluation plan?	Has time for reflection been built in to the plan?

Thematic Analysis

Braun and Clarke (2006), offer a complete and in-depth paper on what it is, guidelines to using it and pitfalls to avoid. They state that it can be considered a method in its own right - contrary to other authors stating it is not. It is compatible with constructionist paradigms and they stress its flexible nature in its use which can sometimes cause it to be framed by a realist/experimental method (though they don't particularly go along with this).
Importance is placed on explaining 'how' analysis is conducted (as its often omitted) as well as describing what and why they are dong it that way. Terminology is defined - Data corpus, set, item and extract. [all interviews, answers or themes, one interview and quotes].
" Thematic analysis is a method for identifying, analysing and reporting patterns (themes) within data." p.79
we shouldn't say that themes 'emerge' because actually that implies that they reside in the data - in actual fact they reside in the heads of the researcher who plays an active role in identifying, selecting and reporting them.(Ely et al., 1997 and Taylor & Ussher, 2001)
How do we say a theme exists? There are no hard and fast rules - though prevalence is important. A theme could be prevalent in one data item or across the whole corpus. And it may (or may not) be present in every data set or it may be present to only a small extent.
You should think about how a theme captures something of importance to the overall research question(s). This may make it 'Key'. The the question lies in how to report it ie 'most participants' or 'some..' or 'a number of...' etc.
You need to ask yourself whether you want to provide a rich thematic description of the whole data set or do you want to provide a detailed account of just one aspect of the data.
Next, will you provide an inductive analysis - whereby you link the themes to the data ie from specific questions, or a theoretical analysis - whereby your research questions evolve from the themes.
Semantic vs latent themes ie surface level descriptive or more deeper analysis of the underlying causes, assumptions and conceptualizations - this leads towards a more constructivist approach
Back to the paradigm wars - if a constructivist paradigm is used, then the themes are built from sociocultural contexts which enable individual accounts. In comparison the realism framework allows a more simple explanation to develop since meaning, experience and language are unidirectional. ie the language is used to describe experience and provide meaning.

The paper then goes on to describe 6 steps in thematic analysis - to be used flexibly and as a guide.
1. Familiarize yourself with your data: jot down notes for coding schemas that you will come back to in subsequent phases.
2. Generate initial codes. Work systematically through the data set  and identify interesting aspects in the data items that may form the basis of repeated patterns.
3. Search for themes: sort the potential codes into themes - broader analysis of whole data set. Use concept mapping
4. Review themes. this is a two step process - level 1 consists of looking at the extracts for each theme and deciding if they really fit that theme, and are coherent. if not, reassess the theme and perhaps discard extracts if necessary. Level 2 of this stage will depend on your theoretical approach and requires revisiting the whole data set and consider the validity of each of the themes.
5. Define and name the themes. Look at the data extracts for each theme and organise into a coherent account with an accompanying narrative. Look for sub-themes within a theme and finally, use clear (short) names for the themes so that the reader understands quickly what it means.
6. Produce the report. More than just describe the data, tell the story by using the extracts to make an argument in relation to the research questions.

Some questions to ask towards the end fo the analysis:
what does this theme mean?
What are the assumptions underpinning it?
what are the implications of this theme?
what conditions are likely to have given rise to it?
why do people talk about this thing in this particular way?
what is the overall story the different themes reveal about the topic?

Potential pitfalls are described:
1. no real analysis is done and the analysis is just a sting of extracts with little analytic narrative.
2. uses the data collection questions as the themes.
3. no coherence around an idea or concept in all aspects of the theme.
4. no consideration of alternative explanations or variations of the data
5. mismatch between the interpretation of the data and the theoretical framework.

evidence, gaps in the evaluation lit

"Given the poor quality of evaluation performance in education, and the lack of a research base to guide evaluators, it seems urgent to contrive ways of defining, assuring, and documenting the quality of evaluation work." Stufflebeam, 2011.