Evaluative Thinking

Today I read an article given to me by my supervisor and found that it rang more than a few chords with me, both in terms of my research and my work.

Buckley, Archibald, Hargreaves & Trochim (2015). Defining and Teaching Evaluative Thinking: Insights From research on Critical Thinking. American Journal of Evaluation 36(3), 375-388. 

The authors begin by reminding us that links between evaluative thinking (ET) and critical thinking are not entirely new. Scriven has long insisted on the use of critical thinking in evaluation and the authors take the stance of offering ways in this paper of incorporating ET more intentionally in all aspects of evaluative work. They begin by reviewing current definitions of ET and providing their own succinct version.

The literature on ET reached a peak in 2013 which interestingly aligns with my own search of the literature (in 2013) where one of the themes to come from my review was evaluation capacity building.

The authors explain how ET is often compared to reflective practice, "questioning reflecting, learning and modifying.."(p377). I liken this to Wadsworth's MERI (monitor, evaluate, reflect, implement). They talk about systematic inquiry (Preskill & Boyle, 2008), an analytical way of thinking that infuses everything that goes on (Patton, 2005), skeptical questioning (Weiss, 1998), a combination of commitment and expertise comprised of evaluative know-how and an evaluative attitude (Davidson, Howe & Scriven, 2004).

They believe that the lack of meaningful discussion around ET and that is is not researched, measured, promoted or taught is due to the absence of a widely agreed upon definition (p.377). In order to build their own definition they have borrowed from the critical thinking literature.

Evaluative thinking is critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action.(p378)

 They make an interesting statement about not everyone needing to be an evaluator or do evaluative work on a program or team but if they employ evaluative thinking they will have a better chance of success.

Next the authors describe constructivism and how knowledge and understanding is built on what we know and believe. Then onto levels of thinking and that the highest order (Blooms) is evaluative thinking or evaluativist-level thinking. Humans are not born with an ability to engage in such high level critical thinking - it needs to be cultivated or acquired. Similar to walking and sprinting. One we do automatically, the other required training and not everyone reaches the top. (Analogy by David Perkins in Hunkins, 1995). So ET needs to be learned and practiced.

Another interesting point brought to light by the authors is that of "belief preservation" whereby the brain has a natural tendency to disbelieve evidence if it goes against an existing belief (p.380). Metacognition is required (another high-level thinking skill) to overcome this deficiency (Lord, Ross & Leper, 1979).

So the desire to practice ET needs to be intentional and done regularly as with any other thinking-skill, if learning is to occur.

The 5 guiding principles are aimed at ECB practitioners and other 'promoters' such as teachers and facilitators.

1. be opportunistic about engaging learners in ET processes in a way that can maximise intrinsic motivation
2. incorporate incremental experiences by using 'scaffolding'.
3. offer opportunities to intentionally practice ET
4. work to overcome assumptions and belief preservation
5. apply and practice ET in multiple contexts and alongside peers and colleagues.

The authors discuss these practical strategies for promoting ET but the acknowledge that time and resources can be daunting - this aligns with my findings about conducting Evaluation in small projects (Huber & Harvey, 2013).

In terms of changing an organisation's culture one needs to encourage all members to become ETs. But methods and strategies to teach and learn ET have the potential to improve the quality of an organisation through evaluation.

Brown's 8 questions

Planning the next paper (4):

1. Who are the intended readers: grant bodies/grant application reviewers/ grant applicants
2. What did you do: Three recipients of locally funded learning and teaching project grants were 'followed' as they conducted their research projects over 12-18 months. There were three interviews with the project managers (beginning middle and end of project). And one interview with the project leaders (the person who applied for the grant and had the initial idea). Other project documentation was also used as evidence / data and project meetings were attended were relevant. The questions which guided the data collection were developed from the literature.
3. Why did you do it: To collect evidence as it was being generated rather than retrospectively when people have to 'recall' information which may be influenced by their perception of evaluation and other unknown factors. To see how perceptions changed over the course of the project and what influencing factors caused these changes.
4. what happened: all three had different experiences and perceptions of evaluation to begin with. This appeared to dictate how they conducted the evaluations and what they evaluated and what they did with any evaluation data.
5. What do the results mean in theory: don't know at this stage....
6. What do the results mean in practice: production of guidelines for project evaluation for/from the grant funding body. Development of a framework to guide project evaluation could arise from the findings.
7. What is the key benefit for readers: the grant bodies (ie application reviewers) could use the findings to guide their application and review processes. Academics and project leaders could use this framework to support them in their learning through the project and enable the institution to benefit more widely from the projects. 
8. What remains unsolved: How this framework could best be implemented, feedback from use of the framework.

Evaluation Capacity Building

Preskill, H., & Boyle, S. (2008). A Multidisciplinary Model of Evaluation Capacity Building. American Journal of Evaluation 29(4), 443–459.

Evaluation Capacity Building (ECB) has become a hot topic of conversation, activity and study in recent times. This paper offers a comprehensive model for designing and implementing ECB activities and processes. The model draws on the fields of evaluation, organisational learning and change and adult learning.

Trigers for ECB usually come from external demands (accountability, environmental change, policies and plans) or internal needs (organisational change, mandate from leadership, perceived lack of knowledge and skills, increased funding, perceived shortage of evaluators, desire to improve programs).

Assumptions are also required and may include: '(a) organization members can learn how to design and conduct evaluations, (b) making learning intentional enhances learning from and about evaluation, and (c) if organization members think evaluatively, their programs will be more effective' (p.446).

And expectations of any ECB effort may include:

• Evaluations will occur more frequently 
• Evaluation findings will be used more often for a variety of purposes (including program improvement, resource allocations, development of policies and procedures, current and future programming, and accountability demonstrations)
• Funders will be more likely to provide new, continuing, and/or increased resources
• The organization will be able to adapt to changing conditions more effectively
• Leaders will be able to make more timely and effective decisions
• The organization will increase its capacity for learning

The paper looks at each of the 10 teaching and learning strategies (from the inner left circle), but then goes on to stress the importance of the design and implementation of any initiative. With Design, the following are of importance: 

• identifying ECB participants characteristics - need to assess the evaluation competence of potential participants
• determining available organisational resources - including facilitation and time
• relevant evaluation, learning and individual and organisational change theories
• ECB objectives - cognitive, behavioural and affective

** some good references here on improving attitudes towards evaluation and reducing stress and anxiety around evaluation***

There is a section on transfer of learning and acknowledgement that dialogue, reflection and articulating clear expectations for what and how to transfer knowledge and skills are critical for longer term impacts of ECB (p453).

In terms of sustainable practice, the right circle of the model is described in more detail, with each of the 8 elements discussed. And finally the diffusion element is explored. The authors have used a water or reverberations to depict emanation from the organisation.

They conclude that for ECB to be transformational, efforts must be intentional, systematic and sustainable. (p. 457)

An evaluation framework for sustaining the impact of educational development

Hashimoto, Kazuaki, Hitendra Pillay, and Peter Hudson. “An Evaluation Framework for Sustaining the Impact of Educational Development.” Studies In Educational Evaluation 36, no. 3 (2010) 101–110.

The context of this paper is international aid agencies funding of educational development projects in recipient countries and their apparent ineffectiveness. The authors were interested in overcoming donor agencies internal compliance requirements by looking how local evaluation capacity could be developed and also how developments could continue to be sustained after project completion. Although this context is not applicable to the HE sphere, the same could be said of external funding agents vs internal projects.

The authors define process evaluation (quote: DAC Network on Development Evaluation. (2008). Evaluating development cooperation. OECD DAC Network on Development Evaluation. Retrieved January 6, 2010, from http://www.oecd.org/dataoecd/3/56/41069878.pdf.) And state the importance of process evaluation being the involvement of the participants in making decisions on a project such as terminating a project if necessary (p.102.). The authors quote Miyoshi and Stemming (2008) in that most studies on evaluation with participatory approaches are not underpinned by evaluation theories but are method-oriented.

So an Egyptian project was used as a case study (see previous post) and there were two research questions: (1) how can an entire educational development project be evaluated? and (2) how can the capacity development in educational reform be evaluated? Participants included six different groups of stakeholders: funding body, local admin, researchers, teachers, parents and students. The analytic technique used was pattern matching (Yin, 2003, p. 116) to enhance its internal validity. There were three emergent themes to the study, context, outcome and process evaluation.

Outcome evaluation:

• assessing outcomes is necessary for determining the success of an educational reform project.
• Outcome evaluation should include local participant involve- ment for evaluating a project since they are the end users.
• Local stakeholders should not be seen as informants or discussants but rather as evaluators working jointly with aid agencies so they can appreciate the success and failure of achieving the objectives.
• Results supported the use of an external evaluator who in collaboration with the internal evaluators of the project can undertake a macro level evaluation of the project

Context Evaluation

• context evaluation assesses local needs and problems to be addressed, cultural, political and financial issues, assists to design a project and sets objectives before the initiation of an educational project. 
• more local stakeholders such as representatives of local community are needed to join the evaluation to make their voice heard because after all they are the beneficiaries. 
• This engagement of various stakeholders in dialogues throughout the project from the project design phase may enable their opinions and interests to be considered for designing and implementing a more effective project (House & Howe, 2000).

Process evaluation

• There was a need for adopting a systematic participatory evaluation approach involving individuals and groups at the different levels of an educational system, which was central to process evaluation. 
• the linchpin of a sound process evaluation is employing skilled people
• the practice and culture of process evaluation should be nurtured during the life of educational projects and be institutionalized locally. This has the potential to sustain the impact of projects. 

Conclusion - conventional monitoring and evaluation practices do not have the ability to sustain a project beyond its lifetime. And that 'paradigms should shift from outcome-focused evaluation currently dominated by international donor agencies to process evaluation conducted largely by local participants but also supported by donor agencies. ' (p109)

Framework outlined in picture below (from p.108):

Other articles that follows this line of thinking:
 Donnelly, John. Maximising participation in international community-level project evaluation: a strength-based approach. [online]. Evaluation Journal of Australasia, v.10, no.2, 2010: 43-50. Availability:<http://search.informit.com.au/fullText;dn=201104982;res=APAFT> ISSN: 1035-719X. [cited 01 Oct 12].

Challenging times for evaluation of international development assistance

M Nagao - Evaluation Journal of Australasia, 2006 - aes.asn.au

Bazeley 2009

This article is written about the analysis of qualitative data and in specific about the use of themes.

Short introduction on the difference between categories concepts and themes (often used interchangeably). This author uses category for the descriptive and concept for the more abstract. Other authors use concept as the lowest level and category for a group of concepts (Strauss & Corbin 1998).

The author states that producing themes as a goal of research is not much use or interest. Just describing them and using some quotes from the literature to support them is not enough to be convincing.

Suggestions made to share some portions of data with a colleague to get a fresh perspective and alternative avenues to pursue. The author writes that if describing themes is what you are doing then you need to connect them to the literature and contextualise them.'Data must be challenged, extended, supported and linked to reveal their full value' p.8
Are emergent themes really emergent? If you asked questions which produced these themes then the findings are shallow and unsubstantiated. Talks about the 'garden path analysis' (Lyn Richards) just stating what you see. Move towards a Describe-Compare-Relate approach instead.
As a starting point, describe: How did people talk about this theme, how many, what's not included.  [NOTE: in my case could look at the sentiment lens in Leximancer possibly?]
Then compare differences in characteristics and boundaries for that theme. [in my study could compare different ability levels, academic levels, experience of project management levels internal vs external etc]
Then relate to themes others have already written about.

A section on creating and using displays of the data extolls the value of for developing an understanding and presenting conclusions.
Matrix displays - for detecting patterns - for facilitating comparative analysis and sometime for presenting conclusions
Flow charts and models - present conclusions
Typologies - used a a working tool and can become a final presentation tool

And finally - avoid reliance on Quotes for Evidence as this encourages superficial reporting of themes. Try not to write to the sources, voices or methods. Build a coherent argument using evidence and then '...add illustrative quotes to add interest and clarity for the reader'.(p.20)

a meta-evaluation of a different kind

Means, Toyama, Murphy, Bakia, Jones (2010) Evaluation of Evidence-Based Pratices in online Learning: A Meta-Analysis and review of Online Learning Studies

I’m interested in this article for the methodology of how they went about their meta-evaluation (although I have to say the content is really interesting too and leads itself to the FLaMe program).

They specify the type of studies they were evaluating, these studies must have stringent designs (random-assignment or controlled quasi-experimental) and examine the effects for objective measures (of student learning). In relation to my project, I have not really specified the sample such as this and have used a sample of convenience – will this hold up to rigorous testing?

Furthermore I am not looking at quantitative studies. Evaluations tend to be more qualitative in their nature.

there was a main finding from the literature review - few rigorous studies in the area of interest had been published. In fact non had been found in the years of search and so the search was expanded by 2 years.

There were key findings from the meta-analysis and then there were findings from the narrative review. This review came from analysing the studies that were not able to be included in the meta-analysis due to the fact that they did not have a particular control condition. (p.xii)

Interesting comments on the potential for bias stemming from studies' authors dual roles as experimenters and instructors

The lit review and Meta-analysis were guided by four research questions. Context for the meta-analysis was given, explaining who commissioned the study (stakeholders) and the overall goal of the study. Then they described a conceptual framework for the topic [online learning] which included three key components.

Methodology

First define the topic [online learning], state what is included and what is not. Use this to define the categories to be searched for (three in this study).
List data sources and search strategies. Give years from and to and mention dates of different say for thesis etc.
List the databases that were used and the list of keywords (in the appendix).
Additional search activities - included a review of articles cited in recent meta-analyses and narrative syntheses of similar topics.
Key journals were identified and abstracts from each of these in a given period were manually reviewed.
A Google Scholar search engine was used with a series of keywords and any article abstracts retrieved were reviewed to ensure no duplication from pervious searches.

Scrrening was then carried out in 2 stages. Firstly abstracts were reviewed, giving studies the 'benefit of the doubt' (p.11) ensuring they met the inclusion criteria. Statement of number of papers included and number rejected (as percentages) with reasons for rejection.

Full text screen - the next pass had two stages, studies had to meet content relevance criteria (listed) as well as basic Quality (method) criteria (also listed.). A table of the primary reasons for exclusion was included detailing numbers for each reason and percentages. (p.13)

The coding of the study features was then detailed, regarding study features and study quality, then a paragraph explaining how the interrater reliability was checked.  Finally data analysis was explained, but this is completely statistical since the chosen studies were all quantitative, and a meta-analysis software was used for computations.