Visiting UCLA

Well I never thought the day would arrive but finally I found myself on the UCLA campus visiting the Chair of Education at UCLA, Professor Christina Christie along with Professor Marvin Alkin, the renowned Evaluation Scholar.

We had just a short time to meet and there was no real agenda but my aim was to present the findings of my PhD and get some feedback from them.

Of course the time flew and I only achieved half of what I had aimed for but nevertheless some learning was certainly had! Marv's brain was as sharp as a razor and he asked me many questions including lots about the PhD experience in Australia. There is a a marked difference between our two countries in that in the USA you take 2 years of classes in your chosen discipline before beginning the Doctoral journey. They then spend a year 'qualifying' (not exactly sure what that means) and then they defend their research proposal and only then can they start the research. So they basically spend 6-7 years full-time doing their PhD whereas I have spent the same amount of time but as a part-time student.

Anyway here are some of the take home messages - or rather questions for further reflection from this visit:

1. Is there really a difference between a project and a program? 

Marv was quite insistent that the two were interchangeable. I actually disagree, the difference may be minor but I think my understanding of the context in which i was writing and researching helps me define it.

A project can be large or small, funded or unfunded and in this project, there is an aim to change something (in my case improve teaching by introducing an innovation, be it technological or methodological). The project plans how the change will occur, implements the change and observes what happens to the output (in my case student learning or student experience). The evaluation of the project can simply observe any change in outcome but could (and should) formatively evaluate the process and reflect on learning that takes place for both the teacher and the student.

A program can also be large or small, funded or unfunded (though most often, its the former of both options). However the aim of a program is usually to provide a service which will result in an outcome - usually social betterment of the participants of the program. The evaluation of the program often aims to judge whether the program has been successful or not, sometimes with the aim of continuing (or not) the program funding, but sometimes to recommend changes in how the program could be run better.

So as you see the two items and their evaluation are very similar and the terms are often used interchangeably. In my write up of the thesis I need to revisit my definitions and perhaps clarify these nuances to make it clear to the examiners. Having said that though, as I wrote this I actually struggled to clarify the differences - more work needed here!

2. Improvement Science

As I explained these nuanced differences to Tina and Marv, they looked knowingly at each other and said "you should read this". They passed over a copy of the latest issue of the Journal - New Directions for Evaluation. This special edition introduces the field of Improvement Science and discusses the overlap and differences between it and evaluation.

In a nutshell, improvement science is another word for formative evaluation leading to incremental change. There is a wealth of literature about this topic so my summary is just the tip of the iceberg, however I really like this idea because using this terminology could help overcome the misconceptions many people have about evaluation. I think the use of this term would certainly appeal to the science, engineering and IT community.

I will incorporate this and info from the journal special issue articles into my discussion chapter as one of the main topics I discuss from my findings is the learning dimension of evaluation and this aligns perfectly with the term 'improvement science'. In the first article in the journal special issue, Christie discusses the similarities with Developmental Evaluation and Patton's response to how they actually differ. Again useful for my thesis as DE was an angle I discussed in the introductory chapter.

So they were the main two things I took from the visit. Both professors were interested (but not surprised) to hear my findings about misconceptions of evaluation and misalignment of praxis. They were also very interested in the online tool I developed though we ran out of time to get any real feedback on that. I have since sent them the link to the tool and asked them to share and comment if they had time.

There was one final thing I had hoped to get from the meeting and that was some suggestions of names of possible examiners, however we did not get to that. I have sent a follow up email but didn't hear back so will chalk that one up to experience and work with the list I currently have.

3. Taking a class

I was invited to attend a class called 'Procedural issues in evaluation'. This was taught by one of Tina's PostDoc Students (Jenn Ho) and combined about 9 graduate students, some of whom were doing Doctoral studies, others were just 'auditing' and others were doing this subject as an elective from other degree programs. I was able to briefly talk about myself and my research - which in itself was a learning process. Over the trip I had occasion to do this numerous times and it certainly got easier to condense my 7 years into a few sentences!

I was guided through this class to an evaluation resource pack provided by the Kellog Foundation (one of the largest philanthropical foundations in the USA offering grants to offer opportunities to children, families and communities to reach their full potential). This resources has some excellent information (pp.6-16) regarding challenging assumptions and recommendations for good evaluation practices. Again, I'll be able to refer to some of this information in my thesis, possibly in the discussion section and maybe even in the introduction.

Another resource/website I heard much about through this class was the Annie E. Casey foundation and particularly theory-of-change. This resource will also come in handy perhaps not for the thesis but certainly for future work as an evaluator (if that happened). 

The aim of the class was to discuss similarities and differences of using a Logic Model and a TOC approach in an evaluation. It was great to be a student again and actually learn from doing rather than just learn from reading.

After the class I spent an hour or so with Jenn discussing evaluation and PhDs and it was a great way to round of the visit to this famous campus.

Chapter 2

Just looking at Chapter 2, the lit. review.

Some feedback from Marina, try to use some more action words to stress the need for what i did and why.

I then came across a table summarising the articles i used in the lit review - this will be fabulous as an appendix - or perhaps even as part of the intro to chapter 2?

Whilst doing that I recalled and reread an article from Oliver, McBean, Conole & Harvey (2002). They had developed an online toolkit to help evaluators of projects get to grips with evaluation. On the premise that a one size does not fit all:

The toolkit involves six steps, derived from the literature and from research into evaluation practice:

1. Identification of the audience for the evaluation
   
2. Selection of an evaluation question
3. Choice of an evaluation methodology
4. Choice of data collection methods
5. Choice of data analysis methods
6. Selection of the most appropriate format(s) for reporting the findings to the audience 

(p.200-201).

This is eerily similar to mine. however their tool took 4.5 hours to complete and the authors agree that "Although it could be argued that this reflects the complex demands of evaluation, and thus is not unreasonable, it does mean that the toolkit is ill-suited to small, quick studies." (p.207).

Narrative

compiling some literature and words around the use of narrative

The Narrative Construction of Reality (Bruner, 1991):

"...we organise our experience and our memory of human happenings mainly in the form of narrative—stories, excuses, myths, reasons for doing and not doing..." (Bruner, 1991, p4). 

"Narratives, then, are a version of reality whose acceptability is governed by convention and "narrative necessity" rather than by empirical verification and logical requiredness..." (Bruner, 1991, p4). 

Bruner provides 10 features of a narrative:

1. Narrative diachronicity - an account of events occurring over time.
2. Particularity - context or a particular embodiment
3. Intentional state entailment - so no causality, just the basis for interpreation of what happens
4. Hermeneutic composability - interpretation through intention attribution and back- ground knowledge
5. Canonicity and breach - (not sure how this works...)

6. Referentiality
7?
8. Normativeness

9. Context sensitivity and negotiability

10. Narrative accrual

In this paper Bruner has described how reality is described through narrative principles, how thought is enunciated (through discourse). He concludes that his work has just begun and that he wants to show how narrative can organise 'the structure of human experience' (p.21).

The Value of Narrativity in the Representation of Reality (white, 1980):

"narrative might well be considered a solution to a problem of general human concern, namely, the problem of how to translate knowing into telling" (White, 1980, p.6)

So what is the difference between narrative and discourse? The latter is subjective and identifies an 'ego' whereas the former is objective and is just the logical progression of facts and information that join to tell the account.

"Benveniste shows that certain grammatical forms like the pronoun "I" (and its implicit reference "thou"), the pronominal "indicators" (certain demonstrative pronouns), the adverbial indicators (like "here," "now," "yesterday," "today," "tomorrow," etc.) and, at least in French, certain verb tenses like the present, the present perfect, and the future, find themselves limited to discourse, while narrative in the strictest sense is distinguished by the exclusive use of the third person and of such forms as the preterit and the pluperfect." p.7.

So, i think I am using discourse not narrative if I am using 'I'?

 Next I need to read the work of Albert Bandura? - Actually not much use - more about social cognative therapy...

This article may have something of interest:

Narrative Means to Preventative Ends: A Narrative Engagement Framework for Designing Prevention Interventions
Michelle Miller-Day  & Michael L. Hech

http://doi/10.1080/10410236.2012.762861

PGRF

Found out yesterday that I got my funding application accepted. It was a competitive process, very time consuming to get it right and a bit of a pain running around to get signatures, but worth it in the end. The funds will support me to go to the Canadian Evaluation Conference and present my thesis. But I also get to go on a side trip and visit UCLA. I wrote to Professor Christy and Prof. Alkin who are going to host me for 2 days in April to present my research and also meet with their research students. YAY!

But the best news was that I received a DVC (research) Commendation for my application (and an extra payment!)

Now I just have to write the conference abstract and get it accepted.... eek.

Evaluative Thinking

Today I read an article given to me by my supervisor and found that it rang more than a few chords with me, both in terms of my research and my work.

Buckley, Archibald, Hargreaves & Trochim (2015). Defining and Teaching Evaluative Thinking: Insights From research on Critical Thinking. American Journal of Evaluation 36(3), 375-388. 

The authors begin by reminding us that links between evaluative thinking (ET) and critical thinking are not entirely new. Scriven has long insisted on the use of critical thinking in evaluation and the authors take the stance of offering ways in this paper of incorporating ET more intentionally in all aspects of evaluative work. They begin by reviewing current definitions of ET and providing their own succinct version.

The literature on ET reached a peak in 2013 which interestingly aligns with my own search of the literature (in 2013) where one of the themes to come from my review was evaluation capacity building.

The authors explain how ET is often compared to reflective practice, "questioning reflecting, learning and modifying.."(p377). I liken this to Wadsworth's MERI (monitor, evaluate, reflect, implement). They talk about systematic inquiry (Preskill & Boyle, 2008), an analytical way of thinking that infuses everything that goes on (Patton, 2005), skeptical questioning (Weiss, 1998), a combination of commitment and expertise comprised of evaluative know-how and an evaluative attitude (Davidson, Howe & Scriven, 2004).

They believe that the lack of meaningful discussion around ET and that is is not researched, measured, promoted or taught is due to the absence of a widely agreed upon definition (p.377). In order to build their own definition they have borrowed from the critical thinking literature.

Evaluative thinking is critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action.(p378)

 They make an interesting statement about not everyone needing to be an evaluator or do evaluative work on a program or team but if they employ evaluative thinking they will have a better chance of success.

Next the authors describe constructivism and how knowledge and understanding is built on what we know and believe. Then onto levels of thinking and that the highest order (Blooms) is evaluative thinking or evaluativist-level thinking. Humans are not born with an ability to engage in such high level critical thinking - it needs to be cultivated or acquired. Similar to walking and sprinting. One we do automatically, the other required training and not everyone reaches the top. (Analogy by David Perkins in Hunkins, 1995). So ET needs to be learned and practiced.

Another interesting point brought to light by the authors is that of "belief preservation" whereby the brain has a natural tendency to disbelieve evidence if it goes against an existing belief (p.380). Metacognition is required (another high-level thinking skill) to overcome this deficiency (Lord, Ross & Leper, 1979).

So the desire to practice ET needs to be intentional and done regularly as with any other thinking-skill, if learning is to occur.

The 5 guiding principles are aimed at ECB practitioners and other 'promoters' such as teachers and facilitators.

1. be opportunistic about engaging learners in ET processes in a way that can maximise intrinsic motivation
2. incorporate incremental experiences by using 'scaffolding'.
3. offer opportunities to intentionally practice ET
4. work to overcome assumptions and belief preservation
5. apply and practice ET in multiple contexts and alongside peers and colleagues.

The authors discuss these practical strategies for promoting ET but the acknowledge that time and resources can be daunting - this aligns with my findings about conducting Evaluation in small projects (Huber & Harvey, 2013).

In terms of changing an organisation's culture one needs to encourage all members to become ETs. But methods and strategies to teach and learn ET have the potential to improve the quality of an organisation through evaluation.

Evaluation Planning Instrument - next steps

Presented the development of the tool at ISSOTL this week. Great feedback - comments from audeince indicated that they were waiting for this interactive and would like to see exmples included.

Interest from RMIT - could i go and present to seed grant holders (in Science) on what to do around project evaluation.
Interest from UBC - they need help with this and have some skills to exchange - need to follow up with them to find out if they mean programming skills!

Next steps:

• update the steps to include feedback from AES and focus group 2.
• harvest the examples collected from focus groups
• mind map how the online form could work - including branching
• plot out in excel using simple logic
• contact a programmer and think about what this may look like in an online version

2nd focus group for phase 3

After the first focus group I revisited the planning instrument and took on some of the feedback to rethink the content and structure.

The main concern from the focus group was that it was too lengthy for a small project. So some steps were combined and others removed. The resultant checklist has 6 steps:

1. What is the purpose and scope of the evaluation? Consider also how the information will be used. (i, ii, iii, iv, v, vi.)

2. Who are the stakeholders of the project and of the evaluation? Are they also part of the study audience? i, v, vi.

3.  What are the Key Evaluation Questions? i

4. What data and evidence will be collected and how will it be analysed? i, ii, v

5. What are the criteria for judgment? i, ii.

6. What dissemination strategies will be used and how will this help you? i, vi.

i.     Chesterton, P., & Cummings, R. (2011). OLT Grants Scheme - Evaluating Projects.key to literature

ii.    Owen, J. M. (2006). Program Evaluation. Forms and Approaches. Crows Nest: Allen and Unwin.

iii.   Patton, M. (1994). Developmental Evaluation. American Journal of Evaluation, 15(3), 311–319.

iv.   Scriven, M. (1991). Evaluation thesaurus. Newbury Park, Calif. Sage Publications.

v.     Saunders, M. (2000). Beginning an evaluation with RUFDATA: Theorising a practical approach to evaluation planning. Evaluation 6(1) 7-21.

vi.  Stufflebeam, D. L. (2011). Meta-Evaluation. Journal of MultiDisciplinary Evaluation, 7(15). 

Some key points to come from this focus group included:

• allow users to begin at any point as some felt step 1 (outcomes) was not their logical starting point.
• use the langauge of progress rather than achievement.
• add back in a question about recommendations for the future - for people who want to pick up where this project leaves off.
• an iterative approach to the steps would also be useful. Feedback loops. Once you start your project you usually revisit these steps and answers may change as you go.
•  collecting evidence - not only what but also how (many people forget ethics approval)