Evaluation Planning Instrument - next steps

Presented the development of the tool at ISSOTL this week. Great feedback - comments from audeince indicated that they were waiting for this interactive and would like to see exmples included.

Interest from RMIT - could i go and present to seed grant holders (in Science) on what to do around project evaluation.
Interest from UBC - they need help with this and have some skills to exchange - need to follow up with them to find out if they mean programming skills!

Next steps:

• update the steps to include feedback from AES and focus group 2.
• harvest the examples collected from focus groups
• mind map how the online form could work - including branching
• plot out in excel using simple logic
• contact a programmer and think about what this may look like in an online version

2nd focus group for phase 3

After the first focus group I revisited the planning instrument and took on some of the feedback to rethink the content and structure.

The main concern from the focus group was that it was too lengthy for a small project. So some steps were combined and others removed. The resultant checklist has 6 steps:

1. What is the purpose and scope of the evaluation? Consider also how the information will be used. (i, ii, iii, iv, v, vi.)

2. Who are the stakeholders of the project and of the evaluation? Are they also part of the study audience? i, v, vi.

3.  What are the Key Evaluation Questions? i

4. What data and evidence will be collected and how will it be analysed? i, ii, v

5. What are the criteria for judgment? i, ii.

6. What dissemination strategies will be used and how will this help you? i, vi.

i.     Chesterton, P., & Cummings, R. (2011). OLT Grants Scheme - Evaluating Projects.key to literature

ii.    Owen, J. M. (2006). Program Evaluation. Forms and Approaches. Crows Nest: Allen and Unwin.

iii.   Patton, M. (1994). Developmental Evaluation. American Journal of Evaluation, 15(3), 311–319.

iv.   Scriven, M. (1991). Evaluation thesaurus. Newbury Park, Calif. Sage Publications.

v.     Saunders, M. (2000). Beginning an evaluation with RUFDATA: Theorising a practical approach to evaluation planning. Evaluation 6(1) 7-21.

vi.  Stufflebeam, D. L. (2011). Meta-Evaluation. Journal of MultiDisciplinary Evaluation, 7(15). 

Some key points to come from this focus group included:

• allow users to begin at any point as some felt step 1 (outcomes) was not their logical starting point.
• use the langauge of progress rather than achievement.
• add back in a question about recommendations for the future - for people who want to pick up where this project leaves off.
• an iterative approach to the steps would also be useful. Feedback loops. Once you start your project you usually revisit these steps and answers may change as you go.
•  collecting evidence - not only what but also how (many people forget ethics approval)