The aim of the focus group was to test the waters with the framework. Get academics with small L&T projects in a room. Talk them through the different stages and get them to give examples for each step. Then find out what they think of the framework.
I was really excited at the prospect of 10 participants as two previous attempts to run this at MQ had resulted in only 2 and 1 respondents respectively. However luck was not on my side again - the torrential rain and storms in Sydney on the day prevented three from attending. However the focus group went ahead with 7 participants.
2 male 5 female. Four faculties covered included:
Arts and Social Sciences (Journalism; Communication; Education)
Business (Accounting)
Engineering & IT (Civil and Environmental Engineering)
Health (Nursing)
Two of the participants had had large scale (OLT type) L&T grants, one was new to the L&T grant space and others had received numerous small scale L&T grants.
General Observer Comments:
there was a general openness to talking about evaluation and a positive vibe in the room during the discussions particularly when talking about their projects.
A diagram would have helped when describing evaluation and research synergies.
There was also a general understanding of the value of evaluation and a willingness to embrace it in their projects.
However some people felt overwhelmed by the info/framework. Too many steps. This lead to comments regarding 'having to' complete it. And negativity that it would take too much time to complete and if this was required then it would put them off doing evaluation. --> this kid of missed the point (I thought) - as this is meant as a resource to help them formulate their evaluation plan.
I asked for definitions of Evaluation (with respect to L&T projects).
- quality control - monitoring level of proficiency
- feedback - monitoring and improving learning outcomes
- an assessment of a particular project, either qualitative or quantitative
- feedback re: effectiveness in student learning and efficiency/efficacy in delivery of content and development of student directed expectation
- a process to determine if the stated projects aims and objectives were achieved and if not what we can learn from it
- finding out it it meets the intended purpose of the project
- identify what works/needs review
- comparison of outcome against objectives (designed at beginning of project)
I also asked them to say how they 'felt' about evaluation. Words and phrases included:
- useful to gather "lessons learned"
- essential part of project feedback loop
- must be multidimensional
- necessary useful tool
- useful if done halfway through project rather than just at the end
- intrusive on time
- useful at critical stages for modification of delivery
- great! I love to see how its evolved, turned out, even if its a catastrophe!
- its required to improve subject quality
- I welcome evaluation as long as its not an unwieldy or unnecessarily complex process
- I see it as a critical and integral part of any project
Then at the end of the session I asked them to reconsider evaluation and write whether their thoughts or feelings about evaluation had changed since the beginning.
- two people said no change
- four stated they were now more aware of different evaluation foci/purposes
- one stated they were more 'dispirited' because of the quantity of work required to complete the framework
Next steps:
1. Transcribe the audio and analyse data.
2. Transpose examples given by participants in the 'workshop' section to the framework document
3. Think about which steps could be reduced by either combining or removing.
4. Run the session again with the revised framework.
