Phase 3 planing - focus group

After a few false starts with running a workshop on the planning framework/instrument, finally got 10 people volunteering to attend a focus group.

The aim of the focus group was to test the waters with the framework. Get academics with small L&T projects in a room. Talk them through the different stages and get them to give examples for each step. Then find out what they think of the framework.

I was really excited at the prospect of 10 participants as two previous attempts to run this at MQ had resulted in only 2 and 1 respondents respectively. However luck was not on my side again - the torrential rain and storms in Sydney on the day prevented three from attending. However the focus group went ahead with 7 participants.

2 male 5 female. Four faculties covered included:
Arts and Social Sciences (Journalism; Communication; Education)
Business (Accounting)
Engineering & IT (Civil and Environmental Engineering)
Health (Nursing)

Two of the participants had had large scale (OLT type) L&T grants, one was new to the L&T grant space and others had received numerous small scale L&T grants.

General Observer Comments:

there was a general openness to talking about evaluation and a positive vibe in the room during the discussions particularly when talking about their projects.

A diagram would have helped when describing evaluation and research synergies.

There was also a general understanding of the value of evaluation and a willingness to embrace it in their projects.

However some people felt overwhelmed by the info/framework. Too many steps. This lead to comments regarding 'having to' complete it. And negativity that it would take too much time to complete and if this was required then it would put them off doing evaluation. --> this kid of missed the point (I thought) - as this is meant as a resource to help them formulate their evaluation plan.


I asked for definitions of Evaluation (with respect to L&T projects).

• quality control - monitoring level of proficiency
• feedback - monitoring and improving learning outcomes
• an assessment of a particular project, either qualitative or quantitative
• feedback re: effectiveness in student learning and efficiency/efficacy in delivery of content and development of student directed expectation
• a process to determine if the stated projects aims and objectives were achieved and if not what we can learn from it
• finding out it it meets the intended purpose of the project
• identify what works/needs review
• comparison of outcome against objectives (designed at beginning of project)

I also asked them to say how they 'felt' about evaluation. Words and phrases included:

• useful to gather "lessons learned"
• essential part of project feedback loop
• must be multidimensional
• necessary useful tool
• useful if done halfway through project rather than just at the end
• intrusive on time
• useful at critical stages for modification of delivery
• great! I love to see how its evolved, turned out, even if its a catastrophe!
• its required to improve subject quality
• I welcome evaluation as long as its not an unwieldy or unnecessarily complex process
• I see it as a critical and integral part of any project

Then at the end of the session I asked them to reconsider evaluation and write whether their thoughts or feelings about evaluation had changed since the beginning.

• two people said no change
• four stated they were now more aware of different evaluation foci/purposes
• one stated they were more 'dispirited' because of the quantity of work required to complete the framework

Next steps:
1. Transcribe the audio and analyse data.
2. Transpose examples given by participants in the 'workshop' section to the framework document
3. Think about which steps could be reduced by either combining or removing.
4. Run the session again with the revised framework.

first quarter 2015 update

Already I feel like i have gotten off to a better start to the year than last.
I had tried to run a focus group to get feedback on the development of the framework before moving on to constructing some kind of interactive one that I could trial. However it proved impossible to get anyone along to it at MQ.

Checked in with ethics at UTS and providing i did an amendment at MQ to include a UTS staff cohort, they were happy for me to go ahead and run the focus group at UTS (new employer).

Approval was also required by the DVC (Shirley). Once done, the invite went out and now the focus group/ workshop is scheduled for wednesday 22nd April. There are 10 participants rsvp'd. Each has a current L&T grant. MOre on that later.

Also, just in is feedback from IJEM that paper 3 (details of phase 1) has been accepted with minor revisions. So I am now looking back at data and analysis notes from 2012(!) It's dusty up there.... in here.... so am scouring this blog in the hope to find something useful since the reviewers want to see some more detail of analysis in the results and appendices.

more thoughts on analysis

Have finished coding all of the interview transcripts and finished up with 19 codes.

1. others who have connection to a project (steering committee, audience or ref grp)
2. changing nature of projects; contextual info
3. project management lingo
4. issues or challenges
5. time taken or timing
6. types of evaluation or evaluands
7. perceptions or emotions or conceptions
8. communications, dissemination activities
9. quality
10. feedback
11. experience
12. relationships, connections
13. influence
14. support (needing)
15. sustainability
16. impact
17. findings
18. where to from here - what's next, how is this meaningful
19. research

These were combined into 8 themes.


compiling frequncy counts of codes and themes (as %) and looking for patterns. Comparing PL and PM and then across projects. Going to use radar charts to see if there is a particular pattern for the role.

analysis notes - phase 2

Starting a second cycle of coding on the first of three sets of data. Initial coding interspersed with some InVivo Coding and Versus coding was used in first cycle.

Codes were transposed into a spreadsheet and then colour coded using a focused coding (Charmaz, 2006) approach.

The categories thus far are:

• People (who are connected with a project such as steering group, audience etc)
• changing nature of projects; contextual factors
• project management information
• issues or challenges
• tame taken or timing
• types of evaluation or evaluand
• perceptions, affective language, emotions, conceptions
• communications
• quality

Another option for 2nd cycle coding is Elaborative coding (Auerbach & Silverstein, 2003). In this approach, findings from previous research can be supported, strengthened, modified or disconfirmed (Saldana 2013). If I use this, I can work with the first phase findings which unveiled 4 themes from across the 15 completed L&T projects: conflation between research and evaluation; capability building in evaluation; resources (time and money) and an action approach to evaluation.

At this point I will start with Focused coding and perhaps simultaneously note if any other previous themes appear in the current data set.

Once I complete this with the first set of data I can separately work on the 2nd and 3rd data sets (projects/ case studies). Then using Case Study methods I can compare and contrast  findings from each case.

research questions - phase 2

Some reminders for this phase:

Keep revisiting the questions that drive this part of the study:

• Why was a particular evaluation approach chosen and how successful was it?
• (ALT) why did you choose not to evaluate?
• How can we overcome barriers to successful evaluation praxis?
• How does perception influence evaluative practice?

Case studies help to answer these how and why questions.

These are really the Level 2 questions of the case study (as opposed to the Level 1 questions which are those asked in the interviews). The verbal line of enquiry (level 1) is different from the mental line of enquiry (level 2). (Yin, 2009, p 87).

Level 3 questions are those asked about the pattern of findings across multiple cases. In my study, we want to find out whether there were similar issues encountered with the evaluation and how such issues were dealt with.

Level 4 questions - about the entire study. How do findings relate to other literature or to say phase 1 of my study.


Case study evidence can (and should) come from multiple sources. These may include documents, interviews, participant observation, direct observation, physical artifacts and archival records. In this study the first four types were used.

Update Aug 2014

Where did the year go?? I was on a roll back in April, investigating types of coding and starting work thorugh the transcripts. Then suddently its the middle of August and I've completely forgotten where I am at!
I'll start by going back to the coding bible (Saldana) re reading my scribbles and trying to pick up where I left off.

I haven't been completely redundant. I've been scratching my head for a long time considering what to do about my rejection of paper 3 from the Journal of Further and Higher Education. This was the paper that reported outcomes from phase one of the project. I attempted to ask for a fourth reviewer because I felt that one of the reviewers had not read the paper or rather had jumped to inaccurate conclusions about what I was trying to do. Anyway that led nowhere so have now resubmitted to the International Journal of Educational Management. That was about a month ago so will sit tight and wait to hear.

In the meantime I have been trialling the modified Chesterton & Cummings Evaluation Framework with my OLT project for which I am an external evaluator. There has been good feedback from the team about how this has really helped them keep their focus and keep the project on track. We are also in discussion on writing a paper together to describe how the evaluation has worked for them.

I had a workshop abstract accepted at the Australasian Evaluation Society's annual conference in Darwin. However they were unable to attract enough participants so I am now reconsidering my attendance. It would have been a great opportunity to trial the framework and application with a wider audience.

And finally I have submitted an abstract to the AAIR (Australasian Association for Institutional Research) Forum 2014. This year they have partnered up with the Australasian Higher Education Evaluation Forum(AHEEF). I was particularly interested in two of the Themes: 

• Closing the Loop – Putting Evidence into Practice
• Influencing Change through Information and Evaluation

So am just waiting to hear back to see whether I have been successful.

sharing project info

This is an observation that comes from my experience of working on projects and interviewing project managers. Often people say they want feedback from stakeholders. Preferably in a formative sense so that feedback can be reviewed and enacted upon within the project if necessary. Often people choose to use a project wiki and publicise this in order to invite comments. I wonder how effective this is. Will/do people really comment and give feedback in such a public space?

Others often quote final reports and journal articles as effective means of dissemination but also as channels for feedback. Such summative methods are unlikely to provide useful feedback that can be incorporated unless the project has a second phase perhaps either through a new grant or other means.