Update Aug 2014

Where did the year go?? I was on a roll back in April, investigating types of coding and starting work thorugh the transcripts. Then suddently its the middle of August and I've completely forgotten where I am at!
I'll start by going back to the coding bible (Saldana) re reading my scribbles and trying to pick up where I left off.

I haven't been completely redundant. I've been scratching my head for a long time considering what to do about my rejection of paper 3 from the Journal of Further and Higher Education. This was the paper that reported outcomes from phase one of the project. I attempted to ask for a fourth reviewer because I felt that one of the reviewers had not read the paper or rather had jumped to inaccurate conclusions about what I was trying to do. Anyway that led nowhere so have now resubmitted to the International Journal of Educational Management. That was about a month ago so will sit tight and wait to hear.

In the meantime I have been trialling the modified Chesterton & Cummings Evaluation Framework with my OLT project for which I am an external evaluator. There has been good feedback from the team about how this has really helped them keep their focus and keep the project on track. We are also in discussion on writing a paper together to describe how the evaluation has worked for them.

I had a workshop abstract accepted at the Australasian Evaluation Society's annual conference in Darwin. However they were unable to attract enough participants so I am now reconsidering my attendance. It would have been a great opportunity to trial the framework and application with a wider audience.

And finally I have submitted an abstract to the AAIR (Australasian Association for Institutional Research) Forum 2014. This year they have partnered up with the Australasian Higher Education Evaluation Forum(AHEEF). I was particularly interested in two of the Themes: 

• Closing the Loop – Putting Evidence into Practice
• Influencing Change through Information and Evaluation

So am just waiting to hear back to see whether I have been successful.

Update Jan 2014

Agenda for this month is to update the gantt chart - where are we what are we adjusting/modifying: Acting as a project evaluator I am able to test out my framework - it may be useful to include this as a paper in my study as findings will certainly impact on stage 3. Actually if i think about it, that was what my original prposal suggested - that i use the framework to evaluate 3 local projects. I didn't end up doing that as it wasn't my role. Phase 2 has turned into a triangulation/measure of the findings from phase 1 and the lit review.
Thoughts on triangulation: use of three case studies offering an in depth look at the findings from phase one and the literature review - contextualising what's happening. Deeper richer insights to the findings.

There is a need to engage people and bring them into the process of evaluation - what needs to change? what comes next - this is the contribution to the knowledge.

Also need to think about publications and budget spending for this year: aiming for one new publication ie not including the article that went to JFHE. Need to target a journal for the case study manuscript.

progress check

Well its half way through my 2nd full year of PhD and still finding my way. This year feels like it has mostly been writing and rewriting. Phase 1 findings have been 90% written up into a conference paper that was originally intended for the Australasian Evaluation Society Conference in Brisbane. It was accepted but then funding from the department was not forthcoming. So a last minute change of plans required a rewrite of the abstract and have recently received acceptance to The 5th Asian Conference on Education. It's in Osaka but I will hopefully present virtually (funding pending). My dilemma now is how to reduce what is about an 8 and a half thousand word paper into just 5000 for the conference. Also how do I ensure the paper is selected for the journal as a publication is really my intended outcome for participating in this conference.

My literature review has also been submitted to the Journal Studies in Educational Evaluation. This was peer reviewed and received considerable constructive feedback. However the editor did invite resubmission so that has been done and I'm now awaiting the second round of feedback (or rejection).

So it is time to return to data collection for phase 2. I'm transcribing interviews from phase 2 and need to soon turn to analysis and writing. But first need to do wrap up interviews for phase 2 case studies.

What have I learned from all this writing? That improvements can always be made; that people see and read different meanings than you sometimes intend; that writing short (5000 word) articles is difficult; that I am able to write.

Also, I need to do some more work on theory. I have briefly touched on the use of pragmatism or realism but it might be interesting to try and write either a short paper or definitely more for the chapter in the thesis.

And finally whilst considering how to cut down paper on phase 1, we have discussed the possibility of removing the Leximancer information and either trying to publish something on the methodology, using the results tables as examples or  putting that information into the thesis.

The write/right journal

I'm going to try and publish my literature review but the question i'm now finding hard to answer is which journal. The suggestion was to target one of those journals mentioned in the paper. So for each of these I'm checking the journal requirements, particularly the length of papers accepted since this one is particularly long. Summaries of journals are on this page. Studies in evaluation seems like a good possibility, they have really long articles (over 9000 words) and 4 of the articles in my paper came from this journal. Another possibility is Assessment and Evaluation in Higher Education but they limit articles to between 3000 -5000 words. I'm also interested in the journal of multidisciplinary evaluation which has long articles - this is open source and states they have a quick turn around for feedback. They suggest a limit of 10-12 pages though they do accept longer articles.

Another Website?

At my supervision session this week, we have been talking about the possibility of creating a public website to publish the lit review. A website would become a useful resource, allow public comment and become the place where I could show my results particularly if i go on to develop an interactive evaluation instrument.

The question is, can this website act as a publication? I could then refer to the table (which is growing unwieldy for a word doc)  when I write the paper on the literature synthesis. Questions to ponder

Question 10 - reporting strategies and feedback

The following reporting strategies were used:

15 final report to funding body

6 reports or products to audience

5 interim reports

5 conference presentations

4 reports to stakeholders

2 journal articles

2 reports to department (meetings)

2 faculty L&T

1 minutes of project meetings to stakeholders

1 report to faculty executive 


When asked how their reports were received, 10 out of the 15 participants reported receiving no feedback other than thank-you. This provoked a range of reactions from indifference to relief to annoyance. I think it was clear though that more than a tokenistic feedback is required. Even if it was a follow up by someone to ask about the project - show interest in how it has developed or moved on with.


One participant mentioned that it would be good if they could receive help with publishing in the L&T field rather than their discipline field - which of course they can do easily. Considering a number of people mentioned conference and journal articles then perhaps this is a service the LTC could follow up with?

Possible Journals

ERA ID	Title	Notes
39989	American Journal of Evaluation	This has article from all the big names in Evaluation, many are position papers and seminal works. Very American heavy.
19964	*Assessment and Evaluation in Higher Education	ideally between 3000 and 5000 words. publishes papers and reports on all aspects of assessment and evaluation within higher education. Its purpose is to advance understanding of assessment and evaluation practices and processes, particularly the contribution that these make to student learning and to course, staff and institutional development.
19958	Educational Assessment, Evaluation and Accountability	an international journal that investigates and discusses the functions, theories, values and practices of assessment, evaluation and accountability as they impact schools, higher education and educational systems, and looks further to their effects on homes and communities. The topics covered range broadly, encompassing teacher and human resources evaluation; program evaluation; school self-evaluation; policy evaluation; accountability thinking, policy and programs; summative assessment and testing; formative assessment and assessment for learning and both self- and peer assessment. The editors view these not as disparate fields, but as facets of one larger issue – the nature of education in a changing social and policy context. no words or length quoted but from looking at range of articles average size is about 13 -17 pages
19927	Educational Evaluation and Policy Analysis: a quarterly publication of the American Educational Research Association
20022	Educational Research and Evaluation: an international journal on theory and practice	notes imply that this journal incorporates the Evaluation and Research in Education? Has same aims as below. Its possible that that journal has been discontinued? Papers should normally be around 8,000 words in length, but longer or shorter articles will be considered.
10328	Educational Technology Research and Development
19967	*Evaluation and Research in Education last publication 2011	fully refereed international journal that publishes original academic articles aimed at promoting educational processes and outcomes, both well-informed and evidence-based. It is deliberately multi-disciplinary and broad in scope, and addresses rigorous research and evaluation at all levels, and in all sectors and ages of education.  The journal publishes papers which report results of: original research and evaluation studies; secondary analyses of existing (international) data sets; meta-analyses and reviews synthesizing evidence and guidelines for practice.
39946	Evaluation Journal of Australasia	Not really an educational focus - many public sector projects, heavy on the health services. Preference will be given to articles that embody original concepts, significantly new findings or methodological advances, or which present existing knowledge in a form particularly accessible or useful to students, practitioners and/or users of evaluation. As a general rule, simple records of evaluations undertaken will not be considered suitable as articles unless, in the opinion of the editors or peer reviewers, they are innovative in methodology or other aspects of practice, significantly advance knowledge of the relevant field, or represent outstanding models of good practice for others conducting evaluations in the same areas.
20348	Higher Education Research and Development
20056	Practical Assessment Research and Evaluation	journal requirements at http://pareonline.net/Policies.htm Two key questions in the review will be whether the results generalize and whether the results are likely to alter someone's practice. views evaluation as an assessment tool - no real examples of project evaluation
22288	Research Evaluation	Research Evaluation is an interdisciplinary peer-reviewed, international journal. Its subject matter ranges from the evaluation of an individual research project, through portfolios of research and research centres, up to inter-country comparisons of research performance. It covers public and private sectors, natural and life sciences as well as social sciences and humanities. The term 'evaluation' applies to all stages of research from priority setting and proposals, through the monitoring of on-going projects and programmes, to the use of results of research and the increasing trend of integrating evaluation into policy. Evaluations of research output and impact are particularly relevant given the emphasis today on accountability and documenting the value of research. Papers on methods for appraising and evaluating research are also welcome. The Journal is not committed to any specific approach or philosophy, quantitative, qualitative or otherwise.
20767	*Studies in Educational Evaluation	Studies in Educational Evaluation publishes original reports of evaluation studies. Four types of articles are published by the journal: (a) empirical evaluation studies representing evaluation practice in educational systems around the world; (b) theoretical reflections and empirical studies related to issues involved in the evaluation of educational programs, educational institutions, educational personnel and student assessment; (c) articles summarizing the state-of-the-art concerning specific topics in evaluation in general or in a particular country or group of countries; (d) book reviews and brief abstracts of evaluation studies. Appears to be long articles - over 9000 words
20774	*Studies in Learning Evaluation Innovation and Development looks promising but i think discontinued...	A scholarly, peer-reviewed, international journal that supports emerging scholars and the development of evidence-based practice and that publishes research and scholarship about teaching and learning in formal, semi-formal and informal educational settings and sites One binary that STUDIES IN LEARNING, EVALUATION, INNOVATION AND DEVELOPMENT is particularly concerned to challenge is the so-called ‘theory/practice divide’. The journal’s editors assume that, rather than research and practice being separate and even mutually exclusive, they are on the contrary interconnected and interdependent: effective research must be situated in ‘real world’ contexts and settings; and effective practice must be explicitly framed and informed by research. online open access - CQU length varies - about 12 pages but have seen upto 16 pages
20208	Quality Assurance in Education	a journal which is committed to a critical examination of quality and related issues in education. It devotes itself to the dissemination of best practice on the management of change and improvement in the higher education. It invites insights into the perceptions and opinions of quality in education of a number of stakeholders to gain a balanced view. Key audiences: Producers and consumers of education services; Libraries supplying the above; Strategic managers of education services; Local and central government; Employers, trade unions and community groups international articles on quality frameworks and students perceptions of Quality etc
20358	Quality in Higher Education	an international refereed journal aimed at those interested in the theory, practice and policies relating to the control, management and improvement of quality in higher education. The editor especially wishes to encourage papers on: reported research results, especially where these assess the impact of quality assurance systems, procedures and methodologies; theoretical analyses of quality and quality initiatives in higher education; comparative evaluation and international aspects of practice and policy with a view to identifying transportable methods, systems and good practice. Contributions should be 5000–6000 words, should include an abstract of 100–150 words.
	Journal of multidisciplinary Evaluation	http://www.jmde.com JMDE publishes a selection of the following types of material: news of and letters or memos about evaluation activities, meetings, discussions, developments, ideas, controversies, etc., from around the world, and from evaluation publications; scholarly articles about evaluation practice, methods, history, and theory of some general interest (we do not publish plain evaluation reports); scholarly comments on previous material in JMDE; reviews of important books or journal issues; expository materials in modular form, on topics or evaluations that readers have shown some interest in clarifying; evaluation puzzles or problems for competitive submissions; evaluation humor and cartoons.  Submissions can be of any length that the content justifies, from 1-50 pages. But we think longer than 10-12 pages is hard to justify, and brevity takes brains and style.
	Issues in Educational Research	focuses on issues related to educational research must be timely, significant, clear, logical and concise.  IIER is a generalist journal publishing research topics which are drawn from all sectors of education and may be based upon a wide range of contexts and research methodologies. It is important to project your article as relevant for an international readership, in contrast to an article that relates only to a local or limited regional context.
	International Journal of Educational Research	Length For individual papers, a length of between 5,000 - 8,000 words is acceptable. Proposals for special issues and individual papers can be on any contemporary educational topic of international interest. Reports of high quality educational research involving any discipline and methodology will be welcome. The journal's purpose is to enable researchers throughout the world to engage more easily with findings and issues which are relevant to their own interests.  The research reported does not have to be comparative (in the sense of comparing aspects of education in different countries or cultures); a paper may report research carried out in just one location or cultural setting. But to be eligible for publication, a paper must:  • deal with a topic which is of interest to a substantial international audience of educational researchers  • explain the research in a way which will be comprehensible to an international audience of researchers  • draw out explicitly in a discussion of the research those issues or themes which are likely to have international relevance.  As well as papers which report the findings of empirical research, papers which review research on specific educational topics of international interest will also be welcome.
A	Journal of Higher Education	The Journal of Higher Education publishes original research reporting on the academic study of higher education as a broad enterprise. We publish the highest quality empirical, theoretically grounded work addressing the main functions of higher education and the dynamic role of the university in society. We seek to publish scholarship from a wide variety of theoretical perspectives and disciplinary orientations. Articles appearing in the Journal employ an array of methodological approaches, and we welcome work from scholars across a range of career stages. Comparative and international scholarship should make clear connections to the U.S. context. Manuscripts not appropriate for submission to the Journal include purely theoretical papers, methodological treatises, unsolicited essays and reviews, and non-academic, institutional, and program evaluations or reports.

First PhD Publication

I decided to try and get my research proposal published and was very surprised and pleased to find that I was successful.
The conference was part of the AACE series, called Global TIME (Online Conference on Technology, Innovation, Media & Education) and my paper was a virtual brief paper, probably called a concise paper here in Australia.
I was able to include the development of the questions from phase one in addition to the original proposal and I was able to stress a little more about the design of the online interactive as this fit the conference themes a little better. And I renamed the title to 'the design of a meta-evaluation study.......'

Full citation: 
Huber, E. & Harvey, M. (2012). The Design of a Meta-Evaluation Study of Learning and Teaching Projects in Higher Education. Presented at Global TIME 2012: Global Conference on Technology, Innovation, Media & Education - Feb 7-9, 2012.

I had to get up at 3am to attend the online conference because it was an American based one and i had to prerecord the presentation (click here for the recording), but this worked really well as i could answer questions on my presentation whilst everyone listed to me rattle on. There were only a few questions but I was able to make contact with another conference attendee who asked to keep in touch as she was very interested in my topic.

Questions:

Manuel Frutos-Perez: some very interesting points Elaine. It's curious that we tend to be quite methodical when we teach our students research skills, but then we don't apply that to our own practice

Stefanie Panke: How do you choose the projects for phase 1?

Manuel Frutos-Perez: have you experienced any resistance to your analysis? I mean, faculty colleagues might tend to think "I know how to do this.."

Stefanie Panke: A few years ago, we looked at evaluation practices in e-learning in the context of experiences very similar to your insitution - namely, there is no "gold standard". Among other things, we did a content analysis of AACE conference proceedings.

Stefanie Panke: http://jolt.merlot.org/vol3no2/panke.pdf