Visit to the OLT

Today we visited the Office for Learning and Teaching (OLT) in Sydney. We met with Siobhan Lenihan to discuss plans for my PhD project and to get feedback and ideas. Took the following notes:

ALTC changed policy to evaluate projects in 2009 - would be interesting to look at the difference before and after this policy change - what was done differently. Apparently they went from requiring summative evaluation to both formative and summative mechanisms.

contact C&C to find their results of their interviews - should be willing to share.

is there scope to find out what other HE ares doing? Look at the websites of QUT, Griffith, Monash and possibly Deakin

longer term impact - is there scope to do a follow up on projects, down the line to report on change and impact. I have founf this an interesting outcome of may of my interviews - people have expressed a desire to talk about their findings etc and have been pleased that there has been an interest in their work.


**James Dalziel has a teaching fellowship his evaluator is Grainne Conole, she is reviewing the outcomes at each stage of the followship**

visit OLT website and request read some of the evaluation reports - these can be sent.

networks national that could be used for dissemination opportunities - contact Melissa???

Promoting Excellence networks (PEN) - ask if they have knowledge about internal grant processes etc State-based grps with a uni rep on each

Lesley Wilcoxsen - Sally Kift took over the project "the whole of university experience"

Siobhan to send me - (Look out for) the upcoming report: Carmel McNought - Hong Kong Uni
large first year stem classes to do team work Glen Lawrie - project leader

the C&C framework has been tweaked slightly - have checked the changes - very minimal. Wondering if I should follow up on their recommended references/links and summarise some of these. They look like an exhaustive collection of evaluation materials.

Looking through a different lens

I decided this week that it would be a good idea to also interview the selection panel members who sit on the application review committee for the internal grants at MQ. This would give the perspective of what's required etc and fins out what value they put on evaluation.
Have written amendment to the ethics application and reworded the questions to suit. Will discus with Marina.

a meta-evaluation of a different kind

Means, Toyama, Murphy, Bakia, Jones (2010) Evaluation of Evidence-Based Pratices in online Learning: A Meta-Analysis and review of Online Learning Studies

I’m interested in this article for the methodology of how they went about their meta-evaluation (although I have to say the content is really interesting too and leads itself to the FLaMe program).

They specify the type of studies they were evaluating, these studies must have stringent designs (random-assignment or controlled quasi-experimental) and examine the effects for objective measures (of student learning). In relation to my project, I have not really specified the sample such as this and have used a sample of convenience – will this hold up to rigorous testing?

Furthermore I am not looking at quantitative studies. Evaluations tend to be more qualitative in their nature.

there was a main finding from the literature review - few rigorous studies in the area of interest had been published. In fact non had been found in the years of search and so the search was expanded by 2 years.

There were key findings from the meta-analysis and then there were findings from the narrative review. This review came from analysing the studies that were not able to be included in the meta-analysis due to the fact that they did not have a particular control condition. (p.xii)

Interesting comments on the potential for bias stemming from studies' authors dual roles as experimenters and instructors

The lit review and Meta-analysis were guided by four research questions. Context for the meta-analysis was given, explaining who commissioned the study (stakeholders) and the overall goal of the study. Then they described a conceptual framework for the topic [online learning] which included three key components.

Methodology

First define the topic [online learning], state what is included and what is not. Use this to define the categories to be searched for (three in this study).
List data sources and search strategies. Give years from and to and mention dates of different say for thesis etc.
List the databases that were used and the list of keywords (in the appendix).
Additional search activities - included a review of articles cited in recent meta-analyses and narrative syntheses of similar topics.
Key journals were identified and abstracts from each of these in a given period were manually reviewed.
A Google Scholar search engine was used with a series of keywords and any article abstracts retrieved were reviewed to ensure no duplication from pervious searches.

Scrrening was then carried out in 2 stages. Firstly abstracts were reviewed, giving studies the 'benefit of the doubt' (p.11) ensuring they met the inclusion criteria. Statement of number of papers included and number rejected (as percentages) with reasons for rejection.

Full text screen - the next pass had two stages, studies had to meet content relevance criteria (listed) as well as basic Quality (method) criteria (also listed.). A table of the primary reasons for exclusion was included detailing numbers for each reason and percentages. (p.13)

The coding of the study features was then detailed, regarding study features and study quality, then a paragraph explaining how the interrater reliability was checked.  Finally data analysis was explained, but this is completely statistical since the chosen studies were all quantitative, and a meta-analysis software was used for computations.

Possible Journals

ERA ID	Title	Notes
39989	American Journal of Evaluation	This has article from all the big names in Evaluation, many are position papers and seminal works. Very American heavy.
19964	*Assessment and Evaluation in Higher Education	ideally between 3000 and 5000 words. publishes papers and reports on all aspects of assessment and evaluation within higher education. Its purpose is to advance understanding of assessment and evaluation practices and processes, particularly the contribution that these make to student learning and to course, staff and institutional development.
19958	Educational Assessment, Evaluation and Accountability	an international journal that investigates and discusses the functions, theories, values and practices of assessment, evaluation and accountability as they impact schools, higher education and educational systems, and looks further to their effects on homes and communities. The topics covered range broadly, encompassing teacher and human resources evaluation; program evaluation; school self-evaluation; policy evaluation; accountability thinking, policy and programs; summative assessment and testing; formative assessment and assessment for learning and both self- and peer assessment. The editors view these not as disparate fields, but as facets of one larger issue – the nature of education in a changing social and policy context. no words or length quoted but from looking at range of articles average size is about 13 -17 pages
19927	Educational Evaluation and Policy Analysis: a quarterly publication of the American Educational Research Association
20022	Educational Research and Evaluation: an international journal on theory and practice	notes imply that this journal incorporates the Evaluation and Research in Education? Has same aims as below. Its possible that that journal has been discontinued? Papers should normally be around 8,000 words in length, but longer or shorter articles will be considered.
10328	Educational Technology Research and Development
19967	*Evaluation and Research in Education last publication 2011	fully refereed international journal that publishes original academic articles aimed at promoting educational processes and outcomes, both well-informed and evidence-based. It is deliberately multi-disciplinary and broad in scope, and addresses rigorous research and evaluation at all levels, and in all sectors and ages of education.  The journal publishes papers which report results of: original research and evaluation studies; secondary analyses of existing (international) data sets; meta-analyses and reviews synthesizing evidence and guidelines for practice.
39946	Evaluation Journal of Australasia	Not really an educational focus - many public sector projects, heavy on the health services. Preference will be given to articles that embody original concepts, significantly new findings or methodological advances, or which present existing knowledge in a form particularly accessible or useful to students, practitioners and/or users of evaluation. As a general rule, simple records of evaluations undertaken will not be considered suitable as articles unless, in the opinion of the editors or peer reviewers, they are innovative in methodology or other aspects of practice, significantly advance knowledge of the relevant field, or represent outstanding models of good practice for others conducting evaluations in the same areas.
20348	Higher Education Research and Development
20056	Practical Assessment Research and Evaluation	journal requirements at http://pareonline.net/Policies.htm Two key questions in the review will be whether the results generalize and whether the results are likely to alter someone's practice. views evaluation as an assessment tool - no real examples of project evaluation
22288	Research Evaluation	Research Evaluation is an interdisciplinary peer-reviewed, international journal. Its subject matter ranges from the evaluation of an individual research project, through portfolios of research and research centres, up to inter-country comparisons of research performance. It covers public and private sectors, natural and life sciences as well as social sciences and humanities. The term 'evaluation' applies to all stages of research from priority setting and proposals, through the monitoring of on-going projects and programmes, to the use of results of research and the increasing trend of integrating evaluation into policy. Evaluations of research output and impact are particularly relevant given the emphasis today on accountability and documenting the value of research. Papers on methods for appraising and evaluating research are also welcome. The Journal is not committed to any specific approach or philosophy, quantitative, qualitative or otherwise.
20767	*Studies in Educational Evaluation	Studies in Educational Evaluation publishes original reports of evaluation studies. Four types of articles are published by the journal: (a) empirical evaluation studies representing evaluation practice in educational systems around the world; (b) theoretical reflections and empirical studies related to issues involved in the evaluation of educational programs, educational institutions, educational personnel and student assessment; (c) articles summarizing the state-of-the-art concerning specific topics in evaluation in general or in a particular country or group of countries; (d) book reviews and brief abstracts of evaluation studies. Appears to be long articles - over 9000 words
20774	*Studies in Learning Evaluation Innovation and Development looks promising but i think discontinued...	A scholarly, peer-reviewed, international journal that supports emerging scholars and the development of evidence-based practice and that publishes research and scholarship about teaching and learning in formal, semi-formal and informal educational settings and sites One binary that STUDIES IN LEARNING, EVALUATION, INNOVATION AND DEVELOPMENT is particularly concerned to challenge is the so-called ‘theory/practice divide’. The journal’s editors assume that, rather than research and practice being separate and even mutually exclusive, they are on the contrary interconnected and interdependent: effective research must be situated in ‘real world’ contexts and settings; and effective practice must be explicitly framed and informed by research. online open access - CQU length varies - about 12 pages but have seen upto 16 pages
20208	Quality Assurance in Education	a journal which is committed to a critical examination of quality and related issues in education. It devotes itself to the dissemination of best practice on the management of change and improvement in the higher education. It invites insights into the perceptions and opinions of quality in education of a number of stakeholders to gain a balanced view. Key audiences: Producers and consumers of education services; Libraries supplying the above; Strategic managers of education services; Local and central government; Employers, trade unions and community groups international articles on quality frameworks and students perceptions of Quality etc
20358	Quality in Higher Education	an international refereed journal aimed at those interested in the theory, practice and policies relating to the control, management and improvement of quality in higher education. The editor especially wishes to encourage papers on: reported research results, especially where these assess the impact of quality assurance systems, procedures and methodologies; theoretical analyses of quality and quality initiatives in higher education; comparative evaluation and international aspects of practice and policy with a view to identifying transportable methods, systems and good practice. Contributions should be 5000–6000 words, should include an abstract of 100–150 words.
	Journal of multidisciplinary Evaluation	http://www.jmde.com JMDE publishes a selection of the following types of material: news of and letters or memos about evaluation activities, meetings, discussions, developments, ideas, controversies, etc., from around the world, and from evaluation publications; scholarly articles about evaluation practice, methods, history, and theory of some general interest (we do not publish plain evaluation reports); scholarly comments on previous material in JMDE; reviews of important books or journal issues; expository materials in modular form, on topics or evaluations that readers have shown some interest in clarifying; evaluation puzzles or problems for competitive submissions; evaluation humor and cartoons.  Submissions can be of any length that the content justifies, from 1-50 pages. But we think longer than 10-12 pages is hard to justify, and brevity takes brains and style.
	Issues in Educational Research	focuses on issues related to educational research must be timely, significant, clear, logical and concise.  IIER is a generalist journal publishing research topics which are drawn from all sectors of education and may be based upon a wide range of contexts and research methodologies. It is important to project your article as relevant for an international readership, in contrast to an article that relates only to a local or limited regional context.
	International Journal of Educational Research	Length For individual papers, a length of between 5,000 - 8,000 words is acceptable. Proposals for special issues and individual papers can be on any contemporary educational topic of international interest. Reports of high quality educational research involving any discipline and methodology will be welcome. The journal's purpose is to enable researchers throughout the world to engage more easily with findings and issues which are relevant to their own interests.  The research reported does not have to be comparative (in the sense of comparing aspects of education in different countries or cultures); a paper may report research carried out in just one location or cultural setting. But to be eligible for publication, a paper must:  • deal with a topic which is of interest to a substantial international audience of educational researchers  • explain the research in a way which will be comprehensible to an international audience of researchers  • draw out explicitly in a discussion of the research those issues or themes which are likely to have international relevance.  As well as papers which report the findings of empirical research, papers which review research on specific educational topics of international interest will also be welcome.
A	Journal of Higher Education	The Journal of Higher Education publishes original research reporting on the academic study of higher education as a broad enterprise. We publish the highest quality empirical, theoretically grounded work addressing the main functions of higher education and the dynamic role of the university in society. We seek to publish scholarship from a wide variety of theoretical perspectives and disciplinary orientations. Articles appearing in the Journal employ an array of methodological approaches, and we welcome work from scholars across a range of career stages. Comparative and international scholarship should make clear connections to the U.S. context. Manuscripts not appropriate for submission to the Journal include purely theoretical papers, methodological treatises, unsolicited essays and reviews, and non-academic, institutional, and program evaluations or reports.

Phase1 Interview reflections

E6LW

• May be good to also interview the evaluator
• the project leader felt the team's professional development was a large unintended outcome
• it would be good to have an evaluator involved at the grant writing stage
• This participant was very experienced grant writer and project leader. She now does external evaluations of other projects.
• the two year time frame for a project was good for one this size but because of the large number of universities taking part it was hard to find a time to get together and it was the external evaluator who insisted on f2f meetings. It felt like 6 mths passed before anything really happened.

i1NM

• this was the first internal grant applicant that I interviewed and there had been (as I expected) no real evaluation done.
• interviewee cited lack of understanding of what is required in evaluation 
• stated that it would be good to get more support or have some to go to ask questions to regarding evaluation at the beginning and during the start of a project etc
• but basically i couldn't really ask many of my questions as non of them were relevant since the answer to the first question was no.

i2JN

• No evaluation had been done, they considered action research to mean that they were incorporating self reflection as they go to improve the project. 
• They have left summative evaluation till end of the project so it hasn't been done but as we spoke interviewee then questioned it's value other than as personal learning for their next project.
• Liked what saw in the emerging technologies grants as they had a framework specified that was 'in your face' and helped with the grant application.

i3MN

• Wondering about the level of experience, should I rate people on a scale of expert. Surely answers will differ depending on whether ltc person or faculty discipline specific etc.

i4MR

• use of a QM approach was suggested in this interview indicating that there ought to be follow up on reports and results/products etc. 
• Feedback would help the project team 'learn' and this could be an area for development in the LTC.
• it could also lead to more scholarship of L&T of a requirement of the projects was to produce a paper or conference presentation. Not least a presentation at L&T week should be insisted upon.

i5RH

• There was some concern over workload issues. No time to do evaluation properly. 
• Systems and processes in the university can impact heavily on a project, unbeknown at the start of a project.
• Some interest in closing the quality cycle but in reality when is this done, by whom and for what purpose.

i6TA

• Comments around time avail for completing evaluation well 
• also that it would be good to have some type of form to help with evaluations

i7MH

• PAR is similar to developmental evaluation
• Very experienced with evaluation which obviously impacted on the success of this project in terms of taking it to the next level (external grants)

i8BH

• Confusion over this project as it was originally applied for under one type of grant (outside scope of this study) and then granted against another type (within this study's scope) but each grant had different requirements so some answers were responded to as per the original application. 
• However its interesting that no feedback was given as to repurposing the application for the different grant requirements
• Also, spoke of evaluation of the products ie workshops and not of the project.

i9BH

• There was some confusion here over the project. Interviewee was talking about evaluation of a program (which utilized the products that came from the project) rather than the project which was only to produce online modules.

i10PK

• Another interesting situation where the project evaluation seemed confused with evaluation of the product. 
• A request for some guidance for evaluation at the beginning was good to hear. 
• I think the interviewee really enjoyed talking about the project further down the line and the reflection was useful. 
• This could be a recommendation that projects are revisited further down the track and summative reports drafted after say 12 months?

i11PP

• "It's been very interesting reflecting with you on the project, it has highlighted some deficiencies."
• This interviewee was very detailed about the context and somewhat confused about what evaluation was or indeed how it was relevant to the project. 
• It didn't sound like any evaluation had been described in the application.

i12DJ

• No evaluation, but a very thorough final report detailing the whole project. 
• Dissatisfaction that this was tied to KPIs and then when 'done' received no feedback. Felt that it was used negatively ie 'you won't get KPIs signed off till report done.
• Had not done final report for grant committee. 
• Hesitant about evaluation felt that if it was enforced then wouldn't have applied for grant as that would feel like "big brother watching you". 
• Disgruntled at system as report sent into the void with no feedback or response other than from Ian Solomonides which caused some happiness.
• the application was reviewed by a faculty member and this was very valuable
• would be interested to hear what percentage of people who get grants actually deliver on what they said they would deliver

i13MD

• I would rate this person as very experienced in the l&t field and yet there was no evaluation carried out which surprised me. 
• There seemed to be angst about who cared about the project, and how difficult it was to instigate change even though the products were 'great'. 
• There was no steering group or stakeholder buy-in, and no eval, but then surprise that there was no traction for this great resource at the end. Well not exactly surprise but certainly frustration.

i14NA

• This interviewee seemed confused by the word evaluation, became suspicious and wary about being 'judged'. Was also very 'anti' the word generalisable as this created homogeneity and need to be specific in this discipline.. 
• Didn't appear to understand the meaning of evaluation, many questions provoked a response about others attitudes to changes in the curriculum. 
• It was also apparent that the project which was the creation of an online resource was extended to include the implementation of said resources and it was that that was evaluated (to some extent). So scope creep.
• Felt that no one was interested in the project - no one had time to help review it, no one to talk to about it.

First PhD Publication

I decided to try and get my research proposal published and was very surprised and pleased to find that I was successful.
The conference was part of the AACE series, called Global TIME (Online Conference on Technology, Innovation, Media & Education) and my paper was a virtual brief paper, probably called a concise paper here in Australia.
I was able to include the development of the questions from phase one in addition to the original proposal and I was able to stress a little more about the design of the online interactive as this fit the conference themes a little better. And I renamed the title to 'the design of a meta-evaluation study.......'

Full citation: 
Huber, E. & Harvey, M. (2012). The Design of a Meta-Evaluation Study of Learning and Teaching Projects in Higher Education. Presented at Global TIME 2012: Global Conference on Technology, Innovation, Media & Education - Feb 7-9, 2012.

I had to get up at 3am to attend the online conference because it was an American based one and i had to prerecord the presentation (click here for the recording), but this worked really well as i could answer questions on my presentation whilst everyone listed to me rattle on. There were only a few questions but I was able to make contact with another conference attendee who asked to keep in touch as she was very interested in my topic.

Questions:

Manuel Frutos-Perez: some very interesting points Elaine. It's curious that we tend to be quite methodical when we teach our students research skills, but then we don't apply that to our own practice

Stefanie Panke: How do you choose the projects for phase 1?

Manuel Frutos-Perez: have you experienced any resistance to your analysis? I mean, faculty colleagues might tend to think "I know how to do this.."

Stefanie Panke: A few years ago, we looked at evaluation practices in e-learning in the context of experiences very similar to your insitution - namely, there is no "gold standard". Among other things, we did a content analysis of AACE conference proceedings.

Stefanie Panke: http://jolt.merlot.org/vol3no2/panke.pdf

Quality

Reading Cooksey & Caricelli (2005) on Quality, Context and Use : Issues in Achieving the Goals of Metaevaluation.

There are many defined standards of quality and they basically relate to the evaluation models which are used in a study. Each model has a number of criteria under which a study can be evaluated against and therefore any metaevaluation must take into account these quality standards.

That brings me to the question in regards to my own study, what are the quality standards being employed. Since i have created a set of criteria based on a number of different models ie Scriven, Patton, Stufflebeam, Owen, Chesterton and Cummings. Should I be meeting with Stakeholders to find out what they constitute as quality for our context? This may help in turn, in ensuring the evaluation findings are put to better use (Johnson et al., 2005).

Another finding in this study referred to the lack of transparent information in the final reports. I'm finding in my search of the identified data (final project reports) the same thing, a lack of detailed data for a metaevaluation. The reports intended audience is different in the internal projects to the external projects.

The finding is that an organisation should identify what it may need from a metaevaluation and then ensure all evaluations that are conducted will one day be able to be metaevaluated. This would include defining the methodology in detail (for example).