Meta-evaluation of methods for selecting projects

Brandon, Paul R. “A Meta-evaluation of Schools’ Methods for Selecting Site-managed Projects.” Studies In Educational Evaluation 24, no. 3 (1998): 213–228.

A meta-evaluation of 17 schools who apply for funding from a state-wide initiative. The authors were interested in finding out how the schools evaluated which projects were put forward for funding application.
There are three types of evaluative efforts required by schools, needs assessment; project evaluation (when searching for the one to best meet the student and school needs); and summative and formative evaluation after project implementation. This study investigates the second type ie school based evaluative efforts and activities used when selecting educational projects to address their identified needs.

The results showed that the extent to which teachers participated in making decisions about both the process and content of needs assessments was positively related to the validity of these decisions. (p.214)

Evaluation criteria came from using the CIPP approach to evaluation (Stufflebeam, 1983). The four criteria are (a) the extent to which all faculty and staff participated in selecting projects, (b) the extent to which school personnel used the appropriate sources of information, (c) the extent to which the schools compared their preferred projects with other available projects before making their final project selections, and (d) the extent to which the schools considered issues of feasibility such as project cost and ease of implementation. (p.216). A fifth criterion was used, based on the belief that projects are most likely to succeed when they are based on theories of education and have been shown to have succeeded elsewhere (Ellis & Fouts, 1993; Slavin, Karweit, & Madden, 1989). These five criteria then supported the five evaluation questions to be asked: (a) To what extent did school personnel participate in project selection? (b) To what extent did the schools collect information about possible projects from the appropriate types of sources? (c) To what extent did the schools compare their preferred projects with others before making final project selections? (d) To what extent did the schools consider project cost and ease of implementation when selecting projects? (e) To what extent were the selected projects based on theory and supported by empirical findings of previous studies? (p.218)

Two data collection methods were used, a self-report survey questionnaire (for first four questions) and a literature review (for fifth question). Findings: results were encouraging for two of the questions (b and c) and not so encouraging for the remaining three (a, d and e). Three categories of problems that the schools had encountered when implementing their projects were identified.

• inaccurate estimates of project costs.
• misjudging the managerial, administrative, or logistical requirements of the projects.
• underestimating the level of staff, parent, or community understanding or~motivation required for successful project implementation.

[useful when writing about my findings from Phase 1]

And the final question highlighted that empirical evidence about project success was not found for about half of the schools. This could be explained because school personnel often know little about proper use of research findings and this should not be the case with HE project proposals.

And Selecting projects is a task that is added to schools' full schedules, and, typically, few faculty and staff are available or willing to participate. [same could be said of HE] However authors recommend that schools should be shown the advantages of allowing as many of their staff and faculty as feasible to participate in project selection. Assistance of this sort would improve the chance that the best projects would be identified to meet needs and would help ensure that project funding is well spent.