Expert Panels for Promising and Exemplary Innovations:

A Fine Idea From the Feds

by Peter Seidman [*]

[* This article is based on two electronic messages submitted to NCRVE's DISSMN8 list, posted by Sue Klein of the US Department of Education, Office of Educational Research and Improvement. The ideas, points of view, and opinions expressed in this *CenterWork* article are those of its author. They do not necessarily represent Dr. Klein's or the official U.S. Department of Education position or policy including, but not limited to those of its Office of Educational Research and Improvement and the USDED Office of Vocational and Adult Education. All information in this article concerning OERI's initiative is accurate as of this writing, but is not the final word given the initiative is still very much under development and review.]

While national systems whose task is to share information about education are alive and well (for example, ERIC, the Internet, the World Wide Web), this past year has witnessed the demise of several national educational dissemination programs. These programs were part of a national infrastructure that helped state and local educators to identify and replicate successful materials and programs in their areas. Among the "casualties" were the National Diffusion Network (NDN), the National Network for Curriculum Coordination (NNCC), and the Regional Centers on Drug-Free Schools and Communities. While pieces of this infrastructure still remain, the programs no longer exist as national systems for disseminating data, knowledge, and information. And their absence is sorely missed.

For example, the most inquired about topic when folks call the National Center's Dissemination Program is for exemplary curricula and programs. And the question, "What works?" is arguably the most frequently asked question in education. When the NNCC as well as the NDN existed, our information brokers had national systems to which to refer the callers.

Another example comes from a colleague who worked at one of the defunded Regional Centers for Drug-Free Schools and Communities. Each day now she is inundated with calls to her home from frustrated people who have no national system from which to gain needed information. Sadly, she lacks the time and resources to assist these callers.

Looking into the near future, state and federal governments may find that they need the very infra-structure that has recently been dismantled through zero funding. Ironically, they may end up having to rebuild these same systems.

Filling the Infrastructure Gap

Fortunately, the U.S. Department of Education's Office of Educational Research and Improvement (OERI) is now working to fill a part of this vacuum. OERI was mandated by its 1994 reauthorization to develop methods to identify and share both promising and exemplary materials and programs--or what we shall refer to in this article as "educational innovations." OERI seeks an approach to dissemination and replication that is realistic, acknowledging differences in users and the important role of careful adaptation.

In order to carry out its mandate, OERI is establishing a system of expert panels to carry out this mandate. Their job is (1) to help educators learn about effective and replicable innovations so they can make informed choices; (2) to help federal sponsors and other funding organizations make more informed and cost-effective decisions when considering refunding programs for evaluation, continued development or dissemination purposes; and (3) to help such sponsors identify gaps in research and development where new support would be useful.

OERI, which was responsible for maintaining the now extinct NDN, favors the expert panels approach. In the past, NDN utilized a single, independent panel which passed judgment on which programs should receive exemplary status. OERI envisions expanding this idea into a system of expert panels on various topics, established by experts in those areas. OERI would assist panels in the coordination and facilitation of their activities, it would encourage the panels to design many of their own activities to address the needs and opportunities in their topic area.

One of the main charges of the expert panels would be to distinguish between those programs considered "promising" and those considered "exemplary." To do so, evaluators would consider four criteria or standards for educational excellence:

  1. Evidence of success/effectiveness
  2. Quality of the innovation
  3. Educational significance
  4. Usefulness to others

"Promising" Versus "Exemplary" Purposes

A key purpose for designating a program promising is that it clearly merits additional investment for (1) improving it, and (2) evaluating it so that it could justify a future exemplary designation. Using this purpose, a promising designation would encourage only cautious adaptation or adoption, during which time it would undergo careful evaluation. Users would be encouraged to select exemplary rather than promising innovations unless no exemplary innovations meet their need, or unless they wish to participate in the field testing of promising innovations.

The key purpose for designating a program as exemplary is to increase the use of what works best. Users would be more likely to adopt as their model a program that has received the exemplary designation. OERI plans to assist in publicizing the existence of exemplary programs through the World Wide Web, and TV and print media.

"Promising" Versus "Exemplary" Criteria

In recent electronically transmitted requests for comments, OERI first laid out several options by which this system of expert panels could discuss distinctions between promising and exemplary innovations. These requests were sufficiently provocative and of sufficient import to summarize them here.

As you read the following ideas, imagine that you have an innovative program or product which you believe is of such high quality that it would be of genuine use to others. Imagine further that a federal program exists to identify and disseminate promising and exemplary innovations such as yours.

Finally, imagine that your innovation is to be scrutinized by a federal expert panel. Perhaps it has to do with school-to-work, technology education, workplace mentoring, or business/community/school partnerships. Which of the following four options seem most viable to you? Why?

A description of the four options follows, as does the choice OERI has made (Option 3) that it will recommend for the Department of Education regulations on these standards. Nevertheless, as you reflect on the substance of the article--and if you are sufficiently interested--we have included contact information at this article's end so that you, too, can offer your views to the Department.

The Four Options

In its first electronic posting, OERI laid out four options that the expert panels might use when distinguishing between promising and exemplary innovations.

Option One: Distinction Based Only on Effective Replication.

A program may be designated as promising if the panel determines that the program has met the educational effectiveness standard with respect to one context or one population. A program may be designated as exemplary if the panel determines that the program has met the educational effectiveness standard with respect to multiple contexts or multiple populations.

Option Two: No Clear Minimum Expectations for Either Promising or Exemplary on any of the Four Standards of Educational Excellence.

A continuum of excellence would be established by each panel for each of the four standards. No system-wide minimum expectations would be established for any of the four standards for either promising or exemplary innovations. Those innovations rated lower on the four standards categories would be called promising. Those rated substantially higher would be called exemplary.

Option Three: Focus on Evidence of Effectiveness, but Establish Minimums for Promising and Exemplary on the Other Three Standards of Educational Excellence.

An innovation must attain minimum expectations on all four standards to be called promising. Each panel will decide on its own minimums for all categories except that of evidence of effectiveness, for which systemwide minimums will be established. To be exemplary requires an "excellence" rating on all four standards categories as determined by each expert panel.

Option Four: Number of Categories Judged Excellent: Exemplary Requires Four; Promising Requires Only Three.

Exemplary requires an "excellence" rating on all four standards categories as determined by each expert panel. Promising requires an "excellence" rating on at least three standards categories. Thus, any innovation may be approved as promising even if no evidence of effectiveness exists (that is, if the innovation is rated excellent on the other three standards categories).

Why OERI Chose Option 3

Most respondents to OERI's electronic posting preferred a revised Option 3, which allows for distinctions between promising and exemplary within each of the four standards categories. They supported the emphasis on evidence of performance effectiveness, and also liked having all panels use the same distinctions for evidence of effectiveness. A key distinction, preserved from Option 1, is that the educational innovation must demonstrate evidence of effectiveness in at least one site in order to be designated as promising, and in more than one site to be designated as exemplary.

Most respondents agreed, however, that each expert panel should set its own minimums for designations of promising and exemplary in the standards categories for quality, significance, and usefulness to others.

Additionally, the flexibility in Option 2, which empowers panel members to decide on cutoffs within the standards categories according to a continuum of excellence has been integrated into Option 3 by "popular demand" of the respondents. Individual panels will be given this responsibility on all standards categories except effectiveness. Additionally, the revised Option 3 is more specific than Option 2, because it requires the panels to be clear about minimums for promising and for exemplary in each standards category. (For the complete text see the OERI's proposal)

Importance of Evidence of Effectiveness

These revised distinctions between promising and exemplary innovations continue to emphasize the importance of examining evidence of effectiveness. Such emphasis is important because effectiveness is the most commonly omitted criterion in most educational review efforts and consumer decision making. Without such evidence, it is impossible to know if, and with whom, a program works. By making such evidence a requirement, the Department of Education will be providing a powerful incentive for its collection.

Since evidence of effectiveness is particularly critical to the purposes of this system of expert panels, expert panels in all topic areas will use identical effectiveness criteria to distinguish between promising and exemplary programs. However, since there are many topic area differences related to the other three standards categories, each expert panel will establish its own criteria for determining adequacy in the case of promising designations and excellence in the case of exemplary ones.

Exciting Possibilities for Voc Ed Expert Panels

OERI staff has made it quite clear that they welcome vocational education and school-to-work educators to design and operate expert panels to identify and share promising and exemplary innovations. Such panels can branch off into subpanels around domains within the purview of the full panel's charge. For example, within a school-to-work expert panel one could have subpanels encouraging submissions of innovations in school-based learning, work-based learning, and connecting activities. There could be subpanels on tech prep, curricular integration, business/industry/education partnerships. Without question, vocational education, school-to-work, and workforce development all have promising and exemplary innovations worthy of government support.

The question is from where would such public support come? How about legislating it? The School-to-Work Opportunities Act (STWOA) mandates or allows a host of school-based and work-based activities that could evolve into promising and exemplary innovations. Could STWOA monies be used to support expert panels to identify promising and exemplary innovations? How about Perkins funds? Could resources from either of these federal laws be used to establish a pilot project in which an expert panel on a single topic, such as workplace mentoring, would be established? Could the panel's creation and operation then be studied for use in writing future legislative language? And what about future workforce development legislation? Could resources be allocated for expert panels not only to identify promising and exemplary innovation, but also to support continued development of promising ideas and sharing of exemplary ones?

OERI has a solid vision, and one we believe is worth committing oneself to: The creation of federally supported expert panels "owned" by the constituency groups expert in the panel's selected topic, whose purpose is to identify promising and exemplary innovations within their area of expertise. Once identified, resources would exist so that exemplary products and programs could be shared. Equally important, promising innovations could be evaluated and revised into exemplary status--an unusual approach for a federal program, but one with great possibilities.

Thus, OERI is offering us a vision, one which it asks us to accept. It believes that identifying and sharing promising and exemplary innovations is important to the improvement of American education. Simultaneously, the OERI vision allows constituency groups--such as vocational educators--to own the identification and dissemination systems, and to customize these systems to the idiosyncrasies of the topic and those who will use them.

Having your cake and eating it too? Perhaps, and it sounds worth the gamble.

If you have comments on these summary recommendations or would like a copy of the initial posting on the options, please contact Dr. Sue Klein ( Tel. 202-219-2038, or Dr. Sharon Bobbitt ( Tel. 202-219-2126, Fax 202-219-1407, OERI, ORAD, Room 508, 555 New Jersey Ave., NW., Washington, DC 20208. Please provide your e-mail, fax, or postal mail address. Dr. Klein can also send you a copy of the draft regulations.

Peter Seidman is the director of the NCRVE Dissemination Program.

Table of Contents | Next Article | Previous Article