[* This article is based on two electronic messages submitted to NCRVE's DISSMN8 list, posted by Sue Klein of the US Department of Education, Office of Educational Research and Improvement. The ideas, points of view, and opinions expressed in this *CenterWork* article are those of its author. They do not necessarily represent Dr. Klein's or the official U.S. Department of Education position or policy including, but not limited to those of its Office of Educational Research and Improvement and the USDED Office of Vocational and Adult Education. All information in this article concerning OERI's initiative is accurate as of this writing, but is not the final word given the initiative is still very much under development and review.]
While national systems whose task is to share information about education are alive and well (for example, ERIC, the Internet, the World Wide Web), this past year has witnessed the demise of several national educational dissemination programs. These programs were part of a national infrastructure that helped state and local educators to identify and replicate successful materials and programs in their areas. Among the "casualties" were the National Diffusion Network (NDN), the National Network for Curriculum Coordination (NNCC), and the Regional Centers on Drug-Free Schools and Communities. While pieces of this infrastructure still remain, the programs no longer exist as national systems for disseminating data, knowledge, and information. And their absence is sorely missed.
For example, the most inquired about topic when folks call the National Center's Dissemination Program is for exemplary curricula and programs. And the question, "What works?" is arguably the most frequently asked question in education. When the NNCC as well as the NDN existed, our information brokers had national systems to which to refer the callers.
Another example comes from a colleague who worked at one of the defunded Regional Centers for Drug-Free Schools and Communities. Each day now she is inundated with calls to her home from frustrated people who have no national system from which to gain needed information. Sadly, she lacks the time and resources to assist these callers.
Looking into the near future, state and federal governments may find that they need the very infra-structure that has recently been dismantled through zero funding. Ironically, they may end up having to rebuild these same systems.
Fortunately, the U.S. Department of Education's Office of Educational Research
and Improvement (OERI) is now working to fill a part of this vacuum. OERI was
mandated by its 1994 reauthorization to develop methods to identify and share
both promising and exemplary materials and programs--or what we shall refer to
in this article as "educational innovations." OERI seeks an approach to
dissemination and replication that is realistic, acknowledging differences in
users and the important role of careful adaptation.
In order to carry out its mandate, OERI is establishing a
system of expert panels to carry out this mandate. Their job is (1) to help
educators learn about effective and replicable innovations so they can make
informed choices; (2) to help federal sponsors and other funding organizations
make more informed and cost-effective decisions when considering refunding
programs for evaluation, continued development or dissemination purposes; and
(3) to help such sponsors identify gaps in research and development where new
support would be useful.
OERI, which was responsible for maintaining the now extinct NDN, favors the expert panels approach. In the past, NDN utilized a single, independent panel which passed judgment on which programs should receive exemplary status. OERI envisions expanding this idea into a system of expert panels on various topics, established by experts in those areas. OERI would assist panels in the coordination and facilitation of their activities, it would encourage the panels to design many of their own activities to address the needs and opportunities in their topic area.
One of the main charges of the expert panels would be to distinguish between those programs considered "promising" and those considered "exemplary." To do so, evaluators would consider four criteria or standards for educational excellence:
A key purpose for designating a program promising is that it clearly merits additional investment for (1) improving it, and (2) evaluating it so that it could justify a future exemplary designation. Using this purpose, a promising designation would encourage only cautious adaptation or adoption, during which time it would undergo careful evaluation. Users would be encouraged to select exemplary rather than promising innovations unless no exemplary innovations meet their need, or unless they wish to participate in the field testing of promising innovations.
The key purpose for designating a program as exemplary is to increase the use of what works best. Users would be more likely to adopt as their model a program that has received the exemplary designation. OERI plans to assist in publicizing the existence of exemplary programs through the World Wide Web, and TV and print media.
In recent electronically transmitted requests for comments, OERI first laid out several options by which this system of expert panels could discuss distinctions between promising and exemplary innovations. These requests were sufficiently provocative and of sufficient import to summarize them here.
As you read the following ideas, imagine that you have an innovative program or product which you believe is of such high quality that it would be of genuine use to others. Imagine further that a federal program exists to identify and disseminate promising and exemplary innovations such as yours.
Finally, imagine that your innovation is to be scrutinized by a federal expert panel. Perhaps it has to do with school-to-work, technology education, workplace mentoring, or business/community/school partnerships. Which of the following four options seem most viable to you? Why?
A description of the four options follows, as does the choice OERI has made (Option 3) that it will recommend for the Department of Education regulations on these standards. Nevertheless, as you reflect on the substance of the article--and if you are sufficiently interested--we have included contact information at this article's end so that you, too, can offer your views to the Department.
In its first electronic posting, OERI laid out four options that the expert panels might use when distinguishing between promising and exemplary innovations.
Option One: Distinction Based Only on Effective Replication.
A program may be designated as promising if the panel determines that the program has met the educational effectiveness standard with respect to one context or one population. A program may be designated as exemplary if the panel determines that the program has met the educational effectiveness standard with respect to multiple contexts or multiple populations.
Option Two: No Clear Minimum Expectations for Either Promising or Exemplary on any of the Four Standards of Educational Excellence.
A continuum of excellence would be established by each panel for each of the four standards. No system-wide minimum expectations would be established for any of the four standards for either promising or exemplary innovations. Those innovations rated lower on the four standards categories would be called promising. Those rated substantially higher would be called exemplary.
Option Three: Focus on Evidence of Effectiveness, but Establish Minimums for Promising and Exemplary on the Other Three Standards of Educational Excellence.
An innovation must attain minimum expectations on all four standards to be called promising. Each panel will decide on its own minimums for all categories except that of evidence of effectiveness, for which systemwide minimums will be established. To be exemplary requires an "excellence" rating on all four standards categories as determined by each expert panel.
Option Four: Number of Categories Judged Excellent: Exemplary Requires Four; Promising Requires Only Three.
Exemplary requires an "excellence" rating on all four standards categories as determined by each expert panel. Promising requires an "excellence" rating on at least three standards categories. Thus, any innovation may be approved as promising even if no evidence of effectiveness exists (that is, if the innovation is rated excellent on the other three standards categories).
Most respondents agreed, however, that each expert panel should set its own minimums for designations of promising and exemplary in the standards categories for quality, significance, and usefulness to others.
Additionally, the flexibility in Option 2, which empowers panel members to decide on cutoffs within the standards categories according to a continuum of excellence has been integrated into Option 3 by "popular demand" of the respondents. Individual panels will be given this responsibility on all standards categories except effectiveness. Additionally, the revised Option 3 is more specific than Option 2, because it requires the panels to be clear about minimums for promising and for exemplary in each standards category. (For the complete text see the OERI's proposal)
Since evidence of effectiveness is particularly critical to the purposes of this system of expert panels, expert panels in all topic areas will use identical effectiveness criteria to distinguish between promising and exemplary programs. However, since there are many topic area differences related to the other three standards categories, each expert panel will establish its own criteria for determining adequacy in the case of promising designations and excellence in the case of exemplary ones.
The question is from where would such public support come? How about legislating it? The School-to-Work Opportunities Act (STWOA) mandates or allows a host of school-based and work-based activities that could evolve into promising and exemplary innovations. Could STWOA monies be used to support expert panels to identify promising and exemplary innovations? How about Perkins funds? Could resources from either of these federal laws be used to establish a pilot project in which an expert panel on a single topic, such as workplace mentoring, would be established? Could the panel's creation and operation then be studied for use in writing future legislative language? And what about future workforce development legislation? Could resources be allocated for expert panels not only to identify promising and exemplary innovation, but also to support continued development of promising ideas and sharing of exemplary ones?
OERI has a solid vision, and one we believe is worth committing oneself to: The creation of federally supported expert panels "owned" by the constituency groups expert in the panel's selected topic, whose purpose is to identify promising and exemplary innovations within their area of expertise. Once identified, resources would exist so that exemplary products and programs could be shared. Equally important, promising innovations could be evaluated and revised into exemplary status--an unusual approach for a federal program, but one with great possibilities.
Thus, OERI is offering us a vision, one which it asks us to accept. It believes that identifying and sharing promising and exemplary innovations is important to the improvement of American education. Simultaneously, the OERI vision allows constituency groups--such as vocational educators--to own the identification and dissemination systems, and to customize these systems to the idiosyncrasies of the topic and those who will use them.
Having your cake and eating it too? Perhaps, and it sounds worth the gamble.
If you have comments on these summary recommendations or would like a copy of the initial posting on the options, please contact Dr. Sue Klein (sue_klein@ed.gov) Tel. 202-219-2038, or Dr. Sharon Bobbitt (sharon_bobbitt@ed.gov) Tel. 202-219-2126, Fax 202-219-1407, OERI, ORAD, Room 508, 555 New Jersey Ave., NW., Washington, DC 20208. Please provide your e-mail, fax, or postal mail address. Dr. Klein can also send you a copy of the draft regulations.
Peter Seidman is the director of the NCRVE Dissemination Program.