Current federal vocational education legislation expires in 1995, presenting the 104th Congress with an opportunity to reshape federal policy regarding secondary and postsecondary vocational education. At the time of this writing, it remains uncertain whether the Carl D. Perkins Vocational and Applied Technology Education Act (Perkins II) will be reauthorized in its present form or whether federal vocational education initiatives will be "merged into a broader workforce-development bill" (Sommerfeld, 1994, p. 18). Given the growing emphasis on measurable outcomes and standards in government programs, requirements for a system of outcome measures and standards are likely to be included in new legislation, regardless of its format.
A recent study by the National Center for Research in Vocational Education (NCRVE), Improving Perkins II Performance Measures and Standards: Lessons Learned from Early Implementers in Four States (Stecher et al., 1994), suggests ways to enhance accountability in future federal vocational education legislation. This paper reviews the findings and recommendations of that study and illustrates how they could be translated into legislation. Specifically, we offer suggestions for more effective outcome-based program improvement and accountability procedures. The paper focuses only on those provisions that relate to evaluation and program improvement-often discussed under the heading "performance measures and standards." These suggestions are presented as revisions to the existing Perkins legislation, but they are equally relevant in the context of broader workforce development legislation. Therefore, the report should be of general interest to policymakers responsible for workforce training at the state and federal levels.
Examples of changes in language we suggest in this report are less relevant than the principles that guided them. Specifically, the results of the study suggest that any federal vocational education effort should incorporate the following:
Furthermore, the language presented in this document may provide a useful starting point for other conceptualizations of the federal role in vocational education.
Perkins II has guided federal vocational education policy for the past five years. It has been a remarkably influential piece of legislation, in part, because most states have opted to apply its provisions more broadly than required. Although the federal government supplies less than ten percent of the total resources devoted to secondary and postsecondary vocational education, most states have applied Perkins II requirements to locally and state-funded vocational education efforts as well as to those funded with federal resources.
One of the most significant changes embodied in Perkins II was an emphasis on using systematic outcome data as a monitoring and improvement tool for programs. Under Perkins II, states were required to develop statewide "systems of core standards and measures of performance" that would be used to determine the success of vocational programs and to serve as a basis for local program improvement. If necessary, these systems could also be used to justify state intervention. States were given considerable flexibility in developing their statewide systems; the law specified only two outcomes that were required to be measured. At a minimum, each state was required to collect measures of learning and competency gains in basic and more advanced academic skills, as well as at least one measure of occupational competency attainment, job or work skill attainment, or retention or student placement. Any additional measured outcomes were at the discretion of the Committees of Practitioners and state Departments of Education.
Those who endorsed this approach to program monitoring in vocational education hoped it would lead to better evaluation, richer communication, more focused program improvement at the local level, and wiser use of state technical assistance capabilities. It was hoped that states would implement efficient systems that provided local administrators and instructors with meaningful performance data to assess the strengths and weaknesses of their vocational programs and to design new strategies to improve the academic, technical, and labor-market outcomes of their students.
These hopes have not been fully realized, although states have made substantial strides in implementing performance measures and standards systems as envisioned in the federal legislation. However, there remains considerable room for improvement. The following summarizes the results of a recent NCRVE study on the implementation and impact of the Perkins II measures and standards. These findings form the basis for suggested changes in the legislation, which are elaborated in the section entitled "Rationale for Legislative Changes."
In the spring of 1993, NCRVE initiated a two-year study of the effects of Perkins II performance measures and standards for vocational education. At the time of the study, states had three years to implement these provisions. The study examined seven states' progress in implementing statewide systems of performance measures and standards, the effects of these systems on local vocational programs and state agencies, and the factors that influenced local and state actions. [1]
Four states which were "early adopters" of measures and standards were initially selected for study. In each state, we interviewed staff in the state agency (or agencies) that administered secondary and postsecondary vocational education, and administrators and instructors in both a secondary and a postsecondary vocational institution in two geographically separated regions. Respondents were asked about a number of related themes, including the implementation of performance measures and standards, their integration with other educational reform efforts, and the impact of measures and standards on their vocational programs. Repeat visits to each vocational institution were conducted the following year. In the second year, three additional states were added to the sample to provide greater contrast in terms of implementation and to broaden our exposure.
This section summarizes the findings of the NCRVE study (Stecher et al., 1994). Substantial progress had been made in implementing measures and standards in the states that were visited, although much work remains to be done to make the systems function as envisioned in the law. At the time of our visits, little attention had been paid to building local- or state-level capacity for translating the measures and standards data into actions for local program improvement. These "leading edge" states were still largely engaged in developing and implementing their systems.
Furthermore, large variation was found in the states' approaches to the development and implementation of measures and standards. This variation was evident in almost every aspect of program implementation, including how the process was managed, who participated, and the level of resources devoted to it. These differences appeared to be jointly a function of the states' individuality and the flexibility inherent in Perkins II.
We identified several factors that affected implementation and contributed to the variation in state responses to performance measures and standards. [2] Some of these explanatory factors were elements of the local and state context, and are less responsive to federal policy intervention. Other factors are within the sphere of federal policy influence.
The level of flexibility afforded states was the first of five factors that could clearly be affected by federal actions. On the positive side, flexibility permitted states to create systems that were responsive to local conditions. On the negative side, the latitude afforded states increased the influence of state context, which heightened differences between states, and, in some cases, lengthened the implementation process.
The second explanatory factor was the separate and uncoordinated nature of the elements of Perkins II. The Perkins II priorities-measures and standards, integration, Tech Prep, and service to special populations-were not seen as a coordinated system at either the local or state levels. Similarly, performance measures and standards were not being used comprehensively to evaluate the other Perkins initiatives.
Third, there were neither models nor incentives for ensuring that performance measures and standards were used to improve programs. Perkins II contained an explicit framework for structuring systems, and a federal agency checked for compliance at the adoption stage. However, the law and regulations did little to emphasize information use. [3]
The fourth factor concerned resources and expertise at the state level. Perkins II created new responsibilities for state staff, but reduced the set-aside for state administration and provided little technical assistance. This presented a dilemma for states that lacked either the expertise or the resources to address these new demands.
Finally, the law mandated measurement of learning outcomes, even though there were few valid tests available for this purpose. The scarcity of appropriate tools for measuring selected learning outcomes led states to adopt alternatives that were less than optimal. States are still struggling with how to measure important student outcomes such as academic skill gains at the postsecondary level.
The study recommended several actions federal policymakers could take to enhance the future success of performance measures and standards in vocational education and to promote the program improvement goals of Perkins II.
Incorporating these changes into the federal vocational education or workforce preparation legislation will increase the efficacy of statewide systems of measures and standards. While the results of this study do not speak to all aspects of the legislation, they do suggest to us specific changes in the measures and standards provisions. The section in this document entitled "Rationale for Legislative Changes" presents our rationale for translating the findings of the study into changes in legislation. It elaborates on four main themes drawn from the results, and presents specific recommendations for revising the law. The next section, "Proposed 1995 Perkins Act Sections," contains a marked-up version of Perkins II showing where the changes might be made in the event of reauthorization. The paper concludes with a Technical Appendix that contrasts the new and old language along with a commentary on proposed changes.
We remind the reader that we have not attempted to rewrite the whole act or to draft omnibus legislation. We aim only to demonstrate how results of our research have practical value in informing future legislation. Many of our suggested revisions to Perkins II could as easily be imbedded in a consolidated workforce preparation bill.
[1] No formal attempt was made to determine whether states were in compliance with Perkins II or to judge the quality of the measures and standards states had chosen to implement.
[2] See Stecher et al. (1994) for a complete description of the findings.
[3] Some state vocational educators told us that Perkins II could be improved by adding ÒrealÓ incentives and sanctions, that is, by tying funding to program performance. For example, they said that Òjoint planningÓ is not an adequate penalty for continued failure to make substantial progress toward meeting standards. Termination of funding might be more appropriate. This issue created controversy at the time the law was written, and it continues to generate debate today. This study did not provide adequate evidence to support or reject such a recommendation.