NCRVE Home |
Site Search |
Product Search
The changes we propose flow directly from the research findings summarized in the previous section. In the case of performance measures and standards, the links between federal legislation and state actions are relatively clear and direct, and it is possible to trace implementation effects back to legislative causes. This linkage facilitates the task of revising the law to promote desired outcomes. We believe that the intent of the federal legislation was to promote effective program improvement, and that this can best be accomplished by including four major changes in future legislation:
- Coordinate separate components into a more integrated system for planning, implementing, monitoring, and improving vocational education and training.
- Increase the emphasis on the use of the system of performance measures and standards as a program improvement tool.
- Clarify and improve language describing the required measures and standards themselves.
- Increase the amount of technical assistance provided by state and federal agencies to support change at the local and state levels.
To illustrate how these changes could be imbedded in federal legislation, we revised selected portions of Sections 115, 116, and 117 of Perkins II- the legislation describing performance measures and standards and the requirements for local and state assessment and evaluation. The proposed revisions have been italicized. Data limitations prevented us from undertaking a complete redrafting of the law. We have, therefore, limited our efforts to only those sections where our earlier findings justify legislative reworking. Readers should not assume that we endorse all the non-italicized sections of the act; in most cases, these components are left unchanged because they were not informed by our research.
The rest of this section describes in narrative format the major changes we recommend. This approach communicates better the goals we were trying to achieve and the broad changes we made to achieve them. The complete text of the proposed revisions, with detailed commentary comparing the new law to the old, is contained in the Technical Appendix.
Our revisions attempt to coordinate the separate elements found in Perkins II into a more integrated system for planning, implementation, monitoring, and improving vocational programs. The logical model underlying this system is illustrated in Figure 1. Revised language clarifies the interrelationships between the elements of Perkins II, including state needs assessments, measures and standards, annual local evaluations, and program improvement plans. Our revisions also promote greater coordination of measures and standards with other federal workforce and education initiatives in the following ways:
- Performance measures and standards are conceptualized as one part of a larger, interconnected system for improving vocational education at the local and state levels. The system includes many components, all of which interact and inform one another. For example, the state needs assessment drives the system of measures and standards and the local application process. These, in turn, inform the annual local evaluation, which leads to program improvement planning, to periodic review of the system of measures and standards, and ultimately back to the state needs assessment. In this way the information comes full circle and helps to promote improvement of the whole system.
- To make performance measures and standards a more comprehensive program improvement tool, they are applied to all major Perkins instructional initiatives (including programs funded under Title III such as Tech Prep and consumer and home economics), not just to programs receiving funds under Title II.
- The Committee of Practitioners is required to review and revise the measures and standards annually, creating a self-improving system that is more responsive to changes in other vocational education policies.
In our research, we recognized the clear need for coordination among federal workforce preparation programs. As a result, our suggested revisions in this area are particularly relevant in the context of a consolidated bill. Coordination among federal workforce education programs is promoted in the following ways:
- The State Board and the Committee of Practitioners are directed to consider all federal training and workforce preparation efforts when developing measures and standards and the program improvement system. Representatives of the agencies responsible for administering related programs must be consulted in the appointment of the Committee of Practitioners.
- The State Board is directed to provide the Committee of Practitioners with information about other state assessment efforts that might be relevant to a comprehensive program improvement system.
- The federal government is directed to provide technical assistance to help states coordinate measures and standards with other federal initiatives.
Our proposed changes represent a significant change in focus away from the initial development of the systems of measures and standards called for in Perkins II toward the use of these systems for future program improvement. Our research revealed that while states had made significant progress in developing their systems of measures and standards, for the most part they had not yet tackled the next step of using their systems for program improvement. Without explicit provisions for the use of performance measures and standards, the data they generate may languish in government files instead of being used to improve programs. Although the theme of measures and standards as the basis for a system of program improvement runs throughout our proposed revisions, it is most evident in Section 118, which sets specific requirements for program evaluation and improvement.
Increased relevance and usefulness of the annual evaluations for program improvement is promoted in the following ways:
- The requirement for program evaluation is explicitly written to include all the relevant component programs of Perkins (e.g., Title III programs such as Tech Prep), not just those efforts funded under Title II. This change ensures that information about which components of Perkins are or are not working is available to assist in program improvement planning.
- Teachers and parents are explicitly included in the evaluation process because they are important stakeholders and can contribute to the improvement of programs.
- The performance of special populations enrolled in vocational programs is to be specifically compared to the performance of other vocational students to ensure that staff monitor the access and success of students from special populations in each program area.
Additionally, program improvement activities are promoted in the following ways:
- The outcome of the annual evaluation is tied explicitly to a plan of program improvement emphasizing a link that is not sufficiently clear in current legislation.
- To promote program improvement as a continuing and ongoing process, a requirement has been added for all recipients to develop a local improvement plan, regardless of whether standards have been met.
- A requirement that the local improvement plan include a summary of the results of the local annual evaluation strengthens and emphasizes the link between these activities.
- A requirement that those local recipients who fail to meet their standards describe "specific strategies for making substantial progress" encourages follow-through from evaluation to program improvement.
Measures and standards are the cornerstone of the Perkins program improvement system. Our revisions attempt to improve measures and standards in two ways: (1) by clarifying the requirements for such systems and (2) by improving the technical quality of the components. Clarification is achieved by adding definitions, reformulating descriptions, and reorganizing state options to reveal more of the existing similarities. Technical quality is enhanced by requiring that states pay more attention to questions of reliability, validity, and lack of bias, and by requiring that the federal government contribute to the solution of common measurement problems.
Clarification of systems of measures and standards is achieved in the following ways:
- Our revisions provide definitions for key terms that should be common throughout the country, including outcome, measure, standard, program, and special populations. Because of differing local contexts, we feel the definition of "substantial progress" ought to be a state prerogative, so the regulations require states to define this term in a measurable and consistent manner.
- The requirements for measures and standards are reformulated and reorganized to make them more logical and consistent. State systems must contain at least five measures (most already contain more than this), one each from the following categories: learning gains, work-related skills, retention or completion, placement, and access/equity. This ensures that programs are evaluated based upon a complete and balanced picture of their desired effects.
Technical quality is promoted in the following ways:
- States are required to revise measures and standards regularly, based on an examination of technical quality and usefulness. This ensures that important policy decisions are based on sound information.
- The federal government is required to contribute to the solution of common measurement problems such as evaluating the reliability, validity, and lack of bias in measures; measuring academic gains at the postsecondary level; and setting valid performance standards. These common problems, which were created by the legislation, are shared by all states and are most efficiently addressed nationally.
- To focus attention on the goal of program improvement and the difficult problems of measures and standards, the Secretary is required to report to Congress on the status of each state's system for improving vocational education and on the technical quality of the measures and standards that are adopted.
Several activities required under Perkins II were beyond the capacity of the local and state recipients to accomplish on their own. As a result, we have included a new Section 119, titled "Technical Assistance," that specifies actions the state and federal governments will take to assist recipients in carrying out their responsibilities for program improvement under reauthorized or consolidated legislation. We propose fundamentally different roles for the state and federal governments in the provision of technical assistance. Our distinction is based on those problems that we perceive as best solved nationally versus those that are primarily local and state in nature. The technical assistance role we propose for the federal government includes the following:
- Assistance to help states address measurement-related issues, including establishing the technical quality of measures (i.e., reliability, validity, and lack of bias). This category also includes assistance in developing acceptable assessment tools.
- Assistance with identifying, developing, and implementing program improvement strategies on a national level, in particular, helping to disseminate more broadly program improvement models and other lessons learned in individual states.
- Assistance with coordinating the measures and standards requirements with other evolving and emerging workforce enhancement initiatives, including industry skill standards and School-to-Work performance standards. It is our belief that coordination will increase the effectiveness of these efforts.
The states have the essential role of assisting local recipients in evaluating and improving their programs. This role includes the following responsibilities:
- Assistance in evaluating the effectiveness of programs based on performance measures and standards.
- Assistance in identifying and adopting appropriate strategies for improving performance.
- Training local administrators and instructors in using performance data to improve vocational programs and courses.
- Dissemination to local recipients of examples of effective performance data use to improve vocational courses and programs.
NCRVE Home |
Site Search |
Product Search