|National Center for Research in Vocational Education|
University of California, Berkeley
MIKALA L. RAHN AND PATRICIA HOLMES
The passage of the 1990 Carl D. Perkins Vocational-Technical Act (Perkins II) required states to set up accountability systems for their vocational programs. As they grappled with the new mandates, their first concerns were centered around the law's basic requirements. At a minimum, they had to develop two measures in the accountability system. One of these measures had to be an indicator of learning and competency gains, including student achievement of basic or more advanced academic skills. The other measure could be one of the following four:
The first requirement posed the most challenges. Not only did states have to find ways to measure academic skills, but they also had to measure "gains" and distinguish between "basic" and "advanced" skills. For many, the task was daunting, and administrators faced many challenges:
The purpose of this CenterPoint is to provide a status report on issues related to accountability, including standards and assessment, data systems, and the role of incentives for schools, teachers, and students in improving student achievement. Our goal, using information gathered from the literature and from interviews with state administrators, is to provide a picture of the progress made to date and to point to future challenges.
With the recent passage of the Carl D. Perkins Vocational and Applied Technology Education Amendments of 1998 (Perkins III), looking backward to go forward becomes critically important. The new Act requires the implementation of a similar accountability system, however, with the potential for higher stakes associated with implementation in the form of incentive performance money. After eight years of implementing Perkins II, we found that states made dramatic progress toward meaningful accountability systems. The political context for measuring outcomes and commitment and sophistication of state administrators has improved over time. Despite this progress, however, states are far from developing complete accountability systems for vocational education or integrating these into the states' overall education reform agenda. Major challenges exist that require attention and support for state-level administrators in improving systems started under Perkins II into systems envisioned in the new Perkins III.
Progress of State Accountability Efforts in Vocational Education
The next sections provide a progress report on the key areas in state accountability systems:
In spite of some initial trepidation with the accountability requirements in Perkins II, each state developed its own system of performance measures and standards and devised some sort of implementation plan. At the secondary and postsecondary level, all but two states went well beyond the requirements of Perkins II and included more than two performance measures and standards. In fact, most states included three to ten measures (Hoachlander & Rahn, 1992; Rahn & Alt, 1994). These accountability systems were never intended to produce data that could be compared nationally. Instead, the purpose was to give state and local educators data they could use to improve vocational education programs and courses.
States built accountability systems suited to their own history and culture. For example, in more decentralized states, local education agencies were allowed to use their own assessments to measure academic gains and to develop their own instruments on other measures. While in these "local control" states there was no comparability and little standardization, there was more buy-in at the local level. In more centralized states, standardized systems could use statewide assessments that produced more comparability across local education agencies, but perhaps less perceived use for the system at the local level (Stecher, Hanser, Rahn et al., 1995; Rahn & Alt, 1994).
The usefulness of the Perkins II legislated performance measurement systems varies from state to state. In states that integrated the legislative requirements with their own initiatives for improving vocational education, the systems continue to improve. But in states with more of a "compliance" mind set, the accountability systems have not evolved much over time. Administrators in these states will wait for new federal legislation that mandates change.
Fortunately, most state administrators realize that the demand for accountability from the public, politicians, and others is here to stay, and are working hard to figure out what data is reasonable to collect and how it can be used to improve reporting about students and programs. The concerns raised are no longer "Why do we have to measure academic skills?" but "How do we best measure academic and occupational skills and assist locals in their use of data for program improvement?" The next sections will provide a progress report on the key areas in state accountability systems.
The driving force behind standards-based, systemic reform is the public's demand for excellence for all students. However, reaching consensus around what defines excellence has been difficult from state to state, much less nationally. Once adopted, standards become the centerpiece of performance-based, systemic reform and accountability systems measuring progress in reform. The set of standards for student learning that each state has or is adopting has been colored by its own particular political process and local educational traditions. As the effort to establish standards for what students should know and be able to do has spread, a number of organizations have set about evaluating the quality of state standards.
Making Standards Matter, first released in the summer of 1995, is the American Federation of Teacher's (AFT's) effort to compare the quality of academic standards from state to state. According to the AFT, although "other reports have been produced over the last year or two discussing standards-based reforms in the states, our report is the only one we know of that analyzes the quality of the academic standards in every state." To meet the AFT's criteria, states must have or be developing standards in English/language arts, math, science, and social studies and be clear and specific enough to provide the basis for a common core curriculum across the state.
According to the AFT, standards-based reforms will be effective as levers for educational reform only if the "standards are clear and specific enough to guide what we [teachers] do in the schools" and are "applied consistently so that no students get left behind." Standards "should be the glue that holds the various components of the educational system together." Ideally, standards are aligned to the state's student assessment systems and students receive additional help and are given incentives to reach the standards.
Among the AFT's major findings in its 1997 report is that the states' commitment to standards-based reform remains strong. In fact, all states but one are developing common academic standards for their students. While the quality has improved since the 1996 report, most states "still need to improve some of their standards in order to provide the basis for a common core learning," especially in English and social studies. The AFT considers some as models for other states, most notably Virginia's standards, which the AFT judged as "exemplary" in all four core subjects.
A companion volume to Making Standards Matter is the Coucil for Basic Education's (CBE's) Great Expectations? (Joftus & Berman, 1998). It responds to the recommendation that states bolster the rigor and international competitiveness of their standards. The study was framed to answer the question "Are states setting high standards for student learning?"
CBE began with math and English language arts and next year will examine the rigor of science and social studies standards. According to CBE, all states but two, Iowa and Wyoming, have developed or are developing state standards. Because CBE only reviewed drafts of standards that would be approved with no significant changes by January 1, 1998, several states were not included in its analysis.
To arrive at their findings, CBE developed an evaluation process with the help of two advisory panels (one for each subject area) made up of subject specialists, teachers, and parent and business representatives. The panels worked with CBE to develop a definition of rigor in standards and to approve the frameworks and rubrics CBE used in the analysis. CBE developed model standards against which states were analyzed (81 standards for math and 62 for English language arts).
CBE concluded that while states have begun to set high expectations for what students should know and be able to do, the rigor of these standards varies considerably. In general, state math standards tend to be more rigorous than English language arts. States with low to moderate levels of rigor in math "tend to address most essential concepts and skills, but at a lower level of sophistication than states with very rigorous standards."
A third report, The State of State Standards, (Finn, Petrilli, & Vanourek, 1998) is a compilation and summary report by the Fordham Foundation analyzing the results of the organization's five separate studies of state standards in English, history, geography, math, and science. According to the report, the potential for educational improvement that begins by setting high academic standards is theoretically a critical component in nearly every state's efforts. "Schools, teachers, and students will be held accountable for reaching them. Real consequences, like the closure of a school, continued employment of a teacher, retention of a student, and salary of the principal might result from reaching--or not reaching--these state standards. Teacher training and certification, curriculum and textbook selection, and much else will follow from such standards. Thus, they have great potential for good or for harm. Much depends on their quality."
The Fordham Foundation's goal was to find out whether state standards are rigorous and clear; readily accessible to teachers, parents, and students: and likely to boost student achievement. While some states did well in a few subjects, "the main conclusion to be drawn from this ambitious project is that most of the states have a long way to go before their academic standards will be strong enough to bear the considerable burden now being placed on them. Their present weakness, in fact, is a grave threat to standards-based education reform."
While the report concluded that state standards "are bleaker than we had hoped," it also found "signs that American education is finally awakening to the need for rigorous standards, real assessments, and tough-minded accountability systems." Like the AFT's Making Standards Matter, the Fordham Foundation did find a few states with "excellent standards" that can be used as models for other states.
Readiness and Industry Skill Standards
The development of student standards has also been taking place on another front in recent years, with efforts to define skill standards in areas beyond traditional academic areas, such as general workplace readiness, industry core, and job-specific skills. For instance, Hawaii has adopted "work skills" standards, Michigan has developed model content standards for "career and employability skills," and Oklahoma has developed content standards for "hands-on career exploration."
The National Skill Standards Act of 1994 established a board to oversee the development and implementation of a voluntary national skill standards system, dividing the economy into broad industry clusters. During the past five years, six national skill standards pilot projects were funded by the US Department of Labor and another 16 funded by the US Department of Education. The goal was to organize stake holders and to develop industry-based skill standards.
In Standards: Making Them Useful and Workable for the Education Enterprise, Joan Wills (1997) evaluates the effectiveness of these efforts specifically for educators. For a skill standards system to take hold in this nation, she found that state educational systems must be integrally involved in the development of "integrated curriculum, constructing career pathways information systems, engaging the private sector in STW efforts, and issuing certificates of competencies." She recommended that states incorporate generic workplace skills into curriculum frameworks, include standards-driven criteria in the accreditation of secondary and postsecondary institutions, establish industry or occupational clusters for use with curriculum frameworks and career pathway information, and "establish a `single point of contact' panel for skill standards development."
For most states, this effort to develop industry skill standards and assessments has been disappointing. No new standards have been released since the original 22 pilot projects were funded. And because the original projects used varying definitions of "standard," few have resulted in any assessment mechanism.
However, states have learned from the process used to develop the standards. Many are adapting or adopting the relevant standards into their own systems and are developing standards in areas where skill standards still do not exist.
State Standards to Student Assessment Systems
What most of these reports forget to celebrate is that most states have standards. They are developing, analyzing, and implementing standards--a critical first step prior to assessment and curriculum development.
This is a huge step forward from where states began when Perkins II was enacted. Yet according to Making Standards Matter, 1997, the standards are not yet strong enough in English, math, science, and social studies to build a solid foundation for the assessments. In addition, while most states would like their standards to be internationally competitive, many lack the resources to assess their standards against that benchmark.
In its study The Full Measure, Report of the NASBE Study Group on Statewide Assessment Systems, the National Association of State Boards of Education (1997) found that the widely varied assessment systems reflect the states' individual priorities and unique context. Still, there are general characteristics "highly effective state assessment systems share." An effective system "is aligned with state standards; is designed to address specific goals and purposes; balances validity, reliability and efficiency; informs remediation and has consequences attached to some results; provides a framework for school and district initiatives; and has a clearly articulated relationship with national and international measures of student performance."
The study group also examined implementation issues for state-level policy makers and offered the following "lessons learned." These include having broad public engagement in the design and implementation of assessments, keeping the test development process open to public scrutiny, taking the time that is needed, and addressing critics while continuing to implement standards and assessment. The report also stressed the need for strong political leadership.
to Improve Programs
The issuing of education report cards and the reporting of student achievement data are also part of state accountability reforms in the late 1990s that in some states influenced the evolution of their Perkins II accountability system.
According to a recent SREB report, Linking Education Report Cards and Local School Improvement (Gaines, 1995), "school-by-school report cards are part of a move by states to shift the action from the state capital to the local classroom by spotlighting student progress, or lack of progress in every school." The earliest report cards included facts and figures about the state and school district such as student characteristics, finance issues, and student counts. Information about standardized test scores, performance on Advanced Placement examinations, and dropout and graduation rates are also included. Recently, states have begun reporting student performance and achievement based on standards adopted by the state or district.
Very few states have incorporated the performance of vocational education students into statewide, school-by-school report cards. Reports are aimed at measuring the performance of "all" students. States that have attempted to incorporate vocational education explicitly into the report card have reported out the performance of tech prep or non-college-bound students. Vocational educators are not necessarily comfortable with distinctions of college-bound and non-college- bound nor are they comfortable being left out. It is a difficult challenge that will require further exploration in Perkins III.
In addition, states that have school-by-school report cards are finding that simply publicizing the data isn't enough. The information also must be used to boost student performance and to help school staffs make decisions and develop improvement plans.
Some schools are much further along than others in using data-based evaluation tools. According to the SREB report, "most educators have little or no training in analyzing a variety of data to make judgments about the effectiveness of school programs." For education report cards and accountability reports to become useful in raising student achievement, it's clear that schools and teachers must receive more support in integrating their use in planning for school improvement, and being able to improve particular program performance (such as vocational education programs) by using general school report card data.
The accountability movement of:
The accountability movement of the 1970s was driven by the mandate of education "inputs." For example, states mandated education improvement by establishing requirements for school facilities, the age of textbooks, and credit requirements for high school graduation.
In the 1980s, the emphasis shifted toward improving the preparation and training of teachers and principals. States examined teacher licensure laws, required teachers to pass examinations (usually set at minimal levels), improved teaching evaluation systems, and introduced performance incentives for teachers. However, according to another SREB report, Accountability in the 1990s: Holding Schools Responsible for Student Achievement (Cornett & Gaines, 1977), the "plans that eventually emerged rewarded teachers for longevity or extra work--not student success--and most were under funded and eventually abandoned."
In the 1990s, accountability shifted again, this time to the school level. The shift reflected state policy makers' frustration with the modest improvements in student achievement during the 1980s and the realization that educational change and improvement is a complex process that must involve the whole school.
According to the SREB report, as policy makers passed responsibility to the school level, educators were essentially told: "Here is the standard of progress you must achieve and here is the flexibility you need to get there." In keeping with the emphasis on accountability at the school level, states have introduced sanctions and assistance programs for low performing schools in concert with policies to improve the quality of teacher education, licensing requirements, and continuing professional development.
Incentives for Students, Teachers, and Schools
Another critical element of standards-based accountability is . . . state level policiies that motivate students to reach the higher standards . . . What follows are examples of:
Another critical element of standards-based accountability is the importance of state level policies that motivate students to reach the higher standards. "If high standards and new assessments are going to make a difference in our schools, the results have to `count.'" In 1996, the AFT first warned that state efforts to implement standards-based reforms would be severely hampered by the lack of rewards, consequences, or interventions. In 1997, the AFT found that only seven states were "seeking to end social promotion by requiring students to meet the state standards before being promoted into certain grades (up from four last year)." Only 13 states required that high school graduates pass assessments based on 10th grade standards or higher, and only 13 required and funded intervention programs for low-performing students.
When Perkins II was passed, it did not require that incentives and consequences be included with performance measures and standards. At the time, the concept of developing an outcome-driven accountability system was too new, and many development and measurement issues remained to be solved. With more experience and political support, however, some states now include incentives and consequences in their accountability systems. While it is acknowledged that simply setting standards will not result in improved student achievement, there is yet no consensus on which incentives are most effective in motivating students. Thus, the targets of incentive programs vary. What follows are examples of incentives for students, teachers, and schools from three states.
Along with other reforms centered on standards, accountability, and assessment, in 1991 the Georgia Assembly began requiring all students to pass a new set of tests to receive a high school diploma.
The Georgia High School Graduation Tests differed from the previously required Basic Skills Test by including social studies and science as well as reading, writing, and mathematics. The new tests "include process and application skills as assessed in a range of academic content, and shall exceed minimum and essential skills by extending the assessments' range of difficulty," according to the Georgia Department of Education.
According to the AFT, Georgia is one of only 13 states requiring students to pass exams based on tenth grade standards or higher. Georgia is one of only 20 states with graduation exams linked to their standards, and is one of fewer than 10 states requiring students to meet standards in all four core subjects.
Reform efforts in North Carolina have focused on improving the quality of teachers by investing in recruitment and salaries, improving teacher education training programs by requiring professional accreditation, strengthening licensing requirements, and offering mentoring programs for beginning teachers. North Carolina also offers incentives for experienced teachers to receive advanced certification through the National Board for Professional Teaching Standards. For state-paid teachers with a clear license and a minimum of three years teaching experience in North Carolina, the state will:
According to a report prepared for the National Commission on Teaching and America's Future, Doing What Matters Most: Investing in Quality Teaching (Darling-Hammond, 1997), since introducing these changes, the North Carolina Department of Education says "North Carolina has posted among the largest student achievement gains in mathematics and reading of any state in the nation, now scoring well above the national average in 4th grade reading and mathematics, although it entered the 1990s near the bottom of the state rankings." The state boasts the largest number of National Board Certified Teachers and is "home to 207" of them.
The South Carolina School Incentive Reward Program began in 1984 with a complex accountability system that included merit pay for teachers and principals. According to an official from the state's department of education, South Carolina concluded that judging teacher merit was too difficult, and that the principal incentives were driven by "building a file that showed what they did, but did not prove quality," so merit pay was removed from the reward program. What did survive was the School Incentive Reward Program, which provides funds to schools with exceptional or improved student performance on two statewide assessments, one norm-referenced, the other criterion-referenced. Districts also receive incentive rewards if two thirds of their schools qualify.
To qualify for a reward, schools must post one-year growth or meet the standard for improvement on the state's tests. State- and nationally normed tests are used in grades 3-11 in reading, mathematics, language, writing, and science (not all in every grade). Ninety-eight percent of students must be included, and schools are awarded based on a formula that allocates schools to one of four different percentile ranges (95 or higher, 90-94, 26-89, or 6-25) or whether the school's gain has been equal to or greater than the 65th state percentile rank for three years.
In 1997-98, 291 out of 1015 schools received rewards ranging from $2,800 to $72,400 on a per student basis. District rewards are $2 per pupil. While the typical amount per school is relatively small, about $15-25,000, schools are free to use the money for any instructional program enhancement chosen by a local school improvement council, with the exception of salary supplements or replacement of district funds.
"The door is wide open, schools can purchase PE equipment, computers and software, furniture, decorative murals," says an official from the SC Department of Education. Schools also are given recognition by the superintendent and a flag signifying their award. The program has lasted because the money goes to the school as a community which decides how to best use it to motivate its own students.
Next Steps: Perkins III
With the potential for incentive funding under Perkins III, all systems will need fine-tuning.
We see four areas where more work needs to be done:
The conversation about standards, assessment, data collection and analysis has evolved since the passage of Perkins II in 1990. For the first few years, efforts in academic and vocational education were independent. But with an evolving national context and downsizing at the state level, recent efforts have been geared toward more integration or coordination at the state level.
The requirements for core indicators of performance in Perkins III requires at a minimum measures of each of the following:
States may use previously developed performance measures that meet the above requirements. However, with the potential for incentive funding under Perkins III, all systems will need fine-tuning, and in some cases more serious work. We see four areas where more work needs to be done:
Although most states have established some standards, most are a long way from standards-driven assessment in academic, vocational/technical, and work-readiness areas. States must work to create defensible, useful systems that provide the necessary data on student performance. In Perkins III "attainment" as oppose to "gains" is the operative word which may make this requirement more manageable for some states.
Although some states have included vocational education in their overall accountability systems, many continue to isolate "vocational" and "academic" reporting requirements. The result is duplication in data collection; multiple performance reports; incomplete information and, ultimately, frustrated locals burdened by multiple and conflicting requests and requirements. In Perkins III, not only is it necessary to at least coordinate with academic "side of the house" to implement the first requirement, but the Act encourages collaboration with workforce development in implementing the new Workforce Investment Act.
This push for vocational education to collaborate with many state and local delivery systems does create a tension in purpose. In most states, vocational education is trapped between two often-conflicting ends of the spectrum: workforce development and school reform. On the one hand, vocational educators are attempting to be included (and in a few cases lead) in mainstream school reform through contextual learning, applied methodology, and integrated curriculum. Vocational educators have become advocates for broad career knowledge and skills needed to improve students' mastery of academic standards and to prepare them all for both postsecondary education and careers.
On the other hand, vocational educators are also working with job-training and new welfare-to-work initiatives to help students obtain specific skills for entry-level work. This training is more specific in focus and often shorter in term. These tensions exist in vocational education at both the secondary and postsecondary level and often complicate measurement systems, especially when measuring academic and occupational attainment.
More research and development needs to be done in the area of incentives and consequences. State efforts to implement standards-based reforms can be severely hampered by the lack of rewards, consequences, or interventions based on performance. The question becomes "What are appropriate incentives and consequences given the specific purposes on individual state systems?" Perkins III will provide an opportunity to experiment with one type of incentive through yet to be determined additional incentive money to states from the federal government.
These are not simple issues, and they will require experimentation and fine-tuning to achieve desired results.
American Federation of Teachers. (1997). Making standards matter. Washington, DC: Author.
Cornett, L. M., & Gaines, G. F. (1997). Accountability in the 1990s: Holding schools responsible for student achievement. Atlanta: Southern Regional Education Board.
Council of Chief State School Officers. (1996). Key state education policies on K-12 education: Content standards, graduation, teacher licensure, time, and attendance. Washington, DC: Author.
Council of Chief State School Officers. (1997). State education accountability reports and indicator reports: Status of reports across the states, 1997. Washington, DC: Author.
Council of Chief State School Officers. (1997). Status report: State systemic education improvements. Washington, DC: Author.
Darling-Hammond, L. (1997). Doing what matters most: Investing in quality teaching. New York: National Commission on Teaching and America's Future.
Education Commission of the States and National School-to-Work Office. (1998). 1997 profiles in connecting learning and work: State initiatives. Denver, CO: Author.
Finn, C. E., Jr., Petrilli, M. J., & Vanourek, G. (1998). The state of state standards (Fordham Report, Vol. 2, No. 5). Washington, DC: Thomas B. Fordham Foundation. Retrieved February 24, 1999 from the World Wide Web: http://edexcellence.net/standards/summary/summary.htm.
Gaines, G. (1995). Linking education report cards and local school improvement. Atlanta: Southern Regional Education Board.
Hoachlander, E. G., Levesque, K., & Rahn, M. L. (1992). State systems for accountability in vocational education (MDS-491). Berkeley: National Center for Research in Vocational Education, University of California.
Hoachlander, E. G., & Rahn, M. L. (1992). Performance measures and standards for vocational education: 1991 survey results (MDS-388). Berkeley: National Center for Research in Vocational Education, University of California.
Joftus, S., & Berman, I. (1998). Great expectations? Defining and assessing rigor in state standards for mathematics and English language arts. Washington, DC: Council for Basic Education.
Massell, D., Kirst, M., & Hoppe, M. (1997). Persistence and change: Standards-based systemic reform in nine states (CPRE Policy Brief RB-21-March 1997). Philadelphia: University of Pennsylvania, Graduate School of Education. Retrieved February 26, 1999 from the World Wide Web: http://www.negp.gov/reports/cprebrie.htm.
National Association of State Boards of Education. (1997). The full measure: Report of the NASBE Study Group on Statewide Assessment Systems. Alexandria, VA: Author.
National Association of State Boards of Education. (1998). Public accountability for student success, standards for education accountability systems: Report of the NASBE Study Group on Education Accountability. Alexandria, VA: Author.
Rahn, M. & Alt, M. (1994). Performance Measures and Standards (27-48). In Office of Educational Research and Improvement, National Assessment of Vocational Education: Interim report to Congress, vol. III. Washington, DC: U.S. Department of Education.
Stecher, B. M., Hanser, L. M., Hallmark, B., Rahn, M. L., Levesque, K., Hoachlander, E. G., Emanuel, D., & Klein, S. G. (1995). Improving Perkins II performance measures and standards: Lessons learned from early implementers in four states (MDS-732). Berkeley: National Center for Research in Vocational Education, University of California.
Stecher, B. M., Hanser, L. M., Rahn, M. L., Levesque, K., Klein, S. G., & Emanuel, D. (1995). Improving performance measures and standards for workforce education (MDS-821). Berkeley: National Center for Research in Vocational Education, University of California.
Watts, J. A., Gaines, G. F., & Creech, J. D. (1998). Getting results: A fresh look at accountability. Atlanta: Southern Regional Education Board.
Wills, J. (1997). Standards: Making them useful and workable for the education enterprise. Washington, DC: U.S. Department of Education. Retrieved March 8, 1999 from the World Wide Web: http://www.ed.gov/pubs/Standards/.
|centerpoint Number 4 March 1999|
|Program Management||Peter F. Seidman, Dissemination Program Director|
|Editorial Staff||Mary Carol Randall, Managing Editor
Susan White, Senior Editor
Marilyn Abbink, Copyeditor
David Carlson, Web Publishing
|This publication was published pursuant to a grant from the Office of Vocational |
and Adult Education, U.S. Department of Education, authorized by the
Carl D. Perkins Vocational Education Act.