2.
THE NATURE OF ACCOUNTABILITY IN EDUCATION |
- A CONCEPTUAL MODEL OF LOCAL ACCOUNTABILITY IN EDUCATION
- LIMITATIONS OF STATE- AND FEDERAL-LEVEL ACCOUNTABILITY SYSTEMS
Accountability in education refers to the practice of holding educational systems responsible for the quality of their products--students' knowledge, skills, behaviors, and attitudes. Accountability is neither a new idea nor a new practice in education. Kirst[8] provides a brief history of its origins, tracing the use of the concept in education to the "payment by results" system found in 19th century England. With this system, schools were paid according to the performance of their students on standardized exams.
A "payment by results" system has an internal mechanism that works to optimize performance. In its purest form, when teachers are paid directly by their students to achieve a particular result (e.g., piano skills, SAT scores, ballet virtuosity), the process operates in the following manner. If a teacher cannot satisfy students--because he or she either is not a good teacher or is teaching the wrong subject matter--the teacher fails to make a living and leaves the system. As a result, only those teachers who teach the desired subject matter to the satisfaction of their students remain, and the overall quality of the system is improved. This is a simple illustration of the mechanism by which an accountability system works to optimize performance.
In the United States, educational accountability has roots in "cost accounting," a process for quantifying learning outcomes and attaching costs to them. For example, just as a factory can have an objective production goal for manufacturing automobiles, a school can have objective "production" goals for education. Similarly, as one can determine the cost of producing an automobile, so one could determine the cost of "producing" a trained graduate.[9] With this information, it would be possible to compare the output efficiency among school systems or among the schools within a system. As we have come to apply it in education today, accountability reflects this much more complex notion.
A formal definition of accountability can be found in recent RAND work:[10]
Accountability describes a relationship between two parties in which four conditions apply: first, one party expects the other to perform a service or accomplish a goal; second, the party performing the activity accepts the legitimacy of the other's expectation; third, the party performing the activity derives some benefits from the relationship; and fourth, the party for whom the activity is performed has some capacity to affect the other's benefits.
This is a generic definition of accountability that can be applied at many levels. For example, the "parties" in an accountability relationship could be state and federal agencies, school districts and state agencies, schools and school districts, schools and teachers, or teachers and students. Educational systems have a large number of constituents, ranging from students, parents, and local community members to state administrators, child welfare agents, and federal policymakers, each with a specific interest in the success of the system. Each stands in some type of accountability relationship with schools. Which of these relationships have the greatest impact on education? Traditional policymaking focuses at the state level, but we will argue that local accountability networks deserve greater attention, especially in the field of vocational education.
A CONCEPTUAL MODEL OF LOCAL ACCOUNTABILITY IN EDUCATION
Levin[11] described an idealized educational accountability system as a dynamic, self-contained set of perceptions and responses:
An accountability system is a closed loop reflecting a chain of responses to perceived needs or demands; an activity or set of activities that emerges to fill those demands; outcomes that result from those activities; and feedback on outcomes to the source of the demands. The feedback may generate new demands or a regeneration of the old ones; in either case, the previous set of activities may be modified or remain intact; a new or altered set of activities may be modified or remain intact; a new or altered set of outcomes may be produced; and the loop is completed again with feedback to the source of the demands (p. 375).
Using Levin as a starting point, we elaborated a model for a local accountability system for vocational education that has four major components:
- Goals
(e.g., 90 percent of high school students will graduate);
- Measures--means
for assessing progress toward the goals (e.g., cost accounting, comprehensive
record keeping);
- A
feedback loop--to provide assessment information and constituent input back to
the system; and
- A systemic change mechanism--for reacting to feedback by changing the system, as appropriate.
We will argue that an effective accountability system must include each of these components in a viable form.
Figure 2.1 presents a diagram of our conceptual model of a local accountability system. It reflects both the theoretical perspectives presented above and the input we received from interviews with local constituents. Federal and state agencies are represented as external to the accountability system; that is, the parties in a local accountability system are the local school system and its immediate constituents (students, parents, and local business interests). This conceptual model provides a basis for examining accountability relationships in vocational education programs. To understand local accountability according to the model,
Figure 2.1--A Conceptual Model of Accountability
one must understand goals--how they are set, communicated, and changed; measures--how they relate to goals and how trustworthy they are; feedback loops--how information is exchanged and how constituents communicate their opinions and judgments; and change mechanisms--how the educational system reacts to feedback and how it acts to maximize goal attainment.
Both the individual components in the model and the interrelationships between components are far more complex than this simple language would suggest. For example, constituent feedback to the program includes not only comments but also actions. Students who are dissatisfied with their progress "vote with their feet" by opting to drop out before completing their program. Similarly, employers who are not satisfied with the quality of the school's output, i.e., the graduates they interview as prospective employees, refrain from calling the school for additional referrals. Both of these actions constitute feedback from constituents.
Although such feedback does not contain explicit information about the nature of program deficiencies, it constitutes a strong signal that deficiencies exist. Furthermore, a decline in student or employer participation ultimately has a negative effect on program funding (although the timing and precise mechanisms vary from state to state and locality to locality). Thus, this type of negative feedback leads directly to strong formal sanctions. In contrast, increased demand for classes or graduates leads to rewards in the form of program growth (within local funding constraints). These examples illustrate how rewards and sanctions are elements of constituent feedback.
There also are rewards and sanctions implicit in administrative feedback to constituents. Administrators who respond positively to input from constituents encourage further communication and nurture stronger collaborative relationships. In fact, administrators who react "correctly" to feedback by making appropriate program changes, but who do not respond encouragingly to constituents, may send unintended negative signals that suppress further communication. Nevertheless, the model provides a starting point for investigating accountability in specific programs and for thinking about the way such systems are affected by state and federal policy regarding performance and consequences.
LIMITATIONS OF STATE- AND FEDERAL-LEVEL ACCOUNTABILITY SYSTEMS
Most educational policymaking occurs at the state or federal level,[12] and, as a result, most formal accountability mechanisms focus on information that is highly aggregated. For example, for many years the Secretary of Education published a "Wall Chart" that displayed state-level data on educational resources and attainments. Similarly, Congress authorized the National Assessment of Educational Progress (NAEP) to conduct trial state assessments in 1990 and 1992 to provide achievement comparisons at the state level.[13] Formal evaluations of large-scale, federal, categorical aid programs[14] (e.g., Title I, Title III, and Title VII) drew conclusions at the national level that became significant parts of the policy debate on reauthorizing these programs, just as the National Assessment of Vocational Education produced national results that affected the reauthorization of the federal vocational education programs.[15]
Similarly, most research on accountability is focused on the needs of state and national policymakers. For example, in a companion study conducted under the auspices of the NCRVE, Hill et al.[16] examine the information needs of national policymakers and the networks through which these needs are met.
However, there is a potentially serious problem in basing policy on accountability measured at highly aggregated levels. Mechanisms that appear to support policymaking at state and national levels may result in policies that are inappropriate when implemented at the local level. A simple example of this would be policymaking based on statewide occupational demand data. State policies that require vocational schools to offer or to withdraw occupational training programs based on statewide occupational employment data may be insensitive to local variations in workforce needs. For example, shortages of medical support personnel such as nurses or respiratory technicians can be highly localized, so it is important to base decisions about program initiation or termination on conditions in specific areas rather than on average conditions statewide or nationwide.
Vocational education is particularly susceptible to ineffective, high-level policymaking because local conditions play such a large role in program planning and performance. Programs are planned with an eye toward local economic needs, and program performance is measured in terms of placements, which are found primarily in the local area.
Local business and industry representatives play a significant role in many aspects of vocational programs, giving each program a unique local profile. For example, local business and industry help industrial vocational programs define goals and determine curriculum. Employers serve as advisors to almost all occupational-specific training programs. In this capacity they help programs identify skill needs, target training, and provide direct links to employment. Furthermore, many vocational instructors have direct contact with employers on a regular basis. Instructors are required to maintain their occupational skills and to be aware of the evolving state of the art in their fields by spending time at local businesses. Instructors also interact with employers when they supervise students working in job settings as part of training programs.
Vocational programs are responsive to the business community because they depend on local business for placement. Employers review the output of vocational programs (in the form of potential new employees) and make judgments about its quality (in the form of hiring decisions). In all these ways local business and industry affect vocational education, and policymaking needs to be sensitive to these influences. We believe it is important to understand local accountability mechanisms and their relationship to more highly aggregated state policymaking.
A final reason to focus on local accountability systems is that such mechanisms are likely to lead to program improvement more easily than highly aggregated ones. As Levin[17] noted, an accountability system is more effective when linkages are tight and information is produced and shared freely. These conditions exist more often at the local level than at the state or federal level.
We conducted site visits to vocational programs in five states. These visits confirmed our initial impressions that local accountability systems exist, although they are often informal and unsystematic. Furthermore, the visits revealed a number of commonalities across sites.
In the following section we use the insights gained from our site visits to describe a local accountability network in a hypothetical vocational technical school. The purpose of this generic description is to illustrate the manner in which the abstract relationships from the model are acted out in a real setting. By fleshing out the model in this way, we both enliven the text and elaborate the relationships. This elaboration is useful in later sections when we attempt to describe some of the limitations of local accountability systems.
[8]M. W. Kirst, Accountability: Implications for State and Local Policymakers, Office of Educational Research and Improvement, U.S. Department of Education, Washington, D.C., July 1990.
[9]Ibid.
[10]P. T. Hill and J. J. Bonan, Decentralization and Accountability in Public Education, R-4066-MCF/IET, RAND, Santa Monica, 1991, p. 35.
[11]H. M. Levin, "A Conceptual Framework for Accountability in Education," School Review 82(3), May 1974, pp. 363-391.
[12]See L. M. McDonnell and W. N. Grubb, Education and Training for Work: The Policy Instruments and the Institutions, R-4026-NCRVE/UCB, RAND, Santa Monica, 1991.
[13]In fact, the sampling plan for the original National Assessment of Educational Progress was specifically designed to preclude reporting of results at the state or district level. See S. Messick, A. Beaton, F. Lord, National Assessment of Educational Progress Reconsidered: A New Design for a New Era, NAEP Report 83-1, National Assessment of Educational Progress, Educational Testing Service, March 1983.
[14]For example, P. Berman and M. W. McLaughlin, Federal Programs Supporting Educational Change, Vol. VIII: Implementing and Sustaining Innovations, R-1589/8-HEW, RAND, Santa Monica, 1978.
[15]J. G. Wirt, L. D. Muraskin, D. A. Goodwin, and R. H. Meyer, Final Report, Volume I, Summary of Findings and Recommendations, National Assessment of Vocational Education, U.S. Department of Education, Washington, D.C., July 1989.
[16]P. T. Hill, J. Harvey, and A. Praskac, Pandora's Box: Accountability and Performance Standards in Vocational Education, R-4271-NCRVE/UCB, RAND, Santa Monica, 1992.
[17]Levin, p. 375.