In this study we sought to examine accountability relationships in vocational education at the local level as a complement to earlier NCRVE-sponsored research on accountability at the federal level.[32] Our purposes were to investigate the nature of local accountability in vocational education and to examine the effectiveness of such local accountability systems.

        To this end we reviewed the literature on accountability in vocational education, and we conducted extensive interviews with constituents of vocational programs in five states. Based on these data we developed a model of accountability at the local level and a collection of anecdotes about limitations of the model in practice. Both the model and the practical limitations should be useful in future research on accountability and on the effects of changes in federal and state policy regarding vocational education.

        For example, as a result of this study we believe that accountability systems are impaired if the components--goals, measures, feedback, and change mechanisms--are out of balance. That is, if they differ dramatically in terms of emphasis, credibility, sophistication, and efficacy. Recent federal efforts to promote accountability in vocational education have focused primarily on measurement and, to a somewhat lesser extent, on change. Much less attention has been given to goals and goal setting. One consequence of this emphasis on measurement is that measures may begin to supplant goal setting rather than having goals drive the choice of measures. To what extent is this occurring? What are the goals that are implicit in the measures being adopted, and how do they differ from the goals constituents hold for vocational education? These are empirical questions that might not be asked unless one has a systemic perspective on accountability. The general model of local accountability described in this study provides such a perspective, and it can help researchers generate appropriate questions.

        We draw four main conclusions from this study:

• There are local accountability systems operating in vocational education programs that are distinct from state- or federally imposed standards and requirements for program review.
  
  
• These systems can be described reasonably well by a model with four components: goals, measures of goal attainment, feedback from constituents, and organizational change mechanisms.
  
  
• The quality of these components and the relationships between them account for much of the variance in local accountability.
  
  
• Practical constraints exist to limit the effectiveness of these systems.

        Understanding these limitations can lead to prescriptions for improving local accountability and to better understanding of the impact of state and federal policy.

        To elaborate on the first conclusion, we found it makes sense to talk about local accountability in vocational education as distinct from federally mandated accountability. This, in itself, is an important result, for many would argue that one of the problems facing general education is that schools are not accountable to students and parents. In vocational education the situation is different. The nature of the educational enterprise is such that one often finds meaningful local goals, clear signals of goal attainment, and additional local constituents with a vested interest in outcomes. All these create a framework for local accountability, and although this framework often is informal, it is systematic.

        Second, a simple model captures the important elements of this accountability system. Every school system we visited had formal goals. Although goals may differ for different stakeholders, if they are shared, they can be the basis for accountability. Next, there must be ways to measure and publicly communicate about attainment of goals. Most measures are defined in terms of outcomes, e.g., program completion, skills attained, and placements achieved. Feedback is the third component. Measures must be communicated to program staff and constituents, and the opinions and interpretations of constituents and staff must be communicated back to program administrators in some form--either directly through discussions or indirectly through actions. Finally, the system must have some mechanism for change in response to feedback. In most cases the responsibility falls on the shoulders of administrators, who may be more or less effective at translating feedback into program improvement.

        Third, the quality of these four components affects the capacity of the local accountability system to produce program improvement. Poorly stated or unrecognized goals reduce the focus of administrators. Measures that are not consonant with goals limit judgment about program effectiveness. Lack of feedback from constituents fails to portray adequately their needs. Ineffective change agents fail to use the information provided by the system to make necessary changes. Limits in these components interfere with the effectiveness of the overall accountability system.

        Fourth, it is possible to identify a number of the most common practical constraints that limit the effectiveness of goals, measures, feedback, and change mechanisms. For example, goals are often stated in ways that make it impossible to know if they have been achieved. Such goals provide little basis for judging progress while permitting schools to draw positive conclusions about their effectiveness. Even when schools operationalize goals well, they seldom assign priorities that would help guide resource allocation decisions. Similarly, when measures are lacking, people rely on subjective judgments about goal attainment, which can overly politicize the decisionmaking process. The use of measures of unknown or inadequate technical quality may lead to unwise program planning and reform.

        Feedback can be reduced or distorted in ways that constrain its effectiveness. For example, actions that are taken to enhance services (providing job placement specialists) can have the opposite effect by limiting communication between instructors and employers, a critical source of data on student mastery and program effectiveness. Focusing on high job placement rates can mask the fact that the graduates are earning poor wages. Similarly, educational change mechanisms can be seized by well-meaning administrators with strong visions, whose actions may or may not further the needs of their constituents.

        The identification of common limitations such as these in local accountability systems has at least two potential benefits: We can begin to identify criteria to use to evaluate local accountability systems. In addition, we also can develop prescriptions for improving local accountability.

        Furthermore, the process creates a vocabulary for examining the effects of state and federal programs on local systems. We saw ample evidence of the influence of the larger state and federal context on local accountability relationships. In some cases these pressures enhance the accountability system. In other cases they are at odds with the natural accountability relationships that exist at the local level. Our description of the local accountability systems serves as a basis for examining the effects of federal and state initiatives in this area. Exploring this topic is the subject of a subsequent report.

[32]Hill, Harvey, and Praskac, 1992.