Previous Next Title Page Contents NCRVE Home

3.

A GENERIC EXAMPLE OF LOCAL ACCOUNTABILITY IN VOCATIONAL EDUCATION: MIDAMERICA VOCATIONAL TECHNICAL SCHOOL



INTRODUCTION

        Our purpose in describing a generic vocational technical school is to illustrate the elements of the local accountability model described in the previous section. Schools are extremely complex organizations, involving diverse sets of participants, functions, facilities, regulations, and beneficiaries. Of necessity, this description will omit much that is not essential for describing the accountability components and relationships of interest, i.e., the goals, measures, feedback loops, and change mechanisms at work in the setting.

        MidAmerica Vocational Technical School (MAVTS or MidAmerica) is our own creation; to our knowledge there is no such school in the United States. We created MidAmerica to exemplify a "good" vocational technical school, one that serves its constituents well. MAVTS provides high-quality training programs for widely diverse students, and it appears to meet the needs of its constituents and the needs of the regional community for employment training. However, the school is not wholly imaginary; it represents a composite of programs, people, and practices we observed in vocational schools in the United States. Based on our observations, we are confident there are many institutions of equal or better quality in this country.

        The success of MidAmerica can be traced to many factors: a dedicated staff, an informed and supportive community, a cooperative state vocational education department, and a well-grounded curriculum. However, the feature that links these factors together and keeps MidAmerica "on target" is a rich, multilevel, local accountability network. Accountability at MidAmerica is based on a collection of simple components widely present in vocational education programs at the secondary and postsecondary levels. The accountability system is not a rigid set of data elements, reports, and processes, but a diverse collection of measures and communication paths through which all of MidAmerica's local constituents--employers, community members, students, instructors, administrative staff--make their needs and opinions known and in response to which program staff act to improve the programs offered by the school and to be responsive to the needs of students and employers.

        The description of the school will be divided into five parts: a one-paragraph "snapshot," and lengthier sections describing district-level features, school-level features, program-level features (including examples of two typical programs), and the relationship of the school to state and local constituents. Following these discussions, we will describe the local accountability system as it functions at MidAmerica. This description will cover goals, measures, feedback loops, and mechanisms for organizational change.

SNAPSHOT

        MidAmerica serves over 750 full- and part-time students; approximately one-half are high school students and one-half are adults. The school offers certificates of mastery in more than 30 occupational areas. All programs are competency-based and use the state occupational competency task lists as the basis for their curricula. Most programs are organized on an open-entry/open-exit basis. The school receives the bulk of its funding from local taxes but also receives substantial funding from the state and some funds from federal sources. Most of the buildings and facilities are over 30 years old, although many have been modernized as new programs were implemented. The Office of Vocational Education in the State Department of Education considers MidAmerica to be one of the better vocational technical schools in the state.

DISTRICT FEATURES

        As with many schools, MidAmerica is part of a school district that includes a network of elementary, secondary, and vocational schools. In this section, we describe several salient features of the school district that have either a direct or indirect effect on local accountability.

Governance

        MidAmerica is one of four area vocational technical schools operated by the Southwestern Regional Training District, a state educational entity that spans two counties. The district has the power to levy taxes to support secondary and adult vocational training. It also must comply with all state regulations governing secondary schools, including accreditation, licensing, and facilities.

        The district is run by a Board of Education, elected by the citizens. The board sets policies for the four vocational schools under its jurisdiction. The board appoints a superintendent to supervise the operation of the district. The current superintendent, Dr. M. Blanhorn, has been in office for seven years. Dr. Blanhorn recommends the appointment of principals for each school. The superintendent, principals, and all school staff are employees of the Southwestern District.

        Since taking the position of superintendent, Dr. Blanhorn has encouraged individual schools to be responsive to the needs of their local communities, and he has given each of the four school principals moderately wide latitude to implement the policies of the board in ways that are appropriate to its local service area.

Mission

        The mission of the Southwestern Regional Training District is contained in a formal statement, adopted initially by the board in 1985 and reviewed most recently in 1990. It commits the district to a number of specific purposes, including the following:

        The district defines its mission in terms of service to students and the community. Furthermore, it does so in ways that usually are measurable--information can be gathered to determine whether the district is achieving these purposes. The mission statement is shared with all staff and students, and it is a prominent part of the district's institutional culture. Furthermore, each school is required to adopt a set of goals that are consistent with the district's mission, and these goals are generally understood as the key elements that define each school's purpose.

Funding

        Approximately 60 percent of the operating costs of the district are raised by local ad valoremtaxation. Taxing authority must be voted on by the citizens in the region every other year, and the district has received continuing community support. The state provides another 30 percent of the school's revenue in the form of program-based funding and special grants. Federal vocational education funds distributed by the state account for an additional 7 percent of the budget. The remainder of the annual revenue is made up of student fees, Job Training Partnership Act (JTPA) program funds, and miscellaneous sources of revenue.

Services to Schools

        The district provides only limited administrative services for the schools, leaving most administrative functions in local hands. The centralized services include accounting, payroll, employee benefits administration, and legal support.

SCHOOL FEATURES

        Although much of the external context within which the school operates is set by the district, it is at the school level that philosophies and mission statements are translated into action. It is also at this level where the basic information that feeds the accountability system is collected. In this subsection, we briefly discuss the characteristics of the school that directly or indirectly affect accountability.

Administration

        The authority for operating the school resides with the principal, Dr. S. Monet. She is responsible for most school functions, including hiring and firing of staff, developing and administering the budget, and initiating and supervising programs. The superintendent intervenes only when he perceives there is a problem that is not being addressed adequately by the principal.

Goals

        MidAmerica translates the district's mission statement into specific programmatic goals:

Funding

        Although the bulk of MidAmerica's revenue comes from local taxes, the state provides approximately 30 percent of the school's total funding through the Office of Vocational Education (OVE) of the State Department of Education. The distribution of federal vocational education funds also is directed by OVE. State funding is provided on a per-program basis rather than a per-student basis; i.e., a fixed amount of funding is awarded to each authorized program with the intention that the program will serve at least 20 students. However, the funding level does not change if enrollment changes during the year. (Enrollment is one element that is considered in the state's program review, and continued low enrollment can lead to a cutoff in state funds. See below.)

Students

        MAVTS has an enrollment of 760 full- and part-time students (divided approximately equally between adults and concurrently enrolled high school students). Approximately 25 percent of the high school students and 15 percent of the adults are members of minority population groups, and 5 percent of the adults are women re-entering the workplace after many years. In addition, MidAmerica has a special program for deaf students, which draws students from throughout the district. Currently, 18 hearing-impaired students receive special services under the auspices of this program.

Centralized Support Services

        Most of the responsibility for instructional services is delegated to individual programs, but some common support services are provided centrally. These include outreach and recruitment, initial assessment and counseling, and the Learning Center.

        Outreach and Recruitment. The school has an active program of outreach to local high schools and the community. In addition, staff produce printed brochures and radio public-service announcements at the school and use these to inform potential students of the programs that are offered. Once each year, staff members make presentations at each secondary school in the region as part of a "career evening" sponsored by a local business association.

        Counseling and Initial Assessment. All prospective students receive an appointment with a career counselor. The counselor discusses the student's interests and reviews the opportunities available at the school. The counselor also arranges for the student to undergo an hour-long assessment of academic skills and vocational interests. This information is shared with students to help them make better choices and to alert them to special services that are available at the school, such as the Learning Center, which provides instruction in basic skills. The counseling center staff also can arrange for a student to visit classes for a brief period to familiarize the student with career options in that field and to assist the student in choosing among training opportunities.

        Learning Center. The Learning Center is a recent addition to MAVTS, created in response to the growing number of students entering the school with deficiencies in basic literacy or quantitative skills. The center provides specialized instruction in basic skills and coordinates the delivery of this instruction with the student's regular occupational program. Students who lack the basic skills necessary to complete their chosen program enroll in the program so long as they continue to work on basic skills in the Learning Center. The center teachers try to coordinate their instructional program with the occupational training the student is receiving.

        Industry Education. In recent years MidAmerica has actively promoted its retraining capabilities. The Industry Education Office helps individual programs develop, organize, and administer short-term courses designed to meet the needs of local businesses for upgrading the skills of existing workers. Often these courses are held at the job site. The Industry Education Office makes all necessary arrangements so programs do not have to worry about facilities, scheduling, enrollment, or credits. Unless funding is provided by other governmental sources, such as JTPA, the tuition is set at a level that permits the school to cover its costs in offering the course.

PROGRAM FEATURES

        Programs are the most important administrative units at MAVTS. Decisions regarding curriculum and instruction are made at the program level, and it is at the program level that the school has the greatest contact with students, parents, and local businesses. For example, feedback from businesses regarding the appropriateness of training content is received and acted upon primarily by programs. MAVTS has 10 departments that offer 34 different occupational training programs. For example, the graphic communications department offers programs in advertising art, offset printing, composition, and production art.

Organization and Scheduling

        All full-time programs are offered on an open-entry/open-exit basis, and, as a result, instruction is highly individualized. Concurrently enrolled high school students attend for three hours per day and earn academic credit. Adult students can attend for either three or six hours per day. For unlicensed occupations, approximately 600 course hours are required to complete competencies and earn a certificate issued by the school. For licensed occupations, such as cosmetology, state requirements for length of training apply, and students must pass a state examination to earn a license.

        Short-term Industry Education courses are not open-entry/open-exit; they have a fixed length, but they operate on an ad hoc schedule. Course lengths are determined in advance based on the content to be covered.

Competency-Based Curriculum

        All courses, both short-term and full-time, are competency-based. This means their curricula are divided into sequences of discrete elements, each consisting of knowledge to be learned and/or skills to be mastered. The state OVE supervised the development of skill lists for each occupation that are used as the basis for the curriculum. State skill lists are supplemented with additional skills suggested by local employers. Student progress can be monitored in terms of the number of competencies completed.

Measuring Student Performance

        Instructors are responsible for judging student performance. Each instructor determines the method of assessment to be used for each competency: written tests, coursework, or actual performances. Dr. Monet, the principal of MAVTS, encourages the use of hands-on performance assessment whenever appropriate. OVE also has developed pencil-and-paper mastery tests linked to the occupational skills lists; these are used by some instructors to measure competence. Programs maintain individual records and issue certificates of mastery when students complete all competencies required for an occupation. In addition, high school students receive course credit based on attendance and satisfactory performance, whether or not they achieve a certificate of mastery.

EXAMPLES OF PROGRAMS

        In this subsection we briefly describe two programs, one from the industrial/mechanical department and one from the cosmetology department.

Automobile Service Technology

        The automobile service technology program has been in existence since the school was opened, and it is one of the most well-established programs offered at MidAmerica. It has well-maintained, up-to-date equipment, and the instructors are well trained. In fact, the program holds the ASE Master Certification from the Automotive Service Excellence organization in six areas, and four instructors also hold individual Master Certification from ASE.

        The program comprises the following six elements: automotive core (shop safety, tool use and identification, engine parts identification and theory of internal combustion engines), automotive systems (electrical, electronic, brake, air conditioning, transmission, engine, suspension, and fuel), autotronics, front end and brake repair, transmissions, and air conditioning.

        Program graduates have found jobs as technicians in local automobile dealerships and garages or as self-employed automobile repair technicians. Postsecondary students can receive credit toward an associate degree through a cooperative agreement with Audubon Community College.

Cosmetology

        The cosmetology program began three years ago. The program operates a cosmetology salon on the school grounds that is open to residents of the community. The cosmetology curriculum includes hair styling, shampooing, permanent/finger waving, hair coloring, facials, scalp treatments, makeup, manicuring/pedicuring, and nail sculpturing. Each student learns through classroom instruction and has an opportunity to apply this knowledge to hands-on training in the cosmetology salon.

        The cosmetology program follows the regulations established by the State Board of Cosmetology. To qualify to take the state board examination, students must complete 1200 hours of training. Upon completion of the training, students must pass the state examination to obtain a license to work in the field.

        Cosmetology is a profession that is very individualized; a cosmetologist works independently in the development and formation of the business. Program graduates have found jobs in salons, department stores, and cosmetic firms as cosmetologists, hair stylists, nail technicians, makeup artists, or managers.

RELATIONSHIP OF MAVTS TO STATE AGENCIES AND LOCAL CONSTITUENTS

        The relationships that a vocational school has with governmental agencies and local constituents (i.e., students, parents, businesses) may be characterized as an exchange of resources and information. The State Department of Education (SDE) provides vocational education funds to the school districts and establishes regulations governing the use of the funds. School districts, in turn, must comply with the regulations and must provide reports back to the state documenting compliance and program effectiveness. In this subsection, we provide a flavor of these relationships for MAVTS.

State Department of Education and Other State Agencies

        The State Department of Education through its OVE provides a number of resources to regional vocational technical schools. The state also develops regulations that govern the operation of these schools.

        Program Development. The Office of Vocational Education provides a set of state-level job skill lists for each of the occupations covered by vocational programs in the state. Employers representing all regions of the state participated in the development of these skill lists to ensure that they were valid reflections of the demands of the occupations. These skills lists are available to any vocational program in the state to use as a basis for developing curriculum and certifying student preparation.

        OVE also develops competency-based assessment tools that correspond to the occupational skill lists. Most of these assessment materials are pencil-and-paper tests, but some are performance-based assessments. Again, these tests are available to vocational programs in the state to use as they see fit. (Initially, the tests were to be mandatory, but this requirement was dropped after significant objections were raised in the state legislature.)

        Program Review and Evaluation. Programs cannot receive state funding unless they have been authorized by the Office of Vocational Education, and this authorization must be renewed every two years. Reauthorization is based on continued labor market demand and on evaluations of program quality. If OVE determines that there is a need for training in a particular occupation and that the program is qualified to meet the need, it will authorize the program to provide one or more classes of instruction in the occupational area. Each authorized class receives a lump sum annual allocation of funds. The state assumes that, on average, an authorized program will serve the equivalent of 20 students per class.

        The OVE conducts biennial program reviews that are the basis for continued program authorization. The reviews focus primarily on three factors: the quality of the program, the level of demand for the occupation, and the success of the program in finding jobs in related fields for students who complete the program. The reviews also consider employer satisfaction and community support.

        Data Systems. The Office of Vocational Education conducts follow-up surveys of program graduates six months and one year after they finish their program. These surveys collect information on current employment and educational status, length of employment, and the relationship between employment and training. A supplemental survey attempts to measure employer satisfaction. These data are summarized and reported to the school annually.

        Coordination of Economic Development Programs. The State Department of Labor produces semi-annual labor market projections that schools can use to estimate demand for jobs. Unfortunately, the data represent statewide demand, and there is no attempt to estimate demand within particular regions. Over the years schools have become reasonably adept at making local projections based on the statewide data, and most continue to use the statewide data.

        Efforts are under way to create regional planning consortia, including all agencies and contractors providing occupational preparation and training services. This would include Private Industry Councils in each JTPA service delivery area, proprietary schools, adult education programs, community colleges, regional vocational technical schools, and volunteer literacy programs. To date the regional consortium in MidAmerica's region has met only once to "get acquainted," and it is too early to tell how useful this forum will be.

Local Employers

        MidAmerica has both formal and informal contacts with the local business community. Each program must have an industry advisory committee consisting of employers and representatives of trade associations and occupational groups. The advisory committees at MidAmerica have between eight and fifteen members. Committees meet at least twice a year to review curriculum, instructional equipment and materials, training approaches, and standards. Furthermore, Dr. Monet encourages programs to involve committee members in judging actual performances of students approaching program completion.

        Instructors are responsible for student placement (it is one of the criteria that are used to judge instructor performance), so they try to maintain good relations with local employers. Most instructors are on a first-name basis with the owners of small businesses in the area. In addition, they have good relationships with larger employers (sometimes with staff in the personnel office but more often with managers in related occupational departments). As a result, employers, particularly small businesses, often contact instructors at MAVTS directly when they have job openings.

        Instuctors also maintain direct contact with local businesses as part of their professional development. Each must spend 10 hours per term observing at local job sites. This on-the-job time helps instructors maintain their professional skills and their understanding of the needs of local employers.

        Employers also are involved in MidAmerica's Industry Education Office program. Industry Education provides short-term classes to upgrade the skills of currently employed workers. These programs are developed in conjunction with local employers and employee groups, who must help the school prepare the curriculum outline and develop the list of competencies. The number of Industry Education courses has been increasing rapidly over the past two or three years. As a result, there have been increasing contacts between the school and local businesses.

Students and Parents

        The opening of MidAmerica was accompanied by great fanfare, and it generated much local interest. Initial enrollments were high. However, after several years enrollments began to decline, and the school initiated outreach and recruitment efforts aimed at high school students and adults. These efforts were successful, and they have been continued to the present day.

        Most high school students learn about MidAmerica through presentations at their school, recommendations from counselors, or printed material distributed at the school. Most adults learn about the school from brochures distributed by state agencies and private counseling groups, advertisements on radio or billboards, or word of mouth from their friends. The outreach programs to high schools have been particularly successful, and there are often waiting lists for some programs. Adult enrollment is increasing as well.

        MidAmerica has limited contact with parents. Despite efforts to engage parents of high school students in discussions about career planning and training, very few parents actually go beyond the front door of the school.

ACCOUNTABILITY MECHANISMS

        It is easy to describe on paper the way the organizational units are supposed to work together to achieve the school's goals, but it is harder to make this happen in practice. Although Dr. Monet and the staff might not recognize it as such, MidAmerica has an effective local accountability system that helps them stay on target. All the elements of local accountability are present to some degree at MidAmerica--goals, measures, feedback loops, and change mechanisms. They are reflected in the way various decisions are made at the course, program, and institutional levels. The following material highlights specific accountability-related features at MidAmerica.

Goals

        Dr. Monet talks about her "expectations" regarding the activities each organizational unit must engage in so the school as a whole can achieve its goals. These expectations are really Dr. Monet's way of operationalizing goals for each unit, beginning with the administration itself.

        Administrative Goals. The school administration is expected to provide centralized individual student assessment services and training for students needing to improve basic skills. It also is expected to conduct outreach and recruitment activities in the community, making students (both full-time and short-term) aware of the opportunities the school provides. The institution is expected to support the programs in their efforts to build better links to local employers and the community, to act as channels for information and support provided by the Office of Vocational Education of the State Department of Education, and to provide adequate facilities to meet the needs of programs.

        Program Goals. Dr. Monet holds each program responsible for helping MidAmerica achieve its overall goals (see above). Although the school's goals are stated in general terms, they have specific implications at the program level. Each program is responsible for the following:

        These goals are operationalized further in specific course- and program-level objectives, including the following:

        In Dr. Monet's terms, programs are expected to demonstrate that they are meeting a real employment need in the region. They are expected to maintain lists of the skills and abilities needed to perform each job for which they offer training and to base their training on these occupational competencies. Occupational skill lists are expected to be updated regularly and to reflect both local employment conditions and industry standards for comparable positions. Programs are expected to align their curriculum with their skill lists, and they are expected to develop an assessment system to certify mastery of skills as students progress through courses. Finally, they are expected to help students find employment or further training in areas related to the program.

        Instructors in each program are responsible for meeting the program's expectations. They are expected to provide relevant learning opportunities for all students enrolled in their courses, and they are expected to coordinate their efforts with the school's Learning Center. Instructors are expected to build positive relationships with local employers and trade associations, both for the purpose of student placement and to upgrade their own skills to keep pace with changes in the industry. Perhaps most important, they are responsible for helping students locate employment.

        Goals for Students. From the institution's perspective, the most important student goals are the following:

        Specifically, students are expected to attend courses regularly, complete assignments in a timely manner, seek assistance in improving basic skills at the Learning Center if needed, and conduct themselves in an appropriate manner. They are expected to take responsibility for managing their education and employment preparation, monitoring their mastery of competencies, and calling the instructor's attention to their needs.

        In most cases, students hold the same goals for themselves as the institution does: to improve their job-related skills in preparation for meaningful employment and careers. However, not all students share these goals. For example, some secondary students are merely exploring career options; they are not interested in immediate employment upon completion of high school. Other secondary students are enrolled only to get course credit toward graduation; they are not motivated to learn the material in the courses nor to master occupational skills. Despite MidAmerica's efforts to educate schools and counselors about the purpose of the school and the characteristics of successful students, some schools continue to "dump" unmotivated, low-achieving students onto MAVTS without proper preparation or counseling. MidAmerica tries to accommodate the needs of students who are still exploring career options or whose goals are not immediate employment, but it does not see this as its chief mission.

        Similarly, there are some adult students (though the numbers are small) who enroll at MidAmerica for very different reasons. Some adults have multiple problems: they lack basic skills, they have gone through traumatic family or personal experiences, or they have been unsuccessful in attempts at work and they need counseling and assistance in a number of ways. They hope MidAmerica will solve all their problems, but it may not offer all the services they need. Finally, a few adults are enrolled because they are required to do so to continue receiving public assistance. MidAmerica recognizes that there are differences in adult students' goals, but it places its chief emphasis on the more traditional job-oriented adult.

        Industry Education Office programs are more narrowly focused on a specific set of skills. They assume that students are motivated to master skills that improve their standing at work, to qualify for new positions, or to keep abreast of changes in the workplace.

        Goals for Employers. MidAmerica anticipates that employers will:

        All employers are encouraged to participate in the program in some manner. Those who agree to serve on advisory committees are expected to meet regularly, to review the program curricula and skill lists, and to update them to match local needs and industry standards. Those who agree to help assess student performance are expected to attend review sessions and to observe, evaluate, and critique student work products and performances. Those who are interested in hiring students who complete the program are expected to provide feedback to the program about the performance of graduates.

        Employers' immediate goals are to find qualified employees to fill current openings. However, they also are concerned about the long-term needs of their industrial sector. They often look beyond the demands of one store or factory to the ongoing needs of an industry and are willing to help the school take this long-range perspective. However, employers are not always willing to put in the time in an advisory role that the school desires, nor do they always maintain a long-range perspective.

Measures

        MidAmerica's accountability network functions reasonably well because measures exist that are relevant to many of the goals. Although there is not a one-to-one match between goals and measures, there are considerable data that can be used as indicators of the school's success in meeting many of its goals. There remain some school goals that are not easily assessed, as well as some measures that do not relate to a goal deemed to be important to the school. (In the next section we discuss the problems that occur in these two situations.)

        Measures of Program Effectiveness. Programs collect a wide range of information related to course and program objectives in the areas of enrollment, completion, and placement.[18] Measures relating to student and employer goals are discussed in the following text. Aggregated student data provide the chief measures of program success in these areas. For example, each program can track course enrollments and completions and can generate estimates of the percentage of students attending and completing individual courses or course sequences. They also can summarize program success in terms of students' grades and can report on the percentage of students in a course who have mastered a particular skill or collection of skills.

        Programs try to keep records of initial placements in training-related jobs, but they lose contact with many students (for example, those who move out of the area) and the data have always been incomplete. The chief measure of post-training activity is the OVE follow-up survey described above.

        In addition to data on enrollment, completion, and placement, programs also keep track of information related to community contact and to program quality. Data are retained describing the number of inquiries each program receives from potential students and from employers looking for new employees. Programs also maintain anecdotal records of contacts instructors have with high schools, community groups, and other social service agencies. These activities are important for recruitment, placement, and maintaining a positive image in the community.

        The school does not collect any direct measures of instructor quality. Quality is inferred from student success, employer satisfaction, and participation in continuing education. The administration uses aggregated student performance data and employer satisfaction data as ways to judge overall program quality. The only direct measures of an instructor's behavior are reports on employer visits designed to maintain an instructor's own skills and knowledge of the workplace.

        Measures of Student Progress. A variety of data is available to measure student progress, and most are directly relevant to the program's goals for students--attainment of academic and occupational skills and employment in a training-related job (or enrollment in additional training).

        Each instructor is responsible for tracking student performance against the course skills list and for certifying students' mastery of individual skills. Instructors are expected to match the method of assessment to the nature of the skill. Thus, content knowledge can be assessed using paper-and-pencil tests, but procedural knowledge should be assessed through performances. Ultimately instructors are responsible for determining whether student performance on a skill is adequate. Some programs have opted for commercially developed standardized performance assessment, but most use their own tasks and examinations. Some programs use the state competency tests as "end of program" tests to confirm skill mastery. Instructors also are encouraged to involve local employers in judging large-scale student projects/activities, and this is done quite often in some programs.

        Each program maintains files on individual students containing information about their progress in individual courses and in the program as a whole. The files include information about attendance, course grades, and occupational skills mastered. The Learning Center also maintains individual records on student attendance, skill attainment, and overall achievement in basic skills.

        Monitoring the occupational status of students after they leave the program is a more difficult task. Instructors are asked to keep track of the initial employment of all students who complete a program, and they are able to do this for the majority of students. However, they have little or no information about those who leave before completing the program, and data on initial employment are seldom verified.

        The main source of information about employment or continued training is follow-up surveys conducted by the state Office of Vocational Education. The school supplies OVE with information about students who complete the program and OVE tracks their employment and educational status at six and twelve months after completion. The results of these surveys are shared with the school, and they are the major source of data about student employment and training. OVE surveys collect information about current employment status, relationship of training to employment, additional training received, and satisfaction with training.

        Measures of Employer Participation and Satisfaction. The most direct measure of employer satisfaction is the employer follow-up survey conducted by OVE as part of its student follow-up survey efforts. Employer satisfaction also is measured indirectly in terms of continuing participation in program activities. Local industries are asked to participate on industry advisory committees, to participate in judging student performance, to provide opportunities for cooperative work while students continue their training, and to employ program graduates. Each program keeps careful records of employer contacts in all four areas as well as any other ways local businesses are involved in supporting the programs, such as donating equipment, sponsoring events, and permitting tours of the workplace.

Feedback Loops

        Administrators and staff receive feedback about program performance from a variety of sources and in a variety of ways. Some feedback is formal (including the results of OVE student follow-up surveys, students' scores on competency tests, and reports of advisory committee program reviews). Some is informal, such as comments on students' abilities from businesses calling to look for prospective employees. Some feedback is internal to the system (e.g., data on student completion rates are generated as part of the ongoing administrative data system), and some comes from constituents (such as employers' reactions to course competencies and parents' opinions of the school's effectiveness). Finally, some feedback, such as course completion rates, is provided directly to program administrators, while other feedback, such as students' comments on course relevance or job competencies, comes directly to instructors and other staff. Overall, there is a rich network of information from a variety of sources that can be used for program evaluation and improvement.

        Feedback Regarding Program Effectiveness. The school receives a formal report from OVE after each biennial program review. Among other things, this report reviews the need for and performance of each occupational program and contains recommendations about the continuing authorization of the program. In addition, the Board of the Southwestern Regional Training District reviews the school's performance annually as part of its oversight responsibility and makes its opinions known through policies and directives as well as through its endorsement of the superintendent.

        Similarly, MidAmerica's Advisory Council conducts a formal annual review of the school's performance. The Advisory Council reviews each program area. Industry participants on the individual occupational advisory committees have direct input into curriculum, materials, and program planning.

        Informal feedback comes in a number of ways. The citizens of the district express their judgments regarding the value of the schools by voting for the members of the board and for the annual tax levy that supports the district.

        Another source of feedback on program effectiveness is changes in enrollments. The school has learned that one of its strongest avenues of recruitment is referrals from students and graduates. In general, there always has been positive "word of mouth," but discontent among students is reflected quickly in lower enrollments. Students' satisfaction also is evident from their enrollment and attendance patterns. Students attend the school voluntarily, and they "vote with their feet" if they are not satisfied.

        Finally, students also make their concerns known directly to their instructors. They are open about their successes and frustrations, and they provide feedback on everything from examinations to placements to specific occupational competencies.

        Feedback Regarding Student Progress. Student performance on within-course measures and tests and their progress in mastering the occupational competencies provide the most direct feedback about their progress. Employers also provide information about student competence. Employers express their opinions most directly through their hiring decisions and their informal contacts with instructors when they are seeking potential new employees. In addition, employers who are involved in end-of-program student assessment make their judgments of the quality of student performance known directly to instructors and students.

        The State Department of Education also provides important feedback regarding employment status based on its semi-annual follow-up surveys of students who complete the program. These reports include information on the employment and training status of program graduates, the relevance of their training to their employment, and the satisfaction of employers.

        Feedback Regarding Employer Satisfaction. The state follow-up survey provides some direct evidence of employer satisfaction with student graduates. However, much more useful and immediate feedback comes from employers themselves. Employers let instructors know directly if they are dissatisfied with the capabilities of students who complete the program. Furthermore, the industry advisory committees provide a formal channel for employer opinions. MidAmerica has learned that the willingness of local industries to continue to participate in school-related activities is a strong expression of their judgments about the value of the programs.

Change Mechanisms

        There are both formal and informal mechanisms for change at the course, program, and school levels. These change processes are most easily recounted in terms of program-related decisionmaking.

        School-Level Changes. There are a number of formal change mechanisms that include the "teeth" of the accountability system. One is the formal biennial review of the school and each of its programs by the Office of Vocational Education. OVE makes general recommendations for changes at the school and program levels, and in the event of a serious shortcoming, OVE can rescind authorization for a program. Loss of authorization would translate into immediate suspension of the program, a dramatic change that has never occurred at MidAmerica.

        Less drastic actions are taken to improve or change programs long before they reach the stage of state decertification. The district superintendent conducts an annual internal review of each school and its programs. Schools are expected to describe program accomplishments, concerns, and strategies for improvement.

        Within the school, there is an administrative council consisting of one representative from each program and one representative from each special service area that meets quarterly to discuss schoolwide administrative issues and recommend changes. However, most change occurs at the program level.

        Program-Level Changes. In response to the district's annual review, the school principal requires each program to conduct its own annual review. To increase the effectiveness of these reviews, the principal appoints a staff member from another program area to serve as a member of the program review team. As a part of the annual review, each industry advisory committee conducts a formal review of competencies, curriculum, materials, and student outcomes. The advisory committee makes recommendations for the changes it feels are necessary.

        Course-Level Changes. At the course level, instructors make ad hoc decisions about instructional strategies, course content, assessment methods, and course materials on an as-needed basis.

SUMMARY

        In this section we have described a notional, but we believe representative, vocational school. Our purpose has been to portray the information-rich nature of the environment in which vocational schools operate. At the same time, we hoped to point out that much of the information on which vocational schools operate is not easily captured in the summary figures and tables that are so often a mainstay of reporting requirements. For example, to say that each industry advisory committee met twice during the past year does not capture the flavor of involvement of local industry in program evaluation. Nonetheless, it is clear to us that local accountability systems, though often more informal than formal, exist widely in vocational education.

        In the next section, we discuss the practical limitations imposed on local accountability systems.

[18]The most important of the program goals are listed on page 23.

Previous Next Title Page Contents NCRVE Home
NCRVE Home | Site Search | Product Search