Stern, D., Finkelstein, N., Stone, J. R., III, Latting, J., & Dornsife, C. (1994). Research on School-to-Work Programs in the United States (MDS-771). Berkeley: National Center for Research in Vocational Education, University of California.

High Schools and Secondary Vocational Centers

Average Effects

The 1990 Perkins Amendments continue the traditional definition of vocational education as preparation for "occupations requiring other than a baccalaureate or advanced degree" (Section 521(41)). Since jobs that do require bachelor's or advanced degrees on average pay more than those that do not, this definition inevitably means that vocational education prepares people for relatively low-paying occupations. The 1991 Current Population Survey on sources of work-related training found that "about 44 percent of the workers who studied in high school vocational programs to qualify for their jobs were in administrative support occupations"--mainly secretaries, receptionists, bookkeepers, typists and clerks--"and 17 percent were in precision production, craft, and repair positions"--mechanics, welders, and electricians (U.S. Department of Labor, Bureau of Labor Statistics, 1992, p. 14). Accordingly, statistical analysis of earnings finds that individuals who qualified for their jobs by means of vocational education at the secondary level do not earn significantly more than individuals who reported that their jobs had no training requirement at all (Bowers & Swaim, 1992, Table 7).

The question for evaluations of secondary vocational education, therefore, has not been whether graduates of these programs do better in the labor market than the general population. Instead, evaluations have focused on whether students in these programs experience more success than other high school students who do not go to college. Until the mid-1980s, most studies of high school vocational education found no significant effect on students' subsequent employment or earnings, except for girls who studied office occupations (National Institute of Education, 1981; Psacharopoulos, 1987). These studies compared vocational graduates with other high school graduates who did not attend college, controlling for various measures of student background characteristics. The designation of vocational students often depended on self-descriptions by the students themselves.

However, in the mid-1980s, researchers made two improvements. First, they used data from students' transcripts, instead of self-descriptions, to identify vocational students. Second, they discovered an important intervening variable: whether or not students became employed in the field for which they had trained. Using the NLSY data, Rumberger and Daymont (1984) found that boys who had studied trade and industry subjects and girls who had studied for office occupations faced significantly better prospects in the labor market after high school if they in fact found jobs in their fields of study.

Campbell, Elliot, Laughlin, and Seusy (1987) replicated this result with HSB data. They found that vocational education participants who obtained a job in an occupation matching their field of training spent approximately 20% more time in the labor market than a comparison group of general track students and that their unemployment rate was three percentage points lower. In contrast, vocational participants who worked outside their field of training saw no advantage over the general track comparison group. High school students who concentrated in a particular vocational field and obtained related employment earned seven to eight percent more than vocational students who found employment in unrelated fields or students who pursued a general program in high school. Vocational education participants who spent 100% of their time working in related employment after graduation earned 31% more than vocationally prepared students who took non-training related employment.

The importance of finding jobs related to field of training has also been demonstrated with secondary data from Hong Kong (Chung, 1990) and Israel (Neuman & Ziderman, 1991). Similar results with postsecondary data from the U.S. are mentioned later. Along a similar line, Stevens (1993) has found that recent vocational graduates do best if they remain with the same employer for whom they worked while in school.

Still, even using these better methods, not all studies find positive effects of vocational education. Farkas, Hotchkiss, and Stromsdorfer (1989) analyzed data from the second HSB follow-up, using the transcript classification scheme devised by Campbell et al. (1987) to define vocational concentrators. They found no effect of vocational education concentration, except for males who entered business and office jobs and for females who entered trade and industrial jobs. They concluded that there may be some training advantage for those pursuing jobs that are not traditional for their gender.

Given the evidence that placement in training-related jobs seems to improve the employment prospects of vocational graduates in the first few years after high school, it seems likely that the immediate payoff from vocational education would be higher on average if more students did in fact obtain jobs related to their training. Yet, Bishop (1989) notes that the proportion of vocational graduates who do find training-related placements ranges from a low of 17% in some studies of trade and industrial graduates to a high of 52% in some studies of health education graduates. He contrasts this with surveys finding that two-thirds of German apprentices obtained jobs related to their training.

If the only purpose of secondary vocational education were to improve students' employment and earnings immediately after high school, then these findings would imply that schools should try to design vocational education programs so as to increase the number of training-related placements. However, the purposes of secondary vocational education presumably also include promoting students' economic viability over the course of their working lifetimes, helping them find work that provides non-monetary satisfactions and preparing them to be effective citizens as well as effective workers.

Campbell and Basinger (1985) studied some of these broader issues. They found that taking vocational courses or working in a training related job did not significantly affect students' subsequent job satisfaction. They also found that vocational students were less likely to participate in school organizations or non-school youth organizations than a comparable group of general track students. Controlling for social background and years of schooling, there were no differences in voting behavior, participation in political activity, or views of whether women should work.

Hotchkiss (1987) used HSB and NLSY data to examine four in-school outcomes--test scores, career expectations, grades, and attitudes--as well as five post-high school outcomes: two- or four-year college attendance, marriage and family, crime and substance use, and voting behavior. The effects of vocational curriculum on post-high school outcomes appeared to be small and inconsistent. There is evidence of a tendency for vocational concentrators to use drugs less than general students. Hotchkiss found no important effects of vocational curriculum on voting behaviors. With regard to in-school outcomes, he found that participation in vocational education was not detrimental, but the absence of an academic curriculum was.

Altonji (1992) analyzed the effects of secondary school course taking on students' subsequent education and labor market performance. Using the 1986 follow-up of the NLS72, he examined the effects of coursework in science, math, English, foreign language, social studies, fine arts, trades and industry, and business and marketing. An innovative statistical procedure was to use variation in course offerings among high schools to correct for bias resulting from students with unobserved differences choosing or being assigned to different courses. He found that students who took more vocational courses were earning higher wages fourteen years after graduation, but they also had finished fewer years of postsecondary education. This indicates that vocational programs in the early 1970s were preparing students for work but not for further education.

Conversely, Campbell et al. (1987) found that students who pursued an academic curriculum in high school but did not attend college earned less than those who pursued a general program. This indicates that academic studies prepared students for further education but not for work.

To summarize, high school vocational education has been found to have positive effects on earnings if students take a coherent sequence of courses and find jobs related to their training. Bishop (1994) believes the positive effects were greater in the 1980s than in the 1970s. However, only a small minority of high school students do complete a coherent sequence of vocational courses then go on to related employment. Furthermore, students who take more vocational courses in high school also complete less further education beyond high school.

Altonji's (1992) is the first study that has tried to control for selection bias in measuring effects of secondary vocational education. Previous studies have not attempted to allow for the possibility that unmeasured characteristics of students affect both their likelihood of enrolling in vocational education and their subsequent performance in further education and the labor market. It is conceivable that some students are just cut out for the kinds of work they can get with only a high school diploma, while others have a comparative advantage in more intellectual pursuits. Such pre-existing differences could explain the finding that vocational students do better in the labor market when they find jobs related to their training.

The possibility of pre-existing differences that make some young people naturally college-bound and others naturally work-bound has implications not only for research, but also, more importantly, for policy. It would imply that it is most efficient to divide students into different kinds of programs according to their natural inclinations. However, there is no known empirical method that can reliably determine at, say, age fourteen or sixteen, which students can potentially benefit most from further education. The most obvious indicator would be prior performance in school, but this is strongly correlated with socioeconomic and demographic variables, and disentangling these from the inclinations and talents of individual students is an empirically intractable task. Therefore, current policy, as given in the 1990 Perkins Amendments, seeks to keep the postsecondary option open for all students through the integration of vocational and academic education. The next section examines evidence on the effectiveness of this approach.

Programs Integrating Vocational and Academic Education.

The integration of vocational and academic education has been interpreted broadly by practitioners, researchers, and policymakers. Grubb, Davis, Plihal, and Lum (1991) have described eight models that capture the broad range of approaches that are being used in a variety of schools. The models range from the integrated curriculum of one course to occupational clusters and career academies on the other. Orr (1987), in a discussion of dropout prevention, advocates six different models that encourage the student's retention partially through integrated education and a connection to work. In 1988, the previous NAVE published a review of seven exemplary vocational education programs that describe a wide range of integrated curricula and educational approaches (Hayward, Adelman, & Apling, 1988)

The Southern Regional Education Board (SREB) promotes the integration of vocational and academic education in the context of workplace preparation. Student performance in eight of the original sites improved, with the greatest gains in the National Assessment of Educational Progress made by those students who followed an integrated vocational and academic course of study (Harp, 1993). As of 1993, SREB was expanding this program to three hundred sites.

Across the board, researchers and practitioners have commented on the incremental approach that integration takes as it is practically applied in the school site. Further, the variation of integration efforts as they are practically seen directly relate to the wide range of evaluation methods that have been seen as well. We should not expect similar methodologies or similar results from programs that vary so widely and therefore should be looking not for one but for a range of methodologies that can be used to evaluate the effectiveness of programs. Figure 4 lists the programs that will be reviewed here, and summarizes the evaluation process that was used. Most of these programs began before the 1990 Perkins Amendments, but they nevertheless illustrate the integration of vocational and academic education as required by those amendments.

Principes of Technology

This is an applied physics course developed by the Agency for Instructional Technology (AIT) and the Center for Occupational Research and Development (CORD). It has been widely adopted by teachers for integration efforts. AIT and CORD have evaluated the Principles of Technology program using a pre-/posttest analysis during a pilot phase (AIT, 1986b; AIT & CORD, 1984), and case studies of schools where the courses have been added (AIT, 1986a). Stasz, Kaganoff, and Eden (1993) have reported on this and other evaluations of CORD materials. In each of five studies, generally positive outcomes are reported, although limitations in evaluation methods and research designs are noted.

For the pilot phase, the curriculum for the Principles of Technology course was divided into units, each of which underwent its own testing. In all cases, the tests were developed and refined by content experts at CORD. In addition, student attitude questionnaires and teacher questionnaires were distributed. The results distilled those areas of the curriculum with which students had the most difficulty, usually questions and manipulations using mathematics. Further, the structure of the test allowed for examination of progress based on gender, grade level, and the class schedule and the specialty area of the instructor.

CORD has reported on the limitations of the research methodologies in place. There is no matched control group in this study of students taking a similar course that is readily comparable to the Principles of Technology curriculum. The authors report that the administrative and fiscal costs involved in developing matched control groups would have been significant. But they also note that the matching process itself would have been difficult because of the composite vocational and academic approach in the Principles of Technology curriculum.

Another methodological issue is the use of the same test for pre- and posttest examinations, about one month apart. Although students were never given answers to the pretest, they may have been sensitized by the questions to focus on certain concepts during the course. In addition, there was variation in laboratory equipment, teaching patterns, and student characteristics. There has been no attempt to control for this variability.

The results showed that significant learning gains did occur and they also identified areas where improvement might take place. These conclusions were drawn from the combined qualitative and quantitative assessments. Pre- and posttest scores on the Principles of Technology curriculum have been reported by CORD. The difference between the overall pretest mean of 11.47 and the overall posttest mean of 24.4 was statistically significant at the .01 level using a paired t-test. Results for each curricular unit are listed in Table 12.

In addition to the testing and the case studies that AIT and CORD use as evidence of their program's merits, it should be noted that the design of the CORD curriculum is also relevant. A consortium of thirty-three state and provincial education agencies from across the United States and Canada developed the curriculum that is used in Principles of Technology. Therefore, if we believe that expert opinion is relevant in the development of curricula that integrate vocational and academic skills, the development of this particular program appears to have drawn on a wide range of professional experience.

Even with this type of preparation, barriers to implementation still exist. Bodilly, Ramsey, Stasz, and Eden (1992) examined the use of CORD materials in selected school sites as part of an investigation of efforts to integrate vocational and academic curricula. In Ohio, for example, the State Department of Education encouraged schools to increase the teaching of Applied Academics and invested in CORD materials to this end. Teachers reported that the materials did not provide occupationally specific examples and that overall, the materials were too generic. Release time for teachers for the development of new packages as well as for the assimilation of these packages was recommended.

Project COFFEE

A comprehensive program for dropout prevention was designed by the Cooperative Federation for Educational Experiences (COFFEE) in North Oxford, Massachusetts, in 1979 (see Appendix A). Since its inception, the program has been widely replicated in other dropout prevention efforts and in vocational education for reentry students. Project COFFEE has undergone rigorous scrutiny. It has been validated twice by the U.S. Department of Education's Joint Dissemination Review Panel and is a demonstration program of the National Diffusion Network.

Project COFFEE is an integrated program that emphasizes occupational and academic development. Other significant features include the use of full-time guidance counselors, an in-school suspension policy as a form of discipline, a pre-employment component to assist with job skills, and an active program in physical education. Occupational development relies heavily on simulated work experience in one of five training areas: computer maintenance, word processing, horticulture/agriculture, distributive education, and building/grounds maintenance. Students also engage in job shadowing and internships. School-based enterprises are used to develop technical and interpersonal skills. All sectors of the Project COFFEE program are designed to support and reaffirm this occupational emphasis (French River Education Center, Inc., JOBS, 1989).

Project COFFEE has been evaluated against three goals: (1) gains in language, reading, and math will be greater than national norms; (2) students will significantly lower their absentee rate; and (3) they will have higher employability rates compared to those who had not attended the program. Assessment was accomplished by testing with the California Achievement Test, an ongoing review of attendance figures, and employment figures based on personal interviews, respectively. In all three cases, Project COFFEE has reported statistically significant gains (Tables 13-16).

The first and third claims are supported by comparisons to national norms. The second claim about attendance is a comparison over time of Project COFFEE students and does not include a national comparison group. To gauge improvement in language, reading, and math, Project COFFEE students were compared to a "norm population chosen by McGraw-Hill to be representative of the typical high school student in the United States" (French River Education Center, Inc., JOBS, 1989, p. 26). The actual scores that are used for norm comparisons were not provided by Project COFFEE in their published report, but it does state that "If COFFEE students were to maintain their relative standing with respect to the test's norm population, students would have the same percentile ranking on both pre and post tests. However, [Table 14] shows that a large proportion of COFFEE students actually increase their percentile rankings showing that they are outperforming the norms" (p. 27), although in 1984-1985, a majority of students did not improve their relative scores in three of four subjects.

The claim that Project COFFEE students will have greater employment rates is supported by comparisons to national employment statistics for dropouts. As shown in Table 16, the comparison of Project COFFEE students working is made to U.S. Department of Labor reports showing 36% of all white dropouts are employed.

In addition to the measurement that has been made of Project COFFEE, the wide replication of the program is some evidence that practitioners find the program design to be effective. Six of the ten projects chosen for first year in-depth evaluation by OVAE's Cooperative Demonstration Program were modeled after Project COFFEE (Hayward, Tallmadge, & Leu, 1992). The validation of Project COFFEE, therefore, includes not only a set of evaluations from norm referenced tests, but also validation by program directors who have chosen to replicate some or all of the components of the project's design.

Dropout Prevention and Reentry Demonstration Projects in Vocational Education

Dropout Prevention and Reentry Demonstration Projects in Vocational Education were funded by the Cooperative Demonstration Program of the Carl D. Perkins Act. Hayward, Adelman, and Apling (1988) have prepared the first of three years of longitudinal evaluation of the ten demonstration sites. The programs cover a wide range of target students, but all integrate vocational and academic education as a central focus. Some have close partnerships with business and apprenticeship programs as well. While the evaluation procedures that have been used are notable, the results are, at this time, less significant than the authors would have expected. In general, this is attributed to the newness of the programs. The researchers report that in subsequent years, using the same research design, they expect increases in program success to meet statistical significance.

In assessing program effectiveness, two separate studies are being conducted. The first is a process evaluation that tries to understand the issues involved in replicating a model program. The second is an outcome evaluation to understand how students in these programs perform compared to a control group at each site. Seven used random assignment to form the control group. The three remaining test sites used matched comparison groups.

The evaluation of outcomes used questionnaires and student records to measure attitudes toward self and school, educational and occupational aspirations, academic performance, disciplinary events, attendance, dropout rates, and employability. Factor analysis identified five attitudinal factors: attitude toward teachers/teaching, expectations for the future, attitude toward counselors, perception of academic encouragement received, and sensitivity to classmates' disruptive behavior.

Three separate analytic techniques were used to estimate program effects: regular analysis of covariance, "reliability-corrected" covariance analysis (used when the experimental and control groups were not randomly assigned), and a measure of the "gap reduction" between students in the program and the school average. In many instances, these three methods gave different results; but in some sites, statistically significant changes were found indicating that students in the demonstration program were performing better than the control group. The authors note that significant changes were more often found in sites where the qualitative assessment of program implementation had also been favorable. The results are presented in Table 17, showing which analytical techniques, if any, yielded statistically significant differences.

New York City's Career Magnet Program

New York City's Career Magnet Program has been evaluated by random assignment, taking advantage of the lottery procedure that assigns ninth-grade students to educational options (Crain, Heebner, & Si, 1992). Career magnet programs are designed to combine vocational and academic education in order to make secondary education more relevant to the world of work, to expand postsecondary education options, and to enhance career-specific skills.

According to Crain et al. (1992), the experimental design met three criteria: (1) all students in the selection pool had the same likelihood of being admitted into the program; (2) selection was random; and (3) students who did participate in the career magnet program had a different educational program than the control group. While the structure of this evaluation is notable, the random assignment is part of the normal process of placing students in their high school program in New York City. Some students are selected for certain programs by certain schools through a complex quota system, but others are randomly assigned. Further, it is never clear to teachers, staff, or students whether a student is there as a result of the quota or because the student was selected randomly. This protects against differential treatment of students in the school. The quota system is based mainly on reading ability and guarantees that students at each ability level participate.

The evaluation had to take account of the range of programs from which students came and the variety of magnet programs. Incomplete records kept some students from being included in the final analysis. Tables 18-20 show results of analysis of variance between pools of lottery "winners" and lottery "losers." Table 18 shows outcomes for all career magnet programs in several types of schools; Table 19 gives results for career magnet programs that exist only in Total Career Magnet Schools; and Table 20 shows results from career magnet programs that exist in Comprehensive and Vocational Schools.

Tables 18-20 indicate that lottery winners are significantly more likely to enter high school than lottery losers, though there is also an offsetting tendency, not quite significant statistically, for students who do win admission to career magnet schools to drop out in larger numbers during ninth grade. Students with low reading scores prior to grade nine are more likely to pass the New York State Regents exam in math if they attend a career magnet school, despite the fact that the magnet school students are also more likely to show an increase in absenteeism. Students with average reading scores before grade nine increase their reading scores as much as 50% faster in career magnet schools than in regular schools. However, there is a tendency, though not statistically significant, for career magnet students with low initial reading scores to make less progress in reading than their counterparts in regular schools. On balance, the statistically significant results indicate that winning admission to a career magnet helps students with low initial reading scores to do better in math, students with high initial reading scores to do better in reading, and both groups to stay in school for the start of ninth grade.

Career Academies in California

Career academies in California started in 1981 enrolling students who had shown a combination of poor attendance, low grades, and inadequate course completion by the end of their ninth-grade year. Patterned after the Philadelphia Academies that had started in the late 1960s, the California career academies are schools-within-schools for grades ten through twelve that organize a complete high school curriculum around a career theme. Employers are involved as speakers, mentors, and providers of employment for students during summer and part-time employment during the school year. The first two career academies in California, called the Peninsula Academies, focused on electronics and computer-related occupations. Subsequent academies have been organized around health careers, graphic arts, finance, and a number of other career themes (Stern, Raby, & Dayton, 1992).

An evaluation was performed on ten sites that were funded by the state of California to replicate the Peninsula Academy model. There was a comparison group of non-academy students at each site, individually matched by gender, ethnicity, and performance in ninth grade; but students were not randomly assigned to the program and comparison groups. Results showed that in some, but not all, of the replication sites, academy students had better attendance, higher grades, and a greater likelihood of staying in school (Stern, Dayton, Paik, Weisberg, & Evans, 1988; Stern, Dayton, Paik, & Weisberg, 1989). The gains for each cohort of academy students were greater during their first year than in subsequent years; this does not mean that the first-year gains were subsequently eroded, only that they were not augmented.

Table 21 shows the cumulative results of academy participation on school performance as indicated by cumulative dropout rates for three cohorts of academy and comparison students. For those entering in Fall 1985 (the only cohort the evaluation followed through grade twelve), the higher graduation rate for academy students implies a social benefit that exceeds the incremental cost of the program (Stern et al., 1989).

In summary, the integration of vocational and academic education has been applied to a range of student abilities and interests. Students in some of these integrated programs have done better than comparison groups in reading, math, attendance, and persistence. While these positive school outcomes should contribute to students' career success, most of the evaluations reported here have not measured labor market outcomes. The exception is the evaluation of Project COFFEE, which did demonstrate a higher rate of employment among its graduates than among white high school dropouts nationwide. However, it would be useful to have more rigorous testing of the effects of integrated vocational and academic programs on students' actual transition to work.

Stern, D., Finkelstein, N., Stone, J. R., III, Latting, J., & Dornsife, C. (1994). Research on School-to-Work Programs in the United States (MDS-771). Berkeley: National Center for Research in Vocational Education, University of California.