In Section III.1 I present results for CETA and JTPA programs, the major job training programs. In Section III.2, I present results for welfare-to-work programs, while Section III.3 reviews several experimental programs. Section III.4 then describes what is known for different groups within the population, while Section III.5 does the same for different types of services. Section III.6 describes the results over time, to ascertain whether any benefits of job training programs persist or decay. Finally, Section III.7 presents some findings about variations among specific providers, because of findings of how much variation there is among different providers in their effectiveness.
The results in these sections present information about the effects of job training programs, regardless of cost. In addition, some (but by no means all) of these evaluations have carried out cost-benefit analyses, comparing the effects on earnings with costs of programs to see if they are "worth doing" not in the sense of increasing the employment and earnings of participants, but in the sense of generating benefits that outweigh costs. These cost-benefit analyses are surveyed in Section III.8.
[20] As mentioned above this is not a formal meta-evaluation; for a meta-evaluation, see Fischer (1995). See also the recent review of education and job training programs in What's Working (and What's Not) (Office of the Chief Economist, 1995).
A wide variety of job training program preceded CETA, because of the variety of funding sources and the great latitude local programs had. Partly for this reason, it is difficult to compare evaluations of different programs, because the services they offered and the intensity (or duration) or programs varied so much. In addition, the methodology of evaluation varied as well. The earliest evaluations tended to use conventional regression methods using quasi-experimental control and experimental groups: that is, a regression describing earnings (or the log of earnings) as a function of individual characteristics plus a variable describing program participation (or several variables, if data were available on the intensity of the program) would be estimated. Table 2 presents a summary of the early evaluations of job training programs, distinguishing the results for classroom training, on-the-job training, the Job Corps (a residential, year-long program for youth that is much more intensive than any other job training program), and adult basic education (ABE), a form of remedial reading and writing instruction. The results generally suggest positive effects, generally higher for females than for males, with a rough average of effects in the order of $250 to $300 per year. Given inflation between the early 1970s and the present, such benefits would be worth about $900 to $1,000 per year in 1994 dollars. However, because of unmeasured selection effects and problems with regression to the mean, mentioned above, such regression methods cannot possible control for the true differences between those enrolled in programs and others not-enrolled, so these estimates must be considered over-estimates of the true effects.
When manpower training programs were consolidated in the CETA program in 1973, the kinds of services offered and their administration became somewhat more standardized. In addition, the evaluation of these programs expanded substantially, through two different avenues: the Youth Knowledge Development Project, which generated a large number of qualitative studies of CETA programs; and the generation of the Continuous Longitudinal Manpower Survey (CLMS), which followed a random sample of CETA enrollees from 1975 on. Individuals in the CLMS survey were then matched to comparable individuals from another data set, the Current Population Survey, using different matching methods; then, as in earlier evaluations, regression methods were used to disentangle the effects of personal characteristics (e.g., gender, race, years of formal education, age, prior labor market experience, and the like) from the effects of program participation.
Table 3 presents the results of several studies using the CLMS data. The benefits of participation were generally higher for women; indeed, several studies found statistically insignificant effects for men. In general, the conclusion has been that CETA programs increased earnings for women, from about $500 to $1,000, though there is too much uncertainty to know whether there were increases for men. For youth, the effects were generally zero or even negative. In addition, a smattering of evidence suggests that classroom training and on-the-job training were more effective than work experience and public service employment, where individuals were employed in public service jobs at minimal wages; for example, Taggart (1981, p. 282) concluded that classroom training increased earnings in 1976 by $350 or 10%, on-the-job trainees gained $850 or 18 percent (declining to $600 the second year), and those in public service employment gained $250 the first year and $350 the second; individuals in work experience programs actually lost earnings. However, the various studies are too contradictory to be very sure about this result.
The different studies, varying in their analyses of the same basic data set, were most remarkable for the range of findings: for adult women, the different studies showed variation from no earnings gain at all to $1,300, while for men estimates ranged from negative $700 to $691. One of the most serious problems, aside from the ubiquitous selection effects, is nicely illustrated in Figure 1, from one of these CETA evaluations (Bloom and McLaughlin, 1982). The year before enrollment, CETA clients show a pronounced dip in earnings, compared to the control group; for men, job training simply restores them to the level of earnings of the control group, though for women there is a slight increase above these earnings of the control group. If the earnings dip is merely a transitory component -- for example, caused by bad luck, or a temporary spell of unemployment -- then it is clear that job training programs do nothing for men, though they have modest effects for women (about $800 - $1,300 in this particular study, higher than in most others). If, however, the earnings dip was something connected to a more permanent reduction in earnings capacity, then the benefits of CETA would be considered higher. Disagreement about which of these is true in partly responsible for the range of estimates in Table 3. One conventional conclusion from the range of estimates, therefore, was that quasi-experimental methods were not powerful enough to detect the possible benefits of job training programs, and that experimental methods were advisable (Barnow, 1986). But in part this conclusion resulted from the facts that if there had been any benefits, they must have been modest since very large benefits would have been detected even with quasi-experimental methods and with substantial variation among different studies.
A special program is the Job Corps, a particularly intensive program for youth. The Job Corps is predominantly a residential program in which youth live in a center away from home, and receive a variety of academic instruction, job training, and various other social services for an entire year. It has always been the most expensive job training program, costing about $15,000 in 1994 dollars, and has represented the most serious kind of intervention for those youth judged in the greatest need. As the results in Table 4 indicate, based on a quasi-experimental evaluation using a matched comparison group, it has had positive effects on employment rates and on overall earnings, though not on wage rates, and it has also served to reduce crime among those enrolled. (The findings that wage rates have not increased suggest that the Job Corps did not increase the productivity of its members; but increased employment led to higher earnings, indicating that the Job Corps, like many other job training programs, affected persistence in the labor force instead.) Furthermore, benefit-cost analyses have shown that, despite the high costs of Job Corps, the values of these benefits outweigh the costs (Table 20). These results, which represented some of the earliest benefit-cost analyses of job training, provided some hope that even intensive job training programs, for individuals with the greatest barriers to employment, would be worth doing.
In 1983 CETA -- which had come under fire as an ineffective and politically-manipulated program -- was replaced by JTPA. The two major changes intended to increase effectiveness were the development of performance measures which local programs would be required to meet, and the requirements that local administering boards (call Private Industry Councils, or PICs) have at least 50 percent of their members from private business, an effort to make job training responsive to the needs of employers. In addition, public service employment -- which had been criticized as "make work" -- was eliminated from the services provided.
With the passage of CETA to JTPA, the evaluations based on CLMS ceased. In their place, a random-assignment evaluation of JTPA was planned. The results of this study, which are widely regarded as definitive because of the design and complexity of the evaluation,[21] describe employment effects for individuals from 16 specific programs across the country, both 18 and 30 months after leaving the program.
Table 5 presents the most basic results from this evaluation. As in earlier findings, the impact is higher for adult women ($1,176, or a 9.6 percent increase in earnings) than for men (who experienced a $978 or 5.3 percent increase in earnings).[22] However, while these results are statistically significant, and the benefits of JTPA proved to outweigh their costs for both adult men and women (see Table 22 below), in another sense the benefits are small: for women, who might have to support a family, the program increased earnings only from $12,241 to $13,417 over these 30 months, an average annual gain of $470. Even for those who enrolled in the program, the increase was only $735 per year -- not enough to move individuals out of poverty, for example, or enable them to leave welfare. To be sure, the long-run effects might be more positive, as I explore in Section III.6 below; but clearly JTPA did not provide a substantial boost to the earnings of either women or men.
Furthermore, the effects of these programs on youth are zero or even negative. The negative findings come about because the youth enrolled in the program largely withdrew from employment during the period of training, and the lower earnings during the period of enrollment were not offset by increases in earnings after completing the program. (Since labor market experience is one of the most important criteria for hiring in the sub-baccalaureate labor market -- more important even than many educational credentials below the baccalaureate level [Grubb, Dickinson, Giordano, and Kaplan, 1992] -- the long-run effects for youth enrolled in ineffective training programs may be even worse.) The negative findings are particularly discouraging for the worst-off youth -- those who had been arrested prior to enrolling in the program. To be sure, these results for youth reflect only a subset of youth in JTPA, and are therefore not necessarily comprehensive.[23] Nonetheless, these are very discouraging findings which confirm in many ways the results for CETA in Table 3, and they have caused many analysts and policy-makers to call for eliminating JTPA programs for youth.[24]
Other effects of JTPA can be seen from Tables 6 and 7. It clearly increased the proportion of individuals with a GED or high school diploma, though by only trivial amounts for young males; it seemed to reduce the receipt of welfare benefits for women and female youths (though these effects are insignificant), though it increased welfare benefits among adult males; and it reduced the arrest rate for young males who had not been arrested prior to enrolling in the program. But these effects are quite modest and uncertain; and in the case of increasing the rate at which individuals earn a GED, there is substantial uncertainty about whether this credential improves subsequent employment.[25] Thus examining benefits other than those related to employment doesn't improve the conclusions much: there are benefits, but they are modest for some groups, missing for others, and in still other cases -- e.g., the increase in welfare benefits to males, as well as the overall negative effects for youth -- they operate in the wrong direction.
Overall, the results of the JTPA evaluations are sobering. They reveal modest gains for adult men and women -- in the order of $500 to $750 for women per year, and perhaps $400 to $650 for men[26] -- but earnings increases that are essentially zero or even negative for youth. The results are for adults are statistically significant and, as I will point out in Section III.8 and Table 22 below, JTPA programs are also "worth it" in the sense that their benefits outweigh their costs. But the benefits are not significant in any practical sense -- they are too small to change the life conditions of those who have enrolled in job training, to enable many of them to leave the welfare rolls, or to escape poverty. This kind of finding, replicated in many other studies, leads to the puzzle addressed in Section IV: why. after about 25 years of developing job training programs, are the benefits of job training so small?
[21] There is still a problem of external validity despite the sophistication of the experimental designs: the 16 programs were volunteers, and though they did not differ in any obvious way from JTPA programs as a whole it is still possible that they are more effective than randomly-selected programs would be.
[22] These results indicate the problems even with an experiment: Because many of those individuals assigned to enroll in the program failed to show up, the numbers of enrollees actually undergoing job training was smaller than the number assigned. The benefits for those enrolling were evidently higher than those assigned, as Table 5 shows.
[23] These JTPA evaluations examined only the Title II-A programs, not the Title II-C programs that require local SDAs to spend 40 percent of their funding on special youth programs.
[24] In addition, these results do not cover the summer youth programs, which provide young people short-term employment during the summer. There is little evaluation of these programs, though generally subsidized work experience has not been effective in improving employment of low-income youth (Office of the Chief Economist, 1995, p. 11).
[25] The General Education Diploma (GED) is a multiple choice test that is intended to be the equivalent of a high school diploma. However, the evidence about its effects is mixed: a careful analysis by (Cameron and Heckman, 1993) found that it had no effect on employment once individual differences were controlled, though a recent re-analysis of these data by Murnane Willett, and Bordett (1994) found more positive but still small effects. Other evidence (e.g., Quinn and Haberman, 1986) suggests that the GED does not increase the rate at which individuals enroll in postsecondary education.
[26] These are substantially smaller than the gains recorded for CETA, as might be expected given the upward bias of the quasi-experimental results summarized in Table 3.
As I mentioned in the introduction, the idea of providing services -- including education and training -- to enable welfare recipients to move into employment and off welfare extends back at least to 1962. In addition to the voluntary work programs established during the 1960s, the federal government under Richard Nixon allowed states to experiment with their own welfare-to-work programs. Although the evaluations of these programs were primitive, to say the least, they generated an enormous amount of rhetoric on behalf of welfare-to-work programs. For example, while he was governor of California, Ronald Reagan established a community work experience program (CWEP) in which welfare recipients were required to work in community service jobs in amounts related to their grants (i.e., they were required to "work off" their welfare grants). The California CWEP program was a complete failure: it was able to enroll only a tiny fraction (0.2%) of welfare recipients, and it failed to meet every single one of its employment objectives (Employment Development Department, 1976). Nonetheless, Reagan cited the program as a success virtually every time he discussed welfare, and he used the presumed "success" of this program to press for an expansion of welfare-to-work programs. While the early welfare-to-work experiments tended to emphasize work, rather than education and job training, the emphasis shifted somewhat in the 1980s. During the Reagan administration, states were allowed to implement a series of experiments in their welfare programs, incorporating a mix of work requirements and services (including education and training) in order to reduce their welfare populations. These experiments varied widely, to be sure, though most of them ended up emphasizing job search rather than either job training or mandatory work: the political and practical difficulties involved in forcing welfare recipients into employment -- when they have children as well as deficient skills and, in many cases, other personal characteristics that made employment difficult -- was too great for mandatory work to ever be very substantial.
As was true of state programs in general, the state welfare-to work programs of the 1980s varied substantially in the services they provided. For example, of the five state programs described in Table 8, the Arkansas and San Diego programs provided job search workshops, followed by work experience in public and private agencies; Virginia 's program began with a period of job search followed by either work experience, education, or training; Baltimore included a variety of different services including education, training, job search, on-the-job training, and work experience; and West Virginia required community work experience, potentially of unlimited duration. As was true of other state programs, then, these five tended to emphasize job search assistance and work experience programs over education or job training; a few states did allow education and job training, especially as initiated by the welfare recipient. Thus the services in these welfare-to-work programs overlap those provided by JTPA; but the evaluations of these experiments should not be considered evaluations of training itself.
Several of the state welfare-to-work programs were evaluated with random-assignment methods; the results were revealing in their own right, and they also set the stage for more widespread programs enacted in 1988. Table 8 summarizes the results from 5 state experiments. Four of the five increased the amount of employment; two increased earnings by statistically significant amounts (by $560 per year in San Diego and $156 in Arkansas), with two more increasing earnings by amounts that were not quite statistically significant ($176 in Baltimore and $108 in Virginia). As a result, in three of the five states the total amount of welfare payments fell, by amounts per year ranging from $84 in Virginia to $192 in California. But the effects of these programs in moving welfare recipients off welfare was negligible: in none of these five states was the likelihood of being on welfare reduced.
These evaluations also allowed a number of other useful conclusions, in addition to dispelling the notion that "nothing works". The least effective program was that of West Virginia -- but this was generally attributed to the weak state economy in a largely rural state with few employment prospects. This result suggested that welfare-to-work programs (and job training programs more generally) could not be expected to have much effect in weak economics, perhaps including rural economies generally. Second, these experimental programs appeared to provide the greatest help to the least well-off or the least job-ready individuals -- for women compared to men, and for individuals without prior employment compared to those with a previous work history (Gueron, 1987). When added to similar results from CETA, these results suggested that job training programs ought to concentrate their efforts on individuals with the most barriers to employment -- the very opposite of "creaming".
In these welfare-to-work experiments, therefore, there were benefits of some kind in virtually all states, and -- as Section III.8 and Table 21 clarify -- the benefits exceeded the costs of operating these programs. In addition, the benefits varied among states, and the San Diego experience suggested that earnings increases could be substantial -- at least the order of magnitude of benefits recorded by JTPA. However, the effects were still modest by almost any standard: earnings increases were quite small, ranging between $100 and $200 per year in most states; the reductions in welfare payments by states were similarly small; and these programs had no effect whatsoever in reducing the number of families on the welfare rolls, the goal that is at the heart of welfare-to-work programs. The results could be read as either supporting or undermining the continuation of welfare-to-work programs, but they indicated that such efforts made very little difference to the lives of welfare recipients and provided only trivial savings for taxpayers.
One other evaluation of a state welfare-to-work programs was carried out using a non-experimental design (Nightingale et al., 1991). The Massachusetts Employment and Training (ET) Choices program was evaluated comparing ET participants to a control group of other welfare recipients who did not participate but who were matched on the basis of race or ethnicity, age, sex, age of youngest child, region of the state, and whether the individual was in a one- or two-parent family; then regression methods were used to control for variations among individuals (and between experimental and the comparison groups). The results, summarized in Table 9, are of the same order of magnitude as the experimental results in Table 8 but are somewhat more positive: the increase in the probability of employment was greater than for the five states in Table 8, the average annual earnings increase ($780) was larger, and the reduction in annual welfare payments ($307) was larger than the largest reduction among the five states (where California had a reduction of $192 per year). However, it is difficult to know whether these differences are due to random variation among states -- the possibility that Massachusetts was simply at the upper end of a range of outcomes -- or to the greater variety of services offered in ET Choices (which allowed welfare recipients to enroll in job training, postsecondary education, and remedial education in addition to job search assistance and supported work experience), or to the non-experimental design of the evaluation, which probably resulted in outcomes that were upwardly biased. However, the results confirm once again that welfare-to-work programs can affect both earnings and welfare grants -- though the effects are modest, and unable either to improve the lives of welfare recipients substantially or to end the need for welfare itself.
In 1988, in part based on a favorable reading of the experimental program evaluations, the early welfare-to-work programs were expanded in the Job Opportunities and Basic Skills (JOBS) program. JOBS required states to fund welfare-to-work programs (partly with federal matching funds) with a mix of work requirements, job search assistance, work experience, training, education, counseling, child care, and other supportive services. Again, states set up a variety of programs, varying in their job requirements and their mix of services including job training, education, and less intensive services like job search assistance. However, many states implemented their JOBS programs through JTPA as a matter of state policy, while in other states local JOBS programs accomplished the same thing by convening a variety of local providers and emphasizing the use of the existing JTPA system[27] -- in effect commingling JOBS with the dominant job training system.
So far, two evaluations of JOBS program have been completed: one of the California program called GAIN (Greater Avenues for Independence), and another of the Florida program, Project Independence.[28] The GAIN evaluation, using random-assignment methods, investigated the effects of programs in 6 counties (out of 58), varying by their urban/rural characteristics, the state of the local labor market, and the nature of local programs. Of the many services GAIN provided, the increase caused by enrolling in the program was most substantial for job search activities (since 28.5 percent of the experimental group received these services, compared to 3.9 percent of control group members) and remedial education (defined as adult basic education or GED training, received by 29 percent of experimentals but 5.4 percent of controls) -- but vocational training and post-secondary education did not increased markedly as a result of GAIN. This confirms the bias within welfare-to-work programs: even though welfare recipients are allowed to participate in self-initiated education and job training programs lasting up to two years, in practice the emphasis on short-term job search assistance and on remedial education[29] has reduced the use of lengthier education and training programs.
The overall results are summarized in Table 10, differentiated into effects for single parents (almost all women) and heads of 2-parent families (almost all men). Over the three-year period of the evaluation (from early 1988 to mid-1990) the program increased the probability of employment for both groups; increased earnings for both groups, by $471 per year and $370 respectively; and reduced the amount of welfare payments, by $305 per year and $389 respectively. GAIN also reduced the likelihood of being on welfare by 3 percentage points among the single parents, since 55.5 percent of the controls but only 52.5 percent of the experimental group were on welfare at the end of the third year;[30] however, there was no effect in moving heads of households off welfare. To be sure, these are average effects across all six counties; the substantial variation among counties (reviewed in Section III.7) indicated that some counties were more effective than others. Overall, however, the results are quite consistent with the pre-JOBS results: welfare-to-work programs relying on the most modest services -- in this case, job search assistance and remediation -- can increase employment and earnings and result in some welfare savings, but the effects are small and the reduction in the numbers of families on welfare is trivial.
The results from Project Independence are different in several respects. This welfare-to-work program was really a job search program, since it provide relatively little education or training. Like many other programs, it did increase the amount of employment (as Table 11 shows), and also increased earnings -- though the average increase of $114 per year was very small. It also succeeded in reducing the rate at which individuals received welfare, and in fact the savings in welfare payments were greater than the increases in earnings -- a finding that proves to affect the cost-benefit results (in Table 24 below) substantially. However, two other findings were of greater interest. The evaluation found substantially higher benefits to mothers with children over the age of 6 -- who have many fewer problems related to child care, scheduling, sick days, and the like -- than to mothers with children age 3 to 5, as the results in Table 11 clarify. This finding reinforces the notion that certain individuals -- those with substantial barriers to employment -- may not benefit from job training, an idea that I will examine further in Section III.4.
In addition, the evaluation revealed a common vulnerability of job training and welfare programs. During the evaluation, a worsening recession in Florida caused welfare caseloads to increase while funding stayed the same; the result was that the resources available to clients who enrolled late in the evaluation -- the "late group" -- were substantially smaller. Consistent with this, the increases in earnings and the declines in welfare payments were significant only for the "early group", not for the "late group" -- clarifying that the fiscal conditions of welfare-to-work programs may influence the results substantially. The implication is that individuals who point to certain exemplary welfare-to-work programs as evidence of what job training might accomplish -- for example, the Riverside program described in Section III.7 and Table 19 below -- neglect the more likely possibility that fiscal constraints will lead to low-quality programs with no effects.
One hint about how this program affected behavior came from an examination of attitudes and values. Those who enrolled in the program were more likely to agree that "even a low-paying job is better than being on welfare", were less likely to think than mothers should stay home with their children rather than working, and had lower reservation wages. Even though many of these differences were statistically insignificant, they all support the notion that welfare-to-work programs can change attitudes, replacing an acceptance of welfare with a greater commitment to work -- consistent with one of the intentions behind welfare-to-work programs. But Project Independence also reduced the overall income of those enrolled by a small amount, and it decreased the fraction of those saying that they were satisfied or very satisfied with their overall standard of living, from 45.7 percent among the control group to 41.9 percent among those in the program. Project Independence therefore represents a relatively conservative approach to welfare-to-work programs, in which the costs of welfare to taxpayers declines (see also Table 24) at the expense of welfare recipients themselves.
[27] See Lurie and Hagen (1994), summarizing a large number of monographs on the implementation of JOBS; Grubb et al. (1990); and Grubb and McDonnell (1991).
[28] A random-assignment evaluation of JOBS programs in 6 states is also underway, but the results on outcomes will not be completed for at least another year.
[29] The emphasis on remedial education comes from the fact that individuals in GAIN go through a precise sequence of stages. At an early stage their basic reading and math are evaluated, and those who are deficient are directed to remedial education. In general, one of the discouraging (but unsurprising) findings of welfare-to-work programs is how many welfare recipients lack basic academic skills.
[30] This figure nicely illustrates the amount of movement on and off welfare that occurs "naturally", without any special programs: almost half of the single parents in the control group left welfare by the end of the third year.
There has been a common pattern in the United States of experimenting with social programs, trying out promising practices on a small scale before expanding them to more universal programs. Indeed, the history of job training efforts can be interpreted as part of this larger history. The development in the 1960s of manpower training programs located outside the schools was in part an effort to develop novel efforts that bypassed the presumed deficiencies of the educational system; these "experiments", with the roughly positive results summarized in Table 2, were then institutionalized in CETA and then JTPA. Similarly, the current round of welfare-to-work programs in the JOBS programs emerged from welfare-to-work experiments operated by states during the 1980s, which were in turn based on fledgling efforts of the 1960s and 1970s.
There have been several other, more self-consciously experimental programs developed with private and foundation funding that have contributed to our knowledge of "what works". There have, not surprisingly, been a very large number of experimental programs, each with its partisans -- partly because private foundations in the United States often operate by developing an experimental approach (or discovering an experimental approach underway in some corner of the country) and then replicating it elsewhere, sometimes with some kind of evaluation. Some of these efforts have proved not to be especially effective because they cannot be replicated -- for example, some are dependent on the high energy and charisma of a founding leader, and do not work once they are operated by other people. However, others have been designed to be replicated, and have been intended as tests of particularly approaches to employment and training. In contrast to the mainstream job training and welfare-to-work programs -- which have provided a variety of services, often without much thought or guidance about what might be appropriate -- these experimental programs have developed clear models of what employment-related services ought to be provided to specific population groups, and have then worked to make sure that these models are carefully implemented. One might expect, therefore, that these experimental programs would be of higher quality, and would generate more favorable outcomes, than do the mainstream programs reviewed in the previous two sections.
In this section I present the evidence for 4 programs: two for young mothers with children, the Minority Female Single Parent Demonstration (MFSP) and New Chance, a program for young poor mothers; and two for youth, JOBSTART for high school dropouts and the Summer Training and Employment Program (STEP) for youth at risk of dropping out of high school.
Because the four MFSP sites were somewhat different in their services, the outcomes were reported for each of the four programs. Table 12 presents the most important results from the four. The obvious story is that three of the four programs had no influence whatsoever on employment and earnings, or on the receipt of welfare benefits, over the period of the evaluation. However, one program -- the Center for Employment Training -- had substantial effects, increasing the amount of employment by 13 percent, increasing wage rates by 11 percent, and increasing average earnings by 25 percent or $101 per month ($1,212 on an annual basis) -- a huge effect compared to those of other JTPA programs in Table 5, for example, or the GAIN results in Table 10. Like GAIN and earlier welfare-to-work programs, MFSP did not decrease welfare payments substantially, nor did it decrease the likelihood of being on welfare.
The MFSP Demonstration results can be read either positively or negatively. On the negative side, three of four programs that were funded to create exemplary job training programs, with relatively lavish funding and special attention to the design of the programs, failed to have any effects at all. On the positive side, the substantial effects of CET -- confirmed in the JOBSTART evaluation reviewed below -- suggested that well-designed programs can work. Indeed, the success of the CET program in this evaluation has been the subject of a kind of publicity campaign to trumpet the success of a particular approach. In the interpretation of the evaluators and the founder, the success of CET was due to its efforts to integrate remedial education and vocational skill training, and they then began to promote a model of job training that depends on such an integration (Burghardt and Gordon, 1990; see also Literacy and the Marketplace, 1989). This may in fact be a worthwhile approach, as I argue in the Conclusion of this monograph; but the success of CET is due to many factors in addition to the provision of both remediation and job skill training, as I point out in Section III.7.
Overall, however, the results of the MFSP Demonstration are remarkably consistent with other evaluations, especially those of GAIN: job training programs on the average have modest positive effects on employment and earnings,[31] very little effect if any on welfare payments, and no effect on the likelihood of being on welfare -- though individual programs (like CET) may have much more substantial effects.
[31] Across all four programs in Table 3, the average effect on earnings was 6 percent or about $17 per month ($208 per year)
Table 13 presents the findings of random-assignment evaluations 18 months after entering the program. (Subsequent results will examine results after 42 months; if it takes time for positive results to emerge -- as seems to be true for certain programs, as reviewed in Section III.6 below -- they will not be apparent in the findings available so far.) New Chance was successful in increasing attendance in GED programs and college, and in increasing the proportion of mothers who earned a GED and some credits toward and postsecondary credential. Surprisingly, however, the program did not increase scores on a test of basic skills (the TABE, or Test of Adult Basic Education), and it left 72 percent of those enrolling (compared to 70.2 percent of the controls ) reading at a ninth grade level or below -- suggesting that it is possible to earn a GED without improving academic competencies.[32] For a wide range of other outcomes, however, New Chance made no difference in the short run. Indeed, if anything it appeared to have negative consequences: it appeared to increase pregnancies, though these were balanced by increased abortions so that the number of births stayed about the same[33]; it reduced employment, though not significantly so; it reduced earnings significantly; and it increased welfare slightly. To be sure, it is possible that these results reflect withdrawal from employment during the period of the program itself, and that the early declines in employment and earnings will be reversed after young women get into more stable employment in their third and fourth years after enrolling (as for the JOBSTART program summarized below). However, the early findings, roughly consistent with the MSFP Demonstration results, are not at all encouraging since the only real benefit has been an increase in a credential -- the GED -- which has little effect on either employment or subsequent education. The results are particularly discouraging given the high cost of the programs of about $9,000 per person.
[32] Others have suggested that it is possible to pass the GED with an eighth or nineth grade reading level -- perhaps an indication why earning a GED has so little influence in either employment or in enrolling in postsecondary education.
[33] In the American context where there is vociferous opposition to abortion, it would be politically detrimental to any program to increase both pregnancies and abortions.
The JOBSTART demonstration took place in 13 sites around the country, using JTPA funds.; it was modeled roughly on the successful Job Corps program (reviewed in Table 4 above), though it was less intensive and non-residential. Each site provided remedial education, vocational skill training, job placement assistance, and various support services like child care, transportation, counseling, and instruction in work readiness and job skills; sites were required to offer at least 200 hours of basic education and 500 hours of job training, making them more intensive than conventional JTPA programs. Three different models of service provision were followed: concurrent programs provided remedial education and occupational training concurrently; sequential/in-house programs provided remedial education and then vocational skill training; and sequential/brokered programs provided remedial education and then referred participants to other programs for vocational skill training.
The JOBSTART Demonstration was evaluated using random-assignment methods over a four-year period. The most important results are summarized in Table 14. One major effect is that the program increased the rate at which drop-outs received a GED -- a result that is not surprising because most of the programs emphasized the GED. However, given the limited effects of the GED in increasing employment or access to postsecondary education, this impact may not be of much value. Indeed, over the four-year period the effects on employment and earnings were insignificant, both for the total sample and for selected sub-groups. As in other evaluations, the proportion of women receiving welfare did not decrease overall -- indeed, it increased for women with children -- and the amounts of welfare did not decrease. Nor did rates of pregnancy or giving birth fall significantly -- a special concern because of the negative effect childbearing has on poverty and welfare dependency; indeed, for mothers entering JOBSTART, rates of pregnancy and giving birth increased during the program. One positive result is that rates of arrest appeared to fall, as did drug use (significantly so for hard drugs excluding marijuana).
The results in Table 14 are dismal, and suggest that well-designed job training programs of moderate cost do not work for youth at all. However, these average effects do mask some potentially positive findings for the longer run. A common pattern was for employment and earnings to fall in the first year of the program, while individuals were enrolled in education and job training, with employment increasing in the second year while earnings and earnings increased in the third and fourth years -- suggesting (from the fourth panel of Table 14) a pattern in which those enrolled in the programs increase their earnings about $400 per year over the long run. This pattern emerged for most sub-groups including men, mothers, and other women, and was particularly marked for men arrested before enrolling in JOBSTART -- for whom earnings increases were $1,129 in year 3 and $1,872 (and statistically significant) in year 4 -- and for youth who left school for academic reasons, for whom earnings increases were $726 in year 3 (and statistically significant) and $592 in year 4. In addition, the finding that drug use decreased as a result of the programs suggests that more positive effects might show up in the long run as some individuals avoid drug-related arrests and drug-motivated unemployment and create stable employment records instead.
The other positive finding is that one of the 13 sites -- the Center for Employment Training in San Jose, the successful site in the MFSP Demonstration -- had statistically significant increases in earnings: Over the four-year period, the experimental group earned $32,959 compared to $26,244, an increase of 25.6 percent and $1,679 per year (and $3,044 per year over years 3 and 4). Unfortunately, disaggregating by sites makes the findings more dismal, if anything. Seven of the 13 sites had negative outcomes -- about what one would expect by chance alone -- and two of them had very large negative effects in the range of $6,200 over the four years. Eliminating the one clear success of CET as a special case, the other 12 sites averaged negative effects of $1,393 over four years and $211 over years 3 and 4, making it difficult to conclude that the long-run effects could be positive. The CET program may be a success story, for the special reasons I analyze in Section III.7, but otherwise it is difficult to find much hope for youth programs in the JOBSTART results.
The STEP program was evaluated with a random-assignment approach in which students enrolled in both the school and the work component were compared to others enrolled in the work component only; thus the design tests the additional effects of the summer schooling component. The effects on reading and math scores and on knowledge of contraception after the first summer were positive; gains during the second summer were also positive though somewhat smaller. However, 3 1/2 years after enrolling in the program, STEP youth experienced the same dropout rates, rates of postsecondary enrollment, employment rates, and rates of teenage pregnancy as did the control group (Walker and Vilella-Velez, 1992; Grossman and Sipe, 1992). One widely-cited conclusion from the STEP experiment is that, while it is possible to improve academic performance through a short-term program, long-term results and effects on more fundamental behavior like employment and pregnancy cannot be changed with a short-term intervention that leaves the rest of schooling and the general environment of poor youth unchanged.
Overall, the results of these experimental programs are disheartening. Although two of them identified a particular program -- CET in San Jose -- as having particularly strong effects, the programs on the average had no effects or even negative effects. To be sure, the populations included in these experiments were among the most difficult to employ, since two focused on young mothers with children and two others on low-income youth. But the results clarify that the modest effects achieved in the general JTPA and JOBS programs cannot be improved upon merely by paying somewhat closer attention to the design of the program and to their implementation.
The results in the previous three sections concentrate on the overall effects of job training programs. However, an important question is whether the effects vary for different population groups -- for example, if they are greater for men versus women, or for adults versus youth, or for whites compared to black or Hispanic individuals, or for those who are the most employment-ready compared to those who have multiple barriers to employment. If it were possible to find certain groups for whom job training programs work better than for others, it would be possible to target resources to those groups and increase the overall effectiveness of the limited resources available for job training, or to provide different kinds of services to different groups.
In the early evaluations of manpower projects (e.g., Table 2), there were no clear conclusions about which groups benefit the most from job training. However, the CETA evaluations found greater effects for women compared to men, and for adults compared to youth (e.g., in Table 3, elsewhere in Barnow, 1986, and Bloom and McLaughlin, 1982). In addition, the benefits appeared to be higher for those with little labor market experience, compared to those with substantial experience before enrolling in a program (Bloom and McLaughlin, 1982) -- suggesting that job training programs might benefit those with the least skills and experience, while those with more skills and experience would simply cycle in and out of employment on their own and not be helped by job training programs.[34]
The results of the welfare-to-work experiments in the 1980s tended to confirm the findings that the lest job-ready individuals would benefit the most. Employment increases were generally greater for women on welfare, for example, and for individuals without prior employment histories (Gueron, 1987, p. 28). A more sophisticated reading of the evidence suggested a kind of tripartite result, presented in Table 15: individuals within these welfare-to-work programs who were the most job-ready -- who were first-time welfare recipients -- generally did not benefit (those in Tier 1); and those individuals with the most serious barriers to employment -- those on welfare more than two years and those with no prior earnings -- had low and often insignificant benefits (in Tier 3). A group in the middle, with some prior time on welfare and with low (not non-zero) earnings (in Tier 2) appeared to gain the most (Friedlander, 1988; Gueron and Pauly, 1991, Ch. 4). Such a conclusion is consistent with earlier findings that the most-job-ready individuals do not benefit much from job training, but also with the negative results of the experimental programs reviewed in Section III.3 that focus on the most disadvantaged. This kind of finding also supports a kind of "triage" policy, in which the most job-ready would be denied access to job training programs (in contrast to the practice of "creaming"); the most disadvantaged, with the greatest barriers to employment, would also be denied access to conventional programs or -- because this practice might be difficult to implement because of political and moral considerations -- would enroll in more intensive and expensive programs than are typically offered. Conventional programs would then concentrate on the group in the middle, for which the most substantial gains at lowest cost might be expected.
The more recent JOBS evaluations have also generated results for specific groups within the population. For the California GAIN program, one set of results tended to confirm the kind of tripartite finding from the earlier experiments: recipients of welfare who were moderately disadvantaged earned more, and received lower welfare benefits, than did either more disadvantaged recipients or new applicants (Riccio, Friedlander, and Freidman, 1994, Table 4.6). Unfortunately, for other groups this conceptually straightforward conclusion became less clear, because the effects on sub-groups varied so much among the six counties. For examples, individuals were classified according to whether they needed basic education or not, since those in need of basic education may need remediation before they can benefit from job training, job search assistance, or any other services -- and therefore may be more expensive to return to employment and less effective in finding employment after any job training program. Overall, both groups benefited from GAIN, in both increased earnings and reduced welfare benefits. However, one of the six counties (Riverside) increased the earnings of both groups; two (Alameda and San Diego) increase the earnings of those not needing basic education, but not the other group; and two (Butte and Tulare) increased earnings of those needing basic education but not those with better academic skills.
In addition, long-term welfare recipients benefited in three or four of the six counties, contrary to the results in Table 15; but new applicants also benefited in two of the four counties that collected such information. The programs in Riverside and San Diego benefited both those with and those without prior employment; on the other hand, Alameda and Los Angeles benefited only those without prior employment, while Butte and Tulare benefited only those with prior employment. Thus these results tend to cast doubt on the "triage" solution suggested by earlier evaluations: at least under some conditions, in some programs, the most disadvantaged individuals can benefit from welfare-to-work programs, while in other cases the principle beneficiaries are the least disadvantaged.
The evaluation of the Florida program, Project Independence, also muddied the waters somewhat. As Table 11 shows, mothers with children age 3 - 5 benefited much less than did mothers whose children were 6 and over -- consistent with the practice of "creaming" that selects only the individuals with the fewest barriers to employment; and those considered job-ready (in terms of their education and prior labor market experience) benefited somewhat more than those with less education and experience. However, individuals who had been on welfare for two years or more -- who were presumably less job ready -- benefited much more than did those on welfare less than two years (or those who were first-time applicants). Thus the effects for specific groups provided little clear guidance for selecting individuals, save for the demonstration that mothers with young children benefit less than others.
The results of evaluating JTPA using more sophisticated random assignment methods have confirmed the greater impact for women compared to men (table 5) -- though the differences were not as great as for the earlier CETA evaluation (e.g., Table 3) -- and the finding of zero or even negative effects for youth confirmed the earlier conclusion that job training might be effective for adults but not for youth. (The findings of greater effects for women are consistent with the earlier findings of greater effects for the least job-ready -- since women are likely to have less labor force experience, and to have children that complicate their employment -- but the larger effects for adults rather than youth contradict this pattern.) For other sub-groups, the results were generally statistically insignificant, and therefore the most cautious conclusion is that there were no differences among different groups in the benefits of job training (Orr et al, 1994, Ch. 5, especially Exhibits 5.8, 5.9, 5.19, 5.20). However, the more serious problem is that the differences which can be detected did not fall into any real pattern. Some of the differences confirm the conclusion that the least job-ready individuals, or those with the greatest barriers to employment, benefited the most:
After a great deal of research, therefore, the effectiveness of job training programs for different sub-groups remain murky. Overall, women benefit more than men, and adults more than youth, for whom the results have often been zero or negative. But at least in some programs, the least job-ready individuals benefit, while in others the opposite is true. This in turn suggests that any particular strategy for selecting applicants -- for example, the efforts to "cream" or select the most job-ready, or the contrary efforts to target job training on the most disadvantaged individuals, or the "triage" solution suggested by Table 15 -- cannot be generalized.
[34] This conclusion is also supported in Fischer's (1995) meta-analysis of job training programs. However, he provides no numbers to back up this claim, and the most recent evaluations -- the GAIN study by Riccio, Freidlander, and Freedman (1994), and the JTPA study at 30 months by Orr et al. (1994), both of which weaken his results -- have not been included in his analysis.
[35] This is the only one of these comparisons that was statistically significant.
The early evaluations (e.g., in Table 3) were often interpreted as supporting the greater effectiveness of on-the-job training over classroom training (e.g., Taggart, 1981, p. 282). Some reviews of the CETA experiences similarly concluded that on-the-job training was more effective than classroom training, which in turn was more effective than work experience by itself, without a training component or associated classroom instruction (Taggart, 1981, pp. 282 - 288). This kind of finding was consistent with the evidence that CETA increased earnings not by increasing wage rates -- or, in economist's terms, by increasing productivity -- but by increasing rates of employment, something that might be improved more by on-the-job training and its attention to work-related behavior than by classroom instruction and its emphasis on cognitive and vocational skills. Similarly, Barnow (1986), reviewing a greater number of studies, concluded that public service employment and on-the-job training had greater effects than classroom training, and that work experience had no effects (or even negative effects). In addition, the early evaluations of CETA concluded that longer classrooms training programs were on the whole more effective than shorter programs, especially for women (Taggart, 1981, p. 103 ff.); in part, the strong results from the year-long, residential Job Corps program (Tables 4 and 20) strengthened the case that longer, more intensive programs would be more effective.
The JTPA evaluation, using more stringent random-assignment methods, also examined the effectiveness of different kinds of services provided. Individuals were assigned to three different kinds of services: classroom training in occupational skills; on-the-job training/job search assistance, where individuals typically were enrolled in job search assistance and then sometimes found on-the-job training or unsubsidized jobs; and a category of "other" services that might include basic or remedial education, job search assistance, work experience, and miscellaneous other services. For adults, individuals with classroom training received an average of 551 hours of services costing $3,174, substantially higher than for either OJT/JSA (averaging 222 hours and $1,427 per person) or "other" services (204 hours at $1,116 per person). For youth, "other services" (averaging 328 hours at $2,016 per person) were almost as intensive as OJT/JSA (317 hours at $2,755), while again classroom instruction was the most intensive approach (averaging 596 hours at $3,305). To put these figures in perspective, a one-year certificate program in a community college might take 30 weeks of time with about 25 hours per week, or 750 hours -- so the most intensive of these programs involved only two-thirds of the hours of the least intensive vocational program, and the most common services were only one-third the intensity.
However, the effects of these different kinds of services are not related to the intensity of services.[36] As the results in Table 16 indicate, the benefits are greatest for "other services" for women, followed by OJT/JSA; for men, OJT/JSA is the most effective (though the difference is statistically insignificant). These results tend to confirm earlier findings about the superiority of on-the-job training to classroom instruction. For youth, however, the results are inconclusive because of the lack of statistical significance, though classroom training appears more beneficial (or, in reality, less harmful) than other services.
In the early evaluations of welfare-to-work programs, it was not possible in any formal sense to evaluate the effectiveness of different services because individuals were assigned to various services by non-random methods. The states varied in the mix of services they provided, leading to the conclusion that different kinds of programs could all achieve the kinds of modest results found in Table 8 (Gueron, 1987, p. 30). However, because many control group members in these experiments had access to education and job training programs through educational institutions and JTPA, the greatest difference in services between experimental and control groups were typically in the short, job search assistance provided in welfare-to-work programs.
The design of the GAIN program in California reinforced this kind of conclusion. For many participants (those with a high school diploma or a GED, or who passed a literacy test), GAIN required job search assistance before any other services; therefore the dominant service provided by GAIN was the short-term JSA, and the generally positive effects in Table 10 can be attributed to this kind of service rather than any other. However, the differences among the 6 different counties evaluated in this study did suggest some other conclusions. In four of the six counties, the presence of large earnings increases among those in need of basic academic education suggest that remediation is an important component of job training; on the other hand, the absence of earnings gains for these needing in remediation in two counties that emphasized remediation (Alameda and San Diego) indicates that basic skills instruction cannot guarantee success. Other results in Alameda also suggested that vocational training and postsecondary education could have an effect -- though two counties (Riverside and San Diego) produced large earnings increases without increasing vocational training or postsecondary education. The findings therefore suggest -- as Taggart concluded from the CETA results earlier -- that a variety of different service strategies can be used in effective programs, without strong evidence that one particular service is preferred over another.
On the other hand, the results of the Project Independence program (Table 11) indicate the importance of at least some minimal level of services to outcomes. The group that enrolled late in Project Independence, after a recession forced increases in caseloads without a corresponding increase in resources, did not benefit at all -- presumably because they received very little from the program. (The cost per person for individuals in the "late" cohort was slightly less than $900, less than almost any other job training program.) Even at its best Project Independence was not a particularly intensive program, since it provided little more that job search assistance to most clients; but even so the results suggest that results cannot be expected when resources dip below some critical level.
The experimental programs described in Section III.3 all involved much more intensive and carefully-devised programs, incorporating a range of services. The disappointing outcomes therefore give little support to the idea, associated also with the Job Corps, that a range of services and more intensive programs are necessary to overcome the multiple handicaps to employment of the least job-ready groups. On the other hand, the evidence for the Center for Employment Training in San Jose, reviewed in Section III.7, seems to support the conception of comprehensive services. This idea remains an attractive one, and I return to it in the Conclusion -- but neither the national JTPA evaluation nor the GAIN study nor the experimental programs of the past few years provide conclusive support for this idea.
One possible interpretation is that either of two very different strategies may work in job training programs. The first involves relatively inexpensive approaches. On-the-job training has proved to be effective in CETA and JTPA; similarly, job-search assistance supported in many welfare-to-work programs appears to be modestly effective, even though it is relatively inexpensive and of short duration. Both services are designed to get individuals into employment quickly and to socialize them to the norms and values of employment, though neither do anything to enhance the basic cognitive or vocational skills of individuals who lack such competencies. The fact that job training programs have proved to be effective more because of enhancing employment rates, rather than increasing wage rates (and presumably productivity), suggests that the effectiveness of such services is related to getting individuals into employment quickly. Whether such limited services have much effect over the long run, or whether they merely substitute one group of under-prepared workers for others, are questions that the existing evaluations cannot answer. The second effective strategy has been associated with more intensive services, including the kinds of remedial education supported by GAIN and various experimental programs, and often combined with vocational skills training and supportive services. This intensive approach is designed to increase the competencies of individuals who are enrolled in job training programs, rather than to push them into low-wage jobs, and to increase their productivity and wage rates over the long run. These two strategies serve somewhat different purposes, to be sure. More to the point, they can be linked -- as I argue in the Conclusion.
[36] Because individuals were assigned to services based on which were considered appropriate, the characteristics of clients varied among services and these results are not experimental.
The majority of job training programs are short term, lasting perhaps 15 to 20 weeks; and they are generally self-contained, so that individuals enrolling in them do not typically enroll in other programs to continue the process of education and job training.[37] Therefore job training programs are usually "one shot" efforts to get individuals into employment, rather than the beginning of a longer period of education and job training (as, for example, postsecondary education can be). This is turn raises the question of what the long-term benefits of short-term programs are. If, consistent with the human capital model, job training programs provide their clients with real skills (cognitive, behavioral, or vocational) that increase their productivity and are valued in the labor market, then one would expect to see a permanent increase in wage rates and earnings. If in addition the initial enhancement of skills allows individuals to enter "careers" with subsequent on-the-job training that further increases productivity, one would expect to see wage rates and earnings continuing to increase over time -- as happens in the age-earnings profiles associated with different levels of formal schooling. If, however, job training programs merely push individuals into the labor force without increasing their skills substantially, and fail to gain them access to "careers" or other positions with long-run possibilities for advancement, then the effects of such programs may be short-term, and even with positive short-term benefits the long-run benefits may be essentially zero. Indeed, various education programs have been found to be subject to this kind of "fade-out" or decay of benefits -- including the STEP program described in Section III.3 above -- from which many have concluded that sustained interventions are necessary to improve the life chances of low-income students.
Thus it is important to examine the benefits of job training programs over time, to see whether any initial benefits are sustained (or even increase) or whether they decay. Unfortunately, this has not been easy because most evaluations last a comparatively short period of time, and the long-term effects are therefore unknown. However, a few studies shed some light on this problem.[38] In Table 17 for example, the pattern of earnings for the JTPA evaluation, summarized in Table 5 above, is displayed for different periods over the 30 months of the evaluation. During the first six months, earnings increases are essentially zero, because this is the period when individuals leave the labor force to enroll in the program. In the next 12 months (months 7 to 18), the earnings advantage increases to an average of $68 per months among women and $45 among men; in the next twelve months this advantage is sustained at about $71 per month among women and men (and becomes statistically significant among men). There is no evidence in these figures of further expansion of benefits, but neither do they indicate any "fade-out". (For youth, not surprisingly given the negative results in Table 5, there is no pattern of benefits at any time during the 30 months.) Similarly, the results for JOBSTART in Table 14 suggest that earnings of those enrolling are initially lower (as are hours worked), but then increase to a plateau of about $400 per year more than those in the control group -- though employment rates, after increasing in year 2, seem to fade away. However, the Project Independence results in Table 11 indicate some "fade-out" or decay in benefits, since increases in earnings and employment, and decreases in welfare payments, are greater in year 1 than in year 2.
The evaluations based on the longest period of time involved four of the welfare-to-work experiments begun during the 1980s, and included programs in Virginia, Arkansas, Baltimore, and San Diego (Friedlander and Burtless, 1995). Figure 2 presents the effects of these programs on annual earnings over the five-year period, as well as on welfare payments; the actual figures for employment rates and earnings are given in Table 18. The dominant pattern, in the top panel of Figure 2, is for earnings increases to be trivial in the first year, to increase substantially in year 2, to remain substantial in years 3 and 4, but then to fade in years 4 and 5. Similarly, welfare savings are typically trivial in year 1, increase, but then fade in years 4 and 5. (It's important to note that the period of greatest effect includes years 2 and 3, which is precisely the period covered by months 19 to 30 in the JTPA results in Table 17.) In the more detailed information in Table 18, there appears to be "fade-out" in the employment rate for all programs except perhaps in Arkansas, and "fade-out" in earnings except perhaps in Baltimore.[39] Over a period of time, therefore, these results indicate that the kinds of welfare-to-work programs instituted during the 1980s, and largely continued in the JOBS program, do increase earnings and reduce welfare payments, but only in the moderate run; over the long run they leave individuals with employment rates and earnings no higher than welfare recipients who have not enrolled in such programs, and they do not permanently move individuals off welfare.[40]
There are two possible exceptions to the pattern of decline over time. The only welfare-to-work program with stable benefits was the Baltimore program, where earnings increased to year 3 and then stayed about the same in year 5 (see also Figure 2). One possible explanation is that the Baltimore program stressed human capital development through education and training, more than job search intended to accelerate job finding -- and the development of enhanced competencies increase earnings capacity over the longer run (Friedlander and Burtless, 1995, p. 144). In addition, the CET program in San Jose -- a program that is clearly a special case -- showed relatively stable increases in earnings over a five-year period, though increases in wage rates and in employment rates tended to fall (Zambrowski and Gordon, 1993).
In general, however, these results do not suggest that expansion of benefits over time occurs as a result of job training program -- as happens, typically, when individuals complete educational credentials like Associate degrees and baccalaureate degrees. In the short run like the first year, benefits of job training programs are typically non-existent because individuals withdraw from the labor force -- and short-run evaluations (like that of New Chance in Table 13) are therefore suspect. Over the moderate run, in years 2 and 3 (and perhaps year 4 as well) the benefits increase; but after that they decay.
To be sure, these findings can be interpreted either negatively, because of the lack of long-run effects of job training programs, or positively (Friedlander and Burtless, 1995). In the four programs described in Table 18, two of them saved money for governments since the reduction in welfare payments over five years ($735 in Arkansas and $1,930 in San Diego) outweighed the costs per person (of $118 and $920 respectively). In all four of the programs the increases in earnings over five years were larger than costs, so that from a social standpoint the benefits to all individuals -- recipients plus taxpayers -- outweighed the costs. In this sense these welfare-to-work programs were "worth doing", even though they did not reduce the welfare rolls and did not prepare individuals to leave poverty in any permanent sense.
[37] This may happen informally; in particular, some individuals claim that locating job training programs in community colleges provides individuals easier access to the educational programs of these institutions; see, e.g., Grubb et al. (1991). However, my point is that job training programs are not structured to lead to subsequent training and education opportunities. For a proposal to link education and training programs in "ladders", see also the Conclusion.
[38] In addition to the experimental results reported here, Geraci (1984) used the CLMS to measure the relation between longer-term earnings and short-term indicators. In general the correlations are quite low, indicating the inaccuracy of short-term measures of outcomes.
[39] There are technical problems in each of the two figures that suggest the lack of fade-out: the year 6 employment rates for Arkansas are actually extrapolated from two quarters of data, and the year five earnings in Baltimore are extrapolated from three quarters of data. These two crucial numbers could therefore be considered less certain than the figures based on four quarters of data -- even though the method of calculating standard errors, which relies on variation across individuals rather than over time, does not consider this.
[40] These results are corroborated by Fischer's (1995) meta-analysis, which concluded that across all studies the mean effect sizes for the proportion employed increase gradually until quarter 9, at the beginning of year 3, but then decay rapidly over the next three quarters. Similarly, the proportion on AFDC declines slightly until quarter 6, remains substantial in quarters 7 to 10, and then declines to zero; See Tables A.1 and A.2.
One of the clear implications of many evaluations is the unsurprising finding that there is substantial variation among specific programs. The important question, however, is whether these differences can be explained, either by the characteristics and quality of the programs themselves, by the characteristics of individuals accepted into these programs (which is particularly important if some programs engage in "creaming"),[41] or by the local economic and employment conditions of particular programs. Otherwise, one would normally expect some random variation in the outcomes of programs -- that is, purely by chance some would have better outcomes than others -- but such variation would not be useful to program administrators or policy-makers trying to improve the quality of job training programs.
An example of variation among specific programs than may be purely random comes from the JTPA evaluation (the evaluation summarized in Table 5 above). At the end of 30 months there were substantial differences among the 16 sites examined: the increases in earnings varied from $2,628 to negative $2,033 among women, and from $5,310 to -$2,637 among men. Among youth, for whom the average effects were negative, the variation was even more marked since one program achieved statistically significant earnings increases of $3,372 for females and a whopping $9,473 among males who had not been arrested (Orr et al., 1994, Exhibit 4.5 and 4.16). However, partly because of small sample sizes in the 16 sites, these differences across sites were not statistically significant; and an analysis that tried to attribute the differences across sites to program conditions, economic conditions, and personal characteristics yielded no statistically significant results.[42]
The evaluations of welfare-to-work programs during the 1980s found benefits smaller in West Virginia than in other states (see Table 8), a difference generally attributed to poorer economic conditions in that state. In the GAIN evaluation, one finding was that the Riverside program was consistently more effective than any of the other five counties (Table 19); indeed, the increase in earnings of nearly 50 percent for individuals in Riverside county is one of the largest effects ever found for any job training program, and the correspondingly large net benefits in Riverside suggested that well-designed programs could save taxpayers money by reducing welfare costs sufficiently (see Table 23 below). Conversely, however, the GAIN results indicate that -- even within a state that imposes a certain uniformity on its welfare-to-work programs -- there can be localities (like Los Angeles and Tulare, the latter a rural county) where programs may have no effects on earnings at all. Indeed, the variations among sites were even greater than those revealed in Table 19: within counties, there were substantial differences between the results in outlying, largely suburban offices, where the benefits were substantial, and those in inner-city offices, where the benefits were either insignificant or even negative -- a result suggesting that the demographic composition of those enrolling might be responsible for the differences. However, a regression-based analysis controlling for such demographic characteristics did not eliminate the substantial differences among counties and among offices within counties (Riccio, Friedlander, and Freedman, 1994, Table 8.2); and in any case some counties were successful with those not needing basic skills while others succeeded with those who were less job ready, suggesting that different counties could be successful with different groups of clients.
Nor did differences in local economic conditions -- measured, for example, by unemployment rates and the growth rate of employment -- explain differences; in particular, the effects of the Riverside program were remarkably consistent even though economic conditions in that county varied substantially during the period of the study, and varied in addition among office. Nor was there any obvious explanation of differences among counties based on the kinds of services offered. In the end, the evaluators concluded that the success of Riverside was due to the combination of practices there: a strong message to participants about the importance of getting into jobs early; a strong commitment to job search and job placement efforts; a mix of job search, education, and training; and a commitment to enforcing mandatory participation of all eligible welfare recipients (Riccio, Friedlander, and Freedman, 1994, Ch. 8). Another observer has concluded that the high expectations of the program staff were responsible (Bardach, 1993), and still others have attributed the success to the energy and charisma of the director.
The evaluation of Project Independence also found substantial differences among counties, ranging from earnings increases over two years of $1,333 to earnings losses of $570 (Kemple, Friedlander, and Fellerath, 1995, Table 6.13). However, these differences were not statistically significant -- that is, it is possible that they were generated simply by chance -- and they were not related in any obvious way to variation in labor market conditions, services available, or patterns in program participation. Such results add to the conclusion that, while there are differences among programs that appear substantial, it is not yet possible to explain these differences by labor market conditions or by program characteristics that could be affected by policy.
The experimental programs reviewed in Section III.3 above also found substantial differences among specific programs. One results in particular was striking: two independent evaluations, one of JOBSTART and one of the Minority Female Single Parent Demonstration (in Table 12), found that the Center for Employment Training in San Jose, California, was much more effective than the other programs. The MFSP evaluators concluded that the success of CET was due to its practice of linking (or "integrating") both job-specific skills and remedial education, along with its attention to job placement and its availability of child care at the site (Burghardt and Gordon, 1990).
However, the truth is probably much more complex and -- like Riverside -- the success of CET probably lies in a combination of factors.[43] First, the CET program has been in San Jose for a long time, and has long-standing connections with employers that facilitate its finding placements for its students.[44] Second, the program concentrates on Hispanics, and most of the instructors are both Hispanic and bilingual; the program is therefore providing bilingual education in addition to job skills instruction, as well as mentoring and acculturation to American practices for those individuals who have just immigrated. Third, the site at San Jose performs real work -- for example, it operates a child care center, a copying business, a cafeteria for the CET members themselves, an auto repair shop, and the like, each associated with one of the job training programs -- so that students are getting work-based training and experience as well as classroom instruction in both job skills and remedial subjects. Fourth, the presence of social services at the site -- child care, but also assistance with immigration issues and job placement -- is clearly important. While CET does provide both remediation and job skills training, and is particularly conscious of the need for English language instruction, the two are not integrated in any important sense -- they take place at different times of the day, and there is rarely any mention in one component of the program of the lessons from the other component -- so the emphasis on CET as an "integrated" program seems misplaced.[45] But there are clearly many other positive elements to CET and a broad range of job-related services provided, and it is easy to understand why it is so much more effective than other job training programs.
The conclusion from both the Riverside program and CET is that a combination of practices distinguishes particularly successful programs. I shall return to this finding in the conclusion, since certain current recommendations for job training programs build on the findings presented in this section.
[41] Of course, in random-assignment evaluations the effects of creaming will be eliminated because the experimental and the control groups have roughly the same characteristics by construction.
[42] Bloom et al. (1993, Exhibit 7.12). This analysis was based on the 18-month results, which may be too early, and was not replicated for the 30-month results.
[43] These comments are based not on the published descriptions of the CET program, but on several visits by Judy Kalman and Norton Grubb to CET programs in San Jose and Oakland in the course of research for Grubb et al. (1992).
[44] However, it is quite possible that CET has become a trusted supplier of labor to the low-wage employers, who then hire through CET rather than other sources; in this case the employment effects of CET might be offset by displacement, or reductions in employment from other sources.
[45] The pattern of teaching remedial education and job skills in different and independent segments is typical of job training programs that claim to "integrate" the two; they really mean that they provide both, not that the two are integrated. Like most job training programs, the teaching methods used at CET are quite conventional, teacher-directed methods following the practices of "skills and drills".
The results presented so far describe the outcomes of job training programs, and whether they improve the employment rates and earnings (or other potential outcomes like arrest rates, welfare receipt, and fertility behavior). A different way of asking whether job training programs are "worth it" is to compare the outcomes to the costs, though benefit-cost analysis. Since the late 1970s, many job training programs have been subjected to cost-benefit analyses, using generally-accepted methods to establish values for the net present value of the outcomes that could be attributed to the program -- that is the difference in outcomes between those enrolling in the program and a control group. In most cases the additional earnings of those enrolled and the reduction in welfare benefits represent the major benefits, though in some cases the value of crime prevented and drug and alcohol abuse avoided are counted among the benefits (e.g., in the cost-benefit analysis of the Job Corps, in Table 20). To be sure, there are numerous intangible benefits of job training, especially that associated with greater economic independence and reduced use of welfare programs; and there are probably uncounted costs as well, particularly the opportunity costs of mothers with young children, who might otherwise be caring for their children.
Typically these analyses calculate costs and benefits separately for different groups of potential beneficiaries, usually including those enrolled and the "rest of society" (or taxpayers), in order to capture the distributional effects of job training. That is, it is possible that such programs benefit those enrolled while taxpayers lose, or conversely that the benefits to taxpayers through decreased welfare and crime costs outweigh the costs of the programs, even though those enrolled do not earn enough more to offset the loss of welfare benefits.[46] The hope of job training programs, of course, is that both taxpayers and those enrolled will enjoy benefits in excess of costs.
One of the earliest and most influential cost-benefit analyses was that of the residential Job Corps program, summarized in Table 20. The results of this analysis are fairly typical of subsequent analyses: overall the net benefits to those enrolled in the Job Corps were positive, because the increased earnings per person ($3,397) were high enough to offset the lost of welfare income ($1,357), the opportunity costs associated with being unable to work during the period of the program ($728), and other costs associated with being in the program. But the Job Corps was a net loss to taxpayers, since the very high cost of the program ($5,351 in 1977 dollars, and about $15,000 now) was so large that the benefits from taxes on increased earnings ($1,255), reductions in welfare benefits and administrative costs ($1,515) and reduced criminal activity ($2,281) failed to offset it. From a social point of view, adding the benefits to Corps members plus taxpayers, the benefits outweighed the costs; the general lesson from this analysis was that even an expensive job training program could demonstrate its worth in cost-benefit terms.
Similarly, the welfare-to-work programs developed during the 1980s were evaluated using benefit-cost analyses. Table 21 presents the results of a typical and particularly influential analysis, the San Diego program, for both a work experience program (EWEP) and job search assistance. The results, as for the Job Corps, indicate net benefits for those enrolled in the program, as earnings increase by a small amount (e.g., $461 for work experience and job search together), but net losses for taxpayers since the costs of the program were higher than any reduction in welfare payments and increases in taxes from increased earnings. Since from a social perspective the benefits outweigh the costs, the results of this experiment were widely cited as confirming that welfare-to-work programs were worth undertaking, and were widely cited in legislation for the JOBS program. However, the fondest hopes of those hostile to welfare -- that such job training programs would save money for taxpayers -- were clearly not met by the San Diego program.
In the more general analysis of the JTPA program, the cost-benefit results are of course consistent with the findings of the outcome studies. For adult women and adult men, for whom there were significant increases in earnings, the benefits outweigh the costs both for those enrolled in the programs and for society, as presented in Table 22 -- though again taxpayers lose. But for youths, for whom the effects on earnings were negative because of reductions in earnings during the period of program, the program generated net losses of participants as well as taxpayers, with losses per person especially high ($2,904) in the case of young males. The only conclusion possible is that job training programs are a particularly poor investment for youth, and indeed, since these results became public there have been numerous proposals to reduce or eliminate job training efforts for youth.
The benefit-cost analyses of the GAIN program in California, summarized in Table 23, confirm that different programs of varying effectiveness can generate different conclusions in cost-benefit analyses. This analysis distinguished among three groups of potential beneficiaries: welfare recipients enrolled in GAIN; governments supporting the costs of GAIN; and taxpayers, who also pay for GAIN through their taxes but who experience certain benefits and costs -- for example, the output associated with unpaid work experience -- that do not directly affect government budgets. The results indicate that both government budgets and taxpayers lose for all groups of recipients; only in the case of recipients not needing basic remedial education -- which requires relatively expensive classroom training and not simply low-cost job search assistance -- did the benefits outweigh costs both to recipients themselves and to taxpayers. But the detail for specific county programs reveals that the most effective programs -- those in Riverside and in San Diego -- can generate savings for taxpayers as well as substantial net benefits for recipients; indeed, in Riverside the benefits to governments and to taxpayers are larger than those to recipients. (And, on the contrary, the worst program -- that in Los Angeles County -- actually made recipients worse off, as JTPA did for youth.) From these results, therefore, the hopes for welfare-to-work programs lies in their being able to emulate the characteristics of the most effective programs since on the average the GAIN program generated net losses to society.
The results for Project Independence are somewhat different than they are for virtually any other job training program (see Table 24). Because this program did not increase earnings by much, but did reduce welfare benefits, the net effects for those on welfare were actually negative. However, because the program did not cost much (an average of $1,150 per person) and did reduce welfare costs by a considerable amount (about $1,155, according to the figures in Table 24), the program resulted in a small savings to taxpayers[47] -- quite the opposite of the results for GAIN and most other job training programs, where recipients gain but taxpayers lose. However, overall Project Independence generated losses to society since the very small gains to taxpayers did not outweigh the high losses to welfare clients.
Of course, any positive cost-benefit results depend on job training programs with significantly positive effects on employment. In the case where benefits are essentially zero, then of course benefit-cost analysis indicates net losses to society. Table 25 presents the benefit-cost results of JOBSTART, which had largely insignificant results (in Table 14). Consistent with these findings, the net benefits to participants were very small, and the losses to taxpayers and to society as a whole were substantial.
On the whole, however, the results of cost-benefit analyses of job training programs are relatively consistent -- at least if we ignore the evidence from the least effective programs (e.g., JTPA for youth), the most effective programs (like Riverside), and the different case of Project Independence. In general, the benefits of job training programs outweigh the costs since the modest increases in earnings are larger than the modest expenditures per person in these programs. However, these benefits accrue largely to those enrolling in job training programs, and governments (or taxpayers) typically do not benefit since their costs outweigh any benefits they receive in the form of higher taxes on higher earnings, reduced welfare costs, and reduced costs associated with crime and other social problems. Overall, then, job training programs are "worth doing" -- but it is unclear whether taxpayers would continue to support these programs if they realized that they are likely to be net losers.
[46] In these cost-benefit analyses, the treatment of transfer payments is carried out correctly; that is, a reduction in welfare benefits is simultaneously a cost to recipients and a benefit to taxpayers, and the overall effect of a transfer on society is zero. The only real savings from reducing welfare programs therefore comes from reducing administrative expenses, as well as from the intangible benefits of having fewer individuals rely on welfare.
[47]One could argue that the net gains to the government budget and to taxpayers in Table 24 are so small that they could easily be negative, given random variation in programs.
