Crain, R. L., Allen, A., Thaler, R., Sullivan, D., Zellman, G., Little, J. W., and Quigley, D. D. (1999). The Effects of Academic Career Magnet Education on High Schools and Their Graduates (MDS-779). Berkeley: National Center for Research in Vocational Education, University of California.

APPENDIX B
Using the Experimental Results to Estimate the Impact of Career Magnets on Students

      We have seen that a randomly selected group of students who were offered admission to career magnet schools had generally different educational outcomes than did another randomly selected group of students who were not offered admission by the lottery process. Since the two groups were identical (disregarding sampling error), except that one group was more likely to attend a career magnet school, it follows that the differing outcomes must have been caused by the career magnet schools. So much is obvious. There are two less obvious questions, however: (1) How large is the effect of the career magnet program on the average student? and (2) To what population of students can these results be generalized?

      To make the experimental design work, it was necessary to compare everyone who was randomly admitted to career magnets to everyone who was randomly passed over. If every student who had won the lottery had attended a career magnet school, and every student who lost the lottery had attended a comprehensive school, then the difference in educational outcomes between the lottery winners and lottery losers would be exactly equal to the difference in the school effect of attending a career magnet rather than a comprehensive school. Unfortunately, in comparing lottery winners to lottery losers, we are not just comparing students who were in career magnets to students who were not because some lottery winners chose not to attend career magnets, and many lottery losers found other ways to gain entrance to career magnets or other selective schools. This means that the experimental design will in all probability underestimate the effect of the career magnets. The task of this appendix is to calculate the size of this underestimation. We can do that by writing an equation that decomposes the difference between the experimental "treatment group" (the lottery winners) and the "control group" (the lottery losers) into the school effects of the different kinds of schools the students in the experiment attended.

      Knowing how many students are in each type of program and the overall difference in the performance of lottery winners and lottery losers, we divide students into subgroups and estimate the difference between lottery winners and lottery losers for each subgroup. There are seven subgroups; they are constructed from Table B.1, and are shown in Table B.2.

      Table B.1 allows us to divide the population of career magnet applicants into the seven separate subgroups in Table B.2, based upon whether they won or lost their lottery and what schools they attended. We do this in order to see which of these seven subgroups received differing treatments as a result of winning or losing the lottery (i.e., went to different types of schools) and, thus, contributed to the difference in student outcomes that we found. The point of this exercise is to identify subgroups of students who, in fact, wound up going to the same types of schools whether they won the lottery or not, since these students cannot have different outcomes as a result of winning the lottery, and, hence, are only "dead weight" in the experiment. The subgroup code numbers refer to rows in Table B.2.

      Together, these four groups make up a total of 49% of all lottery winners and, of course, make up 49% of all lottery losers as well (ignoring sampling error), since both groups are random samples from the same population. The type of school these 49% attend is unaffected by the lottery because they would have made exactly the same decision as to type of school whether they won the lottery or lost it. The remaining 51% were affected by the outcome of the lottery: winning or losing the lottery caused them to change the type of school they enrolled in.

Subgroup  5:
      Similarly, we see that whereas 8.3% of lottery losers attended vocational schools, only 2.7% of lottery winners did so. This implies that 5.6% (8.3-2.7) of lottery losers would have preferred the career magnet program they applied to over the vocational school which accepted them. Thus, we have identified a subgroup, making up 5.6% of both lottery winners and lottery losers, who would choose a career magnet if they won the lottery and a vocational school if they lost the lottery. Let us call this subgroup "changing vocational students."

      Taken together, these seven subgroups make up 100% of the lottery winners and 100% of the lottery losers. We list for each group in Table B.2 a coefficient ("imp") which measures the impact of the type of school they attended on some unspecified outcome, a, b, . . . e for each subgroup which experienced career magnet education and u . . . z for those who experienced some other type of schooling. Since each subgroup is made up of different students, it is possible that the impact of a career magnet would be different for each subgroup.

      The impact of a school may be different for different types of students; a career magnet may have effect "e" on a student who would have gone to a comprehensive school if they had not won the lottery, and an effect "d" on a more vocationally oriented student who would have entered a vocational school if he had not won the lottery. We also assume that in some rows of the table, the lottery outcome would make no difference in a student's educational outcome. For example, the same school effect coefficient, "y," appears in both the third and fifth column of the sixth row. For a student who applied to a career magnet, but whose first choice really was their comprehensive school (certain comprehensive), we assume that the honor of being offered a seat at a career magnet might engender a temporary feeling of pride, but this would not be a powerful enough emotion to affect their ninth-grade performance at their comprehensive school.

      If "X" is the measured educational outcome of lottery winners and "Y" the outcome of lottery losers, then changing the percentage distribution to decimals and summing up the effects in Table B.2 gives us equations (1) and (2):

      (1) X = .300b + .021u + .027c + .027 w + .056d + .143y + .427e

      (2) Y = .300a + .021u + .020v + .027w + .056x + .143y + .427z

      The experimental difference between lottery winners and lottery losers is then, after canceling identical terms,

      (3) X - Y = .300 (b-a) + .020 (c-v) + .056 (d-x) + .427 (e-z)

      (For this analysis, we assume that coefficients measure the effect of attending a given school, and that the effect of being selected but not attending is zero.)

      The goal of this study is to estimate (e-z), the effect of attending a career magnet versus attending a comprehensive school. In order to estimate (e-z), we must make assumptions about the size of the other coefficients.

      As a first step, we asked what the most plausible set of assumptions might be. We assumed that the difference in the educational outcomes among lottery winning and lottery losing certain career magnet students (b-a) would probably be small relative to some of the other terms. The certain career magnet students who lost the lottery are highly likely to have been school selected by their first-choice career magnet schools, the same schools they would have attended had they won the lottery. Since they would not know whether they were school selected or randomly selected, there cannot be any differential effect. The only effect would be for the students who were school selected or lottery selected by their second or less-desired choice but who, had they won the lottery, would have gotten into their first choice. The effects of the educational quality of these second-choice career magnet schools versus the first-choice schools would mostly cancel out (since one person's first choice is another person's second). For a portion of the 30%, there would be a motivational effect of not getting one's first choice, but we are inclined to assume that this effect is relatively small.

      Examining the rest of equation (1), we were inclined to assume that for academically oriented students, the effect of being in a selective school, "v," is not much different from being in a career magnet, "c"; however, the effect of being in a vocational school, "x," would be considerably less than the effect of being in a career magnet, "d." Conversely, for students with a strong academic career focus, being in a selective school may be much less beneficial than being in a career magnet, while being in a vocational school may not represent a great loss. We still would assume, however, that students would not improve their educational outcomes as much in a vocational school.

      It seems quite reasonable to assume that comprehensive schools are educationally weaker than selective schools and probably do not motivate students the way a vocational school does. Thus, it is safe to assume that the differences (b-a), (c-v), and (d-x) are all smaller than (e-z).

      In examining the assumptions we made, we saw that they split the difference between two extremes. At one extreme is the assumption that being in one's first choice career magnet means a much better educational outcome than being in any other kind of school, even a selective academic school. At the other extreme was the assumption that getting into any sort of career magnet school, even a vocational school, had an equally positive effect. Thus, we decided to algebraically estimate the implications of both extreme assumptions.

      If we assume, first, that career magnets are not better than either selective or vocational schools, and attending one's second (or less desired) choice career magnet is not harmful, then in equation (1) the terms (b-a), (c-v), and (d-x) would all become 0, and equation (3) would simplify to become equation (4):

      (4) X - Y = .427 (e-z)

      At the other extreme, we can assume that being randomly selected into almost always one's first-choice career magnet program is superior to attending either a selective, vocational, or comprehensive school, or even one's second-choice career magnet school. We assume the benefit over all these other types of schools is the same; we also assume that one-third of the career magnet students who lose the lottery do not get into their first-choice career magnet. If so, then in equation (3) we can set (b-a) = (c-v) = (d-x) = (e-z), and change the coefficient .300 to .100. Then the experimental effect found in the experiment would be given by equation (5):

      (5) X - Y = (.100 + .020 + .056 + .427) (e-z) = .603 (e-z)

      Perhaps the most reasonable thing to do is to split the difference. Let us assume X - Y = approximately .5 (e-z), and our conclusion is that the effect found in the experiment, X - Y, is about half the size of the effect on a student of being in a career magnet program instead of a comprehensive school.

      It is important to look at the ratio of the rate of growth in career magnet schools to the rate of growth in comprehensive schools. If X represents the performance of lottery winners, Y represents the performance of lottery losers, "M" represents the effect of attending a career magnet, and "N" represents the effect of attending a neighborhood comprehensive school, and we assume (based on B.1) that 80% of lottery winners experience educational benefits like those received from a career magnet education while only 30% of lottery losers do, then the educational effects on lottery winners and lottery losers are as follow:

(6) X = .8M + .2N

(7) Y = .3M + .7N

      Solving these two equations for M and N, we conclude that the true ratio of the effect of career magnet schools, M, to the effect of comprehensive schools, N, is

(8) M = .7X - .2Y

N    .8Y - .3X

      This equation implies that if lottery winners have outcomes 25% higher than lottery losers, then we should expect students in career magnets to have outcomes approximately 60% higher than students in comprehensive schools. If the lottery winners have scores 50% higher than those of lottery losers, we should expect students in career magnets to have outcomes about two-and-one-half times greater than students in comprehensive schools.

      This analysis has also answered our second question: To what population can we generalize our results? The particular group of students for whom we can estimate effects are only those whose effects are e and z. Any attempt to solve equation (3) for any of the other school effects would require making very extreme assumptions, including assuming the size of coefficients e and z. The only reasonable assumption is that any difference between X and Y is overwhelmingly the result of a difference between e and z. Because of this, our conclusion is that the impact of career magnets is on students who, if they were not selected randomly, would not have gotten into any career magnet school.

Crain, R. L., Allen, A., Thaler, R., Sullivan, D., Zellman, G., Little, J. W., and Quigley, D. D. (1999). The Effects of Academic Career Magnet Education on High Schools and Their Graduates (MDS-779). Berkeley: National Center for Research in Vocational Education, University of California.