| Crain, R. L., Allen, A., Thaler, R., Sullivan, D., Zellman, G., Little, J. W., and Quigley, D. D. (1999). The Effects of Academic Career Magnet Education on High Schools and Their Graduates (MDS-779). Berkeley: National Center for Research in Vocational Education, University of California. |
APPENDIX A
Additional Notes on the Methodology
We described this study as an experiment-based or random assignment-based research project because its basis is the random separation of lottery winners from lottery losers in the eighth grade before the students enter high school. In all the research, we compare lottery winners to lottery losers, letting the difference between the two groups be our best answer to the question, "Do career magnet students receive a better education than they would get in a comprehensive school?" Merely being a lottery winner or lottery loser does not guarantee that the student will attend or graduate from a career magnet program or a comprehensive school, however.
The lottery plays an important role in removing many of the unmeasurable effects of family background and school experiences before the end of eighth grade. By comparing students who won and lost the lottery after applying to the same career magnet school, we also control all the students' aspirations and interests as of the end of eighth grade. We strengthen the match on attitudes by using as our pool only those students who applied to a career magnet school as their first high school choice. (The lottery always gives preference to a student who gave the school as a first choice over one who gave it as a later choice.) The decisions students made after that, however, do introduce biases.
The first student decision is whether to remain in the public schools after receiving word of their high school admission. Lottery losers who are not school-selected are, not surprisingly, slightly more likely to leave the public schools to attend private schools. Some of them move out of the district or use a false address to attend a more prestigious public school in the area outside that served by the career magnets. (There may also be a small number of eighth Graders who elect not to go to high school at all, but this would be illegal for all but those who are considerably overage.) Since the students who are able to attend private schools presumably receive a better education than they would in public comprehensive schools, retaining these students in the study and using them along with all the other lottery losers to estimate the quality of education in comprehensive schools tends to overestimate that quality and make the career magnet schools' performance look worse than it actually is. Fortunately, this bias is relatively small because not very many students leave the public schools.
Secondly, students may stay in the public schools but not go to the school the lottery result dictated. Lottery-winning students have the option of declining admission to the career magnet school; lottery losers can sometimes get an offer from the school when the school reviews the list of those who lost the lottery. (Students are not informed whether their admission is because of school selection or because of lottery selection, so there is no "Hawthorne Effect" introduced.) As Appendix B describes in detail, about 30% of lottery losers are able to attend a career magnet school, while about 20% of the lottery winners choose not to attend their first-choice career magnet school. In our analyses of the effects of the career magnet schools on dropout rates and achievement test scores, we compare all lottery winners to all lottery losers, so that we do not introduce any bias due to student self-selection. In doing so, we underestimate the difference between the effects of the career magnets and the comprehensive schools, but not so much as to bias the result by changing the direction of the difference. It could only create a false direction if, for example, lottery losers who attended comprehensive schools got a fine education, but those who chose career magnets chose only career magnets with very poor quality, thereby making the lottery-losing pool have worse outcomes than the lottery winners and letting us falsely attribute this to the lack of quality of the comprehensive schools. We think this highly unlikely, since, in many cases, the lottery losers who attend career magnet schools are probably attending career magnet schools of about the same quality as the career magnet schools attended by lottery winners. (Indeed, in many cases, these students will receive an offer from their first choice school, the same one for which they lost the lottery admission). The lottery winners and lottery losers are equally likely to attend highly selective academic schools, which in this area are very prestigious, so that most lottery winners would elect those schools over their career magnet first choice. (The academic career magnets do not use lottery admission.) These assumptions correspond to setting these effects to zero in a structural-equation modeling of this process.
Appendix B considers the issue of bias introduced at this stage and concludes that there is no strong reason to believe that these decisions bias the data in favor of or against the career magnet schools; it also concludes that the measured difference between lottery winners and lottery losers is considerably less than it would be in an experiment in which every lottery winner was required to attend a career magnet school and every lottery loser required to attend a comprehensive school. Where lottery winners perform better (or worse) than lottery losers, we believe that this difference means that career magnets are providing a better (or worse) quality of education, and the actual difference in quality, whether positive or negative, is probably twice as large as the difference between all lottery winners and all lottery losers. For example, the positive impact of career magnets on college credits earned is probably twice as large as it appears to be in Table 5.4, and the negative impact on graduation rates due to attending a career magnet school is probably also twice as large as what we found.
Another self-selection bias appears when we study graduates of the program; since the career magnets have a lower graduation rate, we would expect a random sample of career magnet graduates to have better postgraduate outcomes than graduates of comprehensive schools. We attempted to minimize this difference by drawing a sample of matched pairs, matching the students on eighth grade achievement test scores and other factors. Some of the differences in the performance after graduation of career magnet and comprehensive graduates are much too large to be explained away by any possible bias due to the lower graduation rate from career magnet schools.
The Survey Design
A survey of 110 career magnet school graduates was conducted according to a careful matching plan designed to ensure that the random assignment used to either select or reject students from a particular program was maintained. At the time of selection into the survey study, each potential interviewee had to have graduated from high school within the previous two years; to have scored in the mid-range on reading tests; and to have been enrolled in high school in regular classes, with no special education placement. Lottery winners and losers were matched on their first choice of high school, on age, and on seventh- and eighth-grade school performance. Although we hoped to match interview pairs on the junior high school attended, we were unable to do so. Nor were we able to obtain an even balance of 110 graduates based on gender. There were two reasons for this. First, 60% of our original cohort of 9,174 students were female. Secondly, two of the four career magnet high schools from which we drew the sample of graduates to be surveyed had predominantly female enrollments. Consequently, we had to oversample males from the other two career magnet programs. Nonetheless, our final sample was still overwhelmingly female (72% to 38%). The race/ethnicity of the lottery winners and lottery losers were closely balanced.
Because of confidentiality requirements, initial contact with a pool of potential interviewees for the graduate survey had to be conducted through letters sent by counselors from their high school asking if they might be willing to participate in the interview. Interviewees were paid $40 for their participation, which generally lasted between two and three hours. Interviews were conducted from May 1993 through May 1994. In most cases, interviewees were living in the general area. These interviewees were given the option of either coming to the college for their interview session or having a project interviewer meet them at their home to conduct the interview. In most cases, interviews were conducted at the college; in some cases, interviews were conducted when college students living away from home returned for vacation; and, in a few instances, project interviewers were dispatched to colleges outside the area to conduct interviews with graduates unable to return.
In nearly all cases, interviews were conducted by interviewers who matched the interviewee on race or ethnicity, age, and gender.
|
| Crain, R. L., Allen, A., Thaler, R., Sullivan, D., Zellman, G., Little, J. W., and Quigley, D. D. (1999). The Effects of Academic Career Magnet Education on High Schools and Their Graduates (MDS-779). Berkeley: National Center for Research in Vocational Education, University of California. |