National Center for Research in Vocational Education
Graduate School of Education
University of California at Berkeley
2030 Addison Street, Suite 500
Berkeley, CA 94720-1674

Supported by
The Office of Vocational and Adult Education
U.S. Department of Education

January, 1997

FUNDING INFORMATION

• ACKNOWLEDGMENTS
• EXECUTIVE SUMMARY
• INTRODUCTION
  • Assessing Student Outcomes for Tech Prep
• METHODS
  • Population and Sample
  • Instrumentation and Procedures
  • Data Collection, Analysis, and Interpretation
• FINDINGS AND DISCUSSION
  • The Perspectives of All Participants Toward
    Tech Prep Student Outcomes
  • The Perspective of Each Stakeholder Group Toward
    Tech Prep Student Outcomes
• IMPLICATIONS FOR POLICY AND PRACTICE
• REFERENCES
• NOTES
• APPENDIX A
  LIST OF STUDENT OUTCOMES BY CLASSIFICATION IN THE LITERATURE
• APPENDIX B
  A COMPARISON OF MEAN RATING AND BRIDGING VALUES
  FOR OUTCOMES STATEMENTS BY CLUSTER AND STAKEHOLDER GROUP

ACKNOWLEDGMENTS

Debra D. Bragg

EXECUTIVE SUMMARY

1. Collectively, how do the three stakeholder groups of educators, students, and employers conceptualize and prioritize Tech Prep student outcomes?
2. What are the similarities and differences in how each of the three stakeholder groups of educators, students, and employers conceptualize and prioritize Tech Prep student outcomes?

Results showed the three stakeholder groups of educators, students, and employers gave high priority to a wide array of student outcomes. It seems nearly everything one might think of as associated with a modern high school education is seen as important for Tech Prep. In fact, all three stakeholder groups rated nearly all of the 98 student outcomes statements at a "moderate" or "high" priority level even though they were instructed to spread the ratings of the outcomes statements across the 5-point priority rating scale. In addition, results showed there were many more similarities than differences in how the three groups conceptualized and prioritized Tech Prep student outcomes. A nine-cluster concept map was deemed the most logical way to represent the results for all participants. This concept map contained the following clusters (mean cluster rating on 5-point priority scale in parentheses):

1. Personal attributes, attitudes, and employability skills (4.13)
2. School-to-work transition (3.96)
3. Technology and quality management (3.92)
4. Information use and decision making (3.89)
5. Work and interpersonal relationships (3.75)
6. Educational attainment (3.68)
7. Communications (3.61)
8. Math and science (3.46)
9. Democratic and participatory strategies (3.29)

Beyond these areas, some important differences in how the three stakeholder groups perceived the Tech Prep student outcomes were apparent. Particularly in sorting and rating outcomes related to education, and specifically academic subjects, there was a great deal of disparity in the ways the stakeholder groups perceived student outcomes. For example, educators and students gave higher priority ratings than employers to sets of educational attainment outcomes such as to graduate from high school, make progress on grade level, and graduate from two-year postsecondary college. Employers gave slightly higher priority to clusters of vocationally oriented outcomes, although all three stakeholders tended to give vocationally oriented clusters high priority ratings while academically oriented outcomes received lower (albeit not low but moderate) ratings.

Clusters linked to the academic areas of social studies and humanities received the lowest ratings. In fact, the cluster labeled "democratic and participatory strategies" created by employers rated the lowest of all clusters with an average rating of 2.99. Recalling that the federal Tech Prep Education Act specifies that Tech Prep be comprised of mathematics, science, English/communications, and vocational-technical education, this rating may not be surprising. Similarly to the federal law, most local or state policies associated with Tech Prep have emphasized math, science, English/communications, and vocational-technical education over humanities or other liberal studies. Consequently, the stakeholder participants' responses may reflect a bias in the public policy, influencing how respondents rate various vocational and academic outcomes statements. Of course, this study only examines perceptions and not actual implementation. Therefore, it is not possible to determine whether respondents have experienced a shift in curriculum focus.

Related to this concern, many local consortia and state agencies profess a primary purpose of Tech Prep is to "eliminate the general track," following a vision of Tech Prep articulated by Parnell (1985). Although difficult to determine from this data, it is possible that reforming the general track by emphasizing math, science, and technology may lead to less emphasis on the traditional social and democratic functions of public education. There is only so much time in a school day. Yet, even though the data suggests such a prioritization is occurring among the Tech Prep student outcomes, it is difficult to believe these kinds of ideas are explicit to the respondents in terms of tradeoffs of courses and content (subject matter) within the curriculum. Certainly, more research is needed to understand the actual changes occurring within the curriculum and subsequent effects on students.

In summary, this study attempted to better understand Tech Prep student outcomes from the perspectives of educators, students, and employers actively engaged in implementing Tech Prep. Knowing how these groups conceptualize student outcomes has important implications for understanding the fundamental objectives of Tech Prep, for planning and implementing Tech Prep and related school-to-work programs, and for assessing outcomes. Also, by uncovering various conceptualizations of Tech Prep, it is feasible to identify conflicting perspectives held by disparate stakeholder groups, possibly revealing gaps in the logic that underpins the Tech Prep approach. Using this study as a model, further research could be conducted with still more stakeholder groups (e.g., policymakers, administrators, counselors, parents) and with other localities such as rural and suburban areas. As Tech Prep and school-to-work program implementation continues, more attention must be devoted to student outcomes. Only by better understanding various stakeholder perspectives can future evaluations and outcomes assessments be expected to produce results useful to the nation's goal of reforming education.

INTRODUCTION

Each of these issues is applicable to educational reforms associated with the nation's Tech Prep[1] and related school-to-work opportunities programs.[2] These concerns are apparent in all areas, but particularly in urban and rural areas where educational reform has been particularly difficult. Issues related to implementing effective school-to-work related reforms in urban areas are well-documented in a special issue of Education and Urban Society, edited by Seidman and Ramsey (1995). In that issue, Bragg and Layton (1995) point out that both pieces of federal legislation, the Tech Prep Education Act and the School-to-Work Opportunities (STWO) Act, contain directives to ensure funding is distributed to urban and rural schools and two-year colleges. However, like most other reforms, little is known about the quality of these programs or their impact on students in urban or rural areas.

In most localities, Tech Prep and school-to-work programs have been in operation for less than five years (Bragg, Layton, & Hammons, 1994). As a consequence, too little time has passed for students to have completed an entirely reformed secondary-to-postsecondary Tech Prep program. In many areas of the country, high school graduates participating in Tech Prep only first began to matriculate to the postsecondary level during the 1994-1995 or 1995-1996 school years, creating an unsettling lack of information regarding student outcomes. The National Assessment of Vocational Education (NAVE) drew a similar conclusion regarding the need for systematic evaluation of Tech Prep. Boesel, Rahn, and Deich (1994), authors of the Tech Prep section of the NAVE report, recommended increased emphasis on evaluation of Tech Prep programs "using longitudinal studies of student participation, retention, and educational and employment outcomes" (p. 131). They urged government to collect better information regarding how students participate in Tech Prep at both the secondary and postsecondary levels, and how they progress through the system. The study also emphasized the need for more Tech Prep programs to be developed in sites with high concentrations of special population students and to ensure evaluation is employed to monitor the effectiveness of Tech Prep programs for these students. Often, urban schools and colleges enroll disproportionately large numbers of special population students, making it extremely important to understand how Tech Prep involves and affects these students.

To provide a foundation for future efforts to evaluate programs and assess student outcomes relative to Tech Prep, additional research was needed. This study was designed and conducted to identify, classify, and prioritize student outcomes for Tech Prep. To address concerns that would inevitably be raised concerning stakeholders' differing perspectives toward student outcomes, three stakeholder groups were engaged in the study. They were educators, students, and employers.[3] The research questions that guided the study follow:

1. Collectively, how do the three stakeholder groups of educators, students, and employers conceptualize and prioritize Tech Prep student outcomes?
   • What level of priority does the collective group assign to each student outcome?
   • How does the collective group organize and classify (concept map) the student outcomes?
   • What level of priority is attributed to the clusters of student outcomes that emerge in the concept map?
2. What are the similarities and differences in how each of the three stakeholder groups of educators, employers, and students conceptualize and prioritize Tech Prep student outcomes?
   • What are the similarities and differences in the priorities attributed to the student outcomes by each of the three stakeholder groups?
   • What are the similarities and differences in how each of the three stakeholder groups organize and classify (concept map) the student outcomes?
   • What are the similarities and differences in the priorities attributed to the clusters of student outcomes of each of the three stakeholder groups?

With better information about student outcomes, still other benefits are feasible. Researchers, policy leaders, and practitioners may be more likely to determine the circumstances under which the fundamental elements of Tech Prep (e.g., articulation, applied academics, stakeholder collaboration, and education-business partnerships) are most tenable. In addition, information produced by outcomes assessments can contribute to further refinement of the fundamental Tech Prep concept. Making Tech Prep more accessible and effective for "all" students rather than limiting it to the "neglected majority" is a particularly important issue for urban localities such as those engaged in this study. Finally, having better quality information about student outcomes can help to build more accountability into any evolving Tech Prep system, resulting in better program implementation and evaluation at all levels.

Assessing Student Outcomes for Tech Prep

Similarly, Dale Parnell's vision of Tech Prep focused on what he thought should be the key elements of the Tech Prep process, but he identified few student outcomes. In his book The Neglected Majority (1985), Parnell advocated high-quality vocational education, applied academics, and strong relationships between business and education. He argued forcefully to refocus schooling to better meet the needs of the "neglected majority" of high school students who were unlikely to obtain the baccalaureate degree. The 2+2 Tech Prep Associate Degree (TPAD) model conceived by Parnell and further developed by Hull and Parnell (1991) was envisioned to be an equivalent track in rigor and stature to college prep.

In the TPAD model, a common core of "math, science, communications, and technology--all in an applied setting" (Parnell, 1985, p. 144) was to be taught during the last two years of high school and the first two years of postsecondary education at a community, junior, or technical college. Ultimately, this articulated and applied secondary-to-postsecondary educational track was intended to culminate with a two-year associate degree, "the preferred degree for employers seeking to fill a broad range of mid-level occupations," according to Parnell (p. 145). In that statement, Parnell identified a student outcome that is widely associated with Tech Prep: completion with a two-year college associate degree. Other student outcomes were not as clearly specified in the writings of Parnell or others influential in formulating the Tech Prep education approach.

The Tech Prep Education Act

a combined secondary and postsecondary education program which--

(A) leads to an associate degree or 2-year certificate;

(B) provides technical preparation in at least 1 field of engineering technology, applied science, mechanical, industrial, or practical art or trade, or agriculture, health, or business;

(C) builds student competence in mathematics, science, and communication (including applied academics) through a sequential course of study; and

(D) leads to placement in employment. (U.S. Congress, P.L. 101-392, 1990)

States are also advised to encourage local Tech Prep programs to address dropout prevention and re-entry, the needs of minority youths, youths of limited English proficiency, youths with handicaps, and disadvantaged youths (U.S. Congress, 1990). An "essential element" of the legislation requires that special populations be ensured equal access to the full range of Tech Prep programs, including support services. As such, the federal legislation ensures that Tech Prep not be limited to a select group of students such as the neglected majority or the traditional college-bound, but be inclusive of all students.

It is important to note that while the federal legislation provides some direction in terms of the kinds of student outcomes that should be assessed relative to Tech Prep, the law does not specify that outcomes assessment or any other form of program evaluation be carried out at the local or state levels.[4] Although, according to Layton and Bragg (1992), when Tech Prep programs first began to be implemented, several states built evaluation into the system of performance measures and standards required by the Carl D. Perkins Vocational and Applied Technology Education Act of 1990 (commonly referred to as "Perkins II").[5] However, at that time, only 40% of the states identified student outcomes for local Tech Prep programs, and most often that outcome was academic skill attainment, an outcome mandated by Perkins II. Yet, even then, state officials were questioning how to measure academic attainment and few other student outcomes were being proposed. (For additional discussion of how states have conducted evaluation within the current political context of Perkins II, see Hoachlander & Rahn (1992); McCaslin & Headley (1993); and Stecher, Hanser, & Hallmark (1995). Even though these studies are not directed toward Tech Prep specifically, they do examine how local and state entities have implemented related student outcomes measures according to the Perkins II mandate.)

Formal Evaluations of Tech Prep

In 1993, two years after federal funding became available to plan and implement local Tech Prep programs, local coordinators were asked to rate the stage of implementation of evaluation in regard to Tech Prep programs funded with federal Title IIIE funds (Bragg et al., 1994). Of nearly 50% of all local Tech Prep consortia in the United States, 40% reported they had not even begun to implement formal evaluations of their Tech Prep programs. Another 30% indicated their consortia were in the planning stage of evaluation, showing only a minority of Tech Prep consortia were actively implementing formal evaluations, and most of these were very, very preliminary.[6] Overall, all evaluation-related activities were rated among the lowest of 30 potential components of a Tech Prep program, indicating evaluation continued to be neglected within the first year or two that Tech Prep programs acquired Title IIIE funds.

Generally, indicators of student performance relative to Tech Prep have been compliance-oriented for the purposes of demonstrating accountability to governmental units rather than for improving local programs (Dornsife, 1992). Documentation of student enrollments and program completion primarily at the secondary level but also the postsecondary level have been used most extensively. A doctoral dissertation completed by Hammons in 1992 reported similar findings. However, Hammons did identify outcomes for Tech Prep related to student careers, attitudes and perceptions associated with education and employment, broadening the pool of outcomes that could be associated with student participation in and completion of Tech Prep. This study made an important contribution to the literature in that it did not confine itself to a narrow set of outcomes, but, rather, considered a wide array of potential performance indicators for Tech Prep.

Research conducted by Bragg et al. (1994) concurred with Hammons' earlier conclusions that a broad set of outcomes could be associated with Tech Prep. When local coordinators were asked to rate the priority that should be given to 17 student outcomes, 15 were given a "high" or "very high" priority rating, suggesting evaluations of Tech Prep should be broadly conceptualized and not limited to a few compliance-oriented measures. The following 15 student outcomes were given a "high" or "very high" priority by local coordinators:

1. Improved knowledge and skills in math
2. Improved problem-solving, thinking, and reasoning skills
3. Increased employability skills and work readiness
4. Increased matriculation from secondary to postsecondary levels
5. Increased awareness of and interest in technical careers
6. Improved knowledge and skills in English/communications
7. Increased knowledge and skills in vocational areas
8. Improved knowledge and skills in science
9. Increased motivation for learning
10. Increased secondary school completion
11. Increased interpersonal skills (team, leadership)
12. Increased postsecondary school completion
13. Increased employability in high-wage jobs
14. Increased satisfaction of students and graduates with jobs
15. Increased self-esteem

The National Evaluation of Tech Prep Education

According to Hershey and Silverberg (1994), during FY 1993 all states monitored local Tech Prep implementation by having local consortia make progress reports, usually once or twice per year. These reports typically asked local consortia to document how grant funds were used or how particular processes were functioning, (e.g., staff development, consortium membership, or planning activities). According to Hershey and Silverberg, 30 states required that local consortia report program evaluation activities or results. The majority of states required local consortia to inform them about the number of students in Tech Prep. Just over one-half of the state coordinators indicated their states required data on student outcomes:

State agencies most frequently required outcome data on secondary school program completion (23 states), postsecondary program enrollment (23 states), postsecondary program completion (20 states), and students' academic skills (17 states). Reports on job placements and students' technical skills/competencies were required in 15 and 14 states, respectively. (p. 29)

The national evaluation clearly documents that in FY 1993 very few consortia were engaging students in formal evaluation activities or actively collecting data on student outcomes. When attempting to understand how students move through the Tech Prep system, secondary to postsecondary and beyond, the number of local consortia that were able to provide student outcomes data in the area of participation and completion was so limited as to make most of the estimates meaningless. For example, when asked to provide the number of Tech Prep participants at the secondary level, only 250 of 702 local consortia provided estimates. Far fewer provided estimates regarding high school graduation, employment after high school, postsecondary entry, completion of postsecondary, or employment after postsecondary completion. The national evaluation does not attempt to collect other student outcomes data such as "skills levels, competencies, or grades because they are measured, computed, and interpreted differently across localities" (Bragg et al., 1994, p. 117). Therefore, the census to document Tech Prep implementation nationwide will never report student outcomes beyond student participation and completion. This is a concern since understanding how students benefit from Tech Prep would be useful beyond knowing simply whether they participate and complete prescribed phases of the Tech Prep system.

Here, the national evaluation's primary goal of documenting accountability is apparent, but that may not be as helpful to local and state practitioners as other kinds of evaluation. For example, understanding technical and academic competency attainment among students could help to determine the effectiveness of particular aspects of the school-based curricula. Furthermore, identifying employability skill levels among students could help to determine the quality of the work-based curricula. Fortunately, the related case studies associated with the national evaluation delve into these student outcomes. A report documenting the first in a series of site visits conducted by Hershey, Silverberg, and Owens (1994) focused almost entirely on four processes: (1) articulation; (2) curriculum and instructional enhancement; (3) student recruitment, guidance, and career development; and (4) consortium organization and coordination. Data collected from school records for two cohorts of students in each of the ten case-study sites should help to expand the universe of student outcomes being investigated along with the national evaluation and help to address questions about how students benefit from Tech Prep.

State Evaluations of Tech Prep

Of all the states responding to our request for information about evaluations, five indicated they were primarily participating in and relying on the U.S. Department of Education-sponsored national evaluation of Tech Prep, research on Tech Prep conducted by the National Center for Research in Vocational Education (NCRVE), or other studies to inform them about how Tech Prep is progressing. One of the state coordinators indicated the federal appropriation for Tech Prep was too limited to allow funds to be diverted away from local and state program implementation. Still, most of these states were monitoring Tech Prep implementation as they did other similar programs, and some were engaging local consortia in formal self-assessments as well.

Twenty-eight states were engaging local consortia in data collection, either by having state staff design and carry out the evaluation, usually in conjunction with local personnel, or by contracting the evaluation to a third party. When a third party was chosen, often it was with vocational-technical personnel employed by a state's land-grant university. Several state agencies have established strong relationships with vocational-technical education units in land-grant universities for the purposes of conducting formal program evaluation. Many of these kinds of units are tapped to evaluate Tech Prep programs, including the vocational-technical units in land-grant universities in Illinois, Minnesota, Missouri, Virginia, and Wisconsin. In addition, third-party evaluations were conducted by other universities, regional education laboratories, or private consulting firms in California, Colorado, Florida, Ohio, Texas, Washington state, and West Virginia. Having reviewed the internal and external evaluation documents produced by state agencies and third-party groups, it seemed that evaluations conducted by third-party agencies were more comprehensive and rigorous than the internal evaluations conducted by state agencies. More of the third-party evaluations used a longitudinal design and standardized data collection procedures, and more provided a comprehensible definition of the population and sample of Tech Prep students and other stakeholders engaged in the study.

Thirteen of twenty-eight states reported they were conducting Tech Prep evaluations, but did not provide formal reports showing data, findings, conclusions, or recommendations. Rather, most provided copies of guidelines, surveys, and site-visit instrumentation used to collect data. Most indicated that although data was being collected and sometimes already available for use by local and state personnel, a formal report was not distributed.

Sixteen states provided copies of formal evaluation reports, most reporting results for FY 1994, although the reports for Illinois, Nebraska, New Hampshire, Ohio, Washington state, and West Virginia included FY 1995 data. None of the reports attempted to compare Tech Prep in disparate settings such as rural, urban, and suburban. (See Table 1 for a summary of the goals and methods used to carry out the sixteen formal evaluations of Tech Prep programs reported here. Appendix A contains additional information about each of these state-level evaluations.)

About one-half of the studies were longitudinal in design, typically lasting for three years to comply with the three-year time period for grant awards specified in the federal Tech Prep Education Act. Most of the studies utilized multiple methods--typically a document review, site visits, and surveys involving various stakeholder groups. Sometimes the evaluations also used observational assessments and secondary analysis of data supplied by MPR Associates, Inc., the organization conducting the national evaluation for the U.S. Department of Education. Eight of the thirteen evaluations had plans to examine student outcomes, although often these outcomes were not specified in the evaluation reports. Several of the reports stated that student outcomes could not be examined because of the early stage of implementation of Tech Prep. However, a few evaluations did report findings relative to Tech Prep student outcomes. Several of the state-level evaluations that made such claims are discussed in this section.

In Illinois, an evaluation involving four secondary sites included students classified as Tech Prep, non-Tech Prep, and pre-Tech Prep (Roegge & Evans, 1995). Student data was collected from transcripts, standardized tests, and group interviews. The findings indicated that Tech Prep students took as many or more science, math, social science, and foreign language courses as pre-Tech Prep students. The results were statistically significant for advanced science courses only. Although the group of Tech Prep students had a lower class rank percentile overall than the pre-Tech Prep students, the Tech Prep students obtained significantly higher composite scores on the ACT than pre-Tech Prep students. This result was statistically significant at the p=.005 level. A small sample of students was given a "Work Readiness" instrument developed by the researchers, and the results revealed that Tech Prep students had a more "anticipatory attitude toward work" than non-Tech Prep students. The data also revealed that more Tech Prep students thought their "classes would help prepare them for a career," and they were more "sure of what they wanted to do as an adult" (p. 8).

In Rhode Island, a sample of 1,350 students was drawn from 11th and 12th grade from 24 high schools (Rhode Island Tech Prep Associate Degree Program, no author or date given). The students were grouped into Tech Prep Associate Degree (TPAD) (n=1,115) and non-TPAD (n=235) categories. "The comparison group [of non-TPAD students] was composed of similar students from TPAD schools whose guidance counselors identified them as appropriate candidates for the TPAD Program, but who had declined participation, and similar students from two non-TPAD schools whose faculty were planning to implement the Program during the 1994-95 year" (p. 9). The two groups compared closely on several demographic characteristics such as gender and ethnicity. Data used for the secondary analysis was gleaned from existing transcript and test results. Findings show the TPAD group had significantly higher grade point averages (GPAs) in math, science, and communications than the non-TPAD group, but prior to participation in Tech Prep the TPAD students had significantly lower GPAs in these subjects than the non-TPAD group. The postsecondary participation rate for the TPAD students was 60% compared to 39% for the non-TPAD students, although missing data for both groups raises questions about these estimates. Nevertheless, the rate of participation in postsecondary education suggests a sizable proportion of students are continuing their education beyond high school, an important element of Tech Prep.

Like Rhode Island, Delaware's evaluation of Tech Prep relies heavily on existing student data from secondary and postsecondary sources (Campbell, 1995). Some of the results of the evaluation are presented for Tech Prep versus non-Tech Prep students. For example, the dropout rates for Tech Prep students are lower than for non-Tech Prep students over the 1990-1991 to 1993-1994 time period, 0.39% and 5.0% respectively. Achievement score comparisons for a random sample of Tech Prep and non-Tech Prep students show the Tech Prep group's average scores in advanced skills reading and math were higher than for non-Tech Prep students. Finally, results indicate a steady increase in the number of high school students earning advanced college credits, showing an increasing number of students are accessing college credits while still in high school.

Finally, some student outcomes results are presented in the evaluation report authored by Owens, Lindner, and Wang (1995). In this study, personnel employed by the Northwest Regional Educational Laboratory (NWREL) conducted case studies on four sites in Washington state. The case study involving the Seattle consortium documented several data collection activities focusing on student outcomes. The report showed that in December 1993 there was a higher rate of participation of Tech Prep than non-Tech Prep students (based on self-identification) in career development activities and applied academic courses. In addition, the study reports telephone interview findings obtained by Dr. Mary Beth Celio showing 79% of Tech Prep graduates were enrolled in postsecondary education compared to 66% for other high school graduates. Celio's study also reported the following findings:

Since Tech Prep is particularly focused on the connection with community colleges, it is important to note that 47% of the Tech Prep students went on to the community college, while only 33% of the non-Tech Prep students did so. The fact that nearly identical percentages of each group went on to a four year school (32% for Tech Prep and 33% for non-Tech Prep) demonstrates that Tech Prep does not limit options for attending four year programs. Equally impressive is that the Seattle high school graduates going on to the community colleges include a higher percentage of students who formerly did not progress beyond secondary education (those with a high school GPA of 2.8 or less, Black and Asian populations, and high school graduates age 19 or older). (Owens et al., 1995, p. 17)

METHODS

Structured concept mapping is based on a three-phase model for conceptualizing program theory developed by Trochim and Linton (1986). This model suggests there are three general phases in conceptualizing program theory:

1. Generating a conceptual domain out of thoughts, ideas, intuitions, theories, and problem statements.
2. Structuring a conceptual domain by defining or estimating relationships between and among concepts.
3. Representing the structured set of concepts in a conceptual domain verbally, pictorially, or mathematically.
   
   

Population and Sample

In a letter directed to the coordinators, the purpose of the study was explained, along with the basic requirements and procedures for concept mapping. A nomination form was included with the letter of invitation so that each coordinator could nominate an educator, student, and employer from his or her site who were actively involved and strongly committed to Tech Prep. (See Table 2 for a summary of stakeholder participant affiliations by site.)

Table 3 provides a profile of the three stakeholder participant groups. Briefly, representation by males and females is nearly equal in the study. There is not as equal a representation of other demographic characteristics (although the findings may be representative of the local consortia sites). The majority of participants are White/Caucasian and affiliated primarily with secondary vocational education. However, looking at the data closely gives a unique profile for each group.

Note: UK indicates unknown.

First, the demographic information collected from educators shows that the majority are female, White/Caucasian, and affiliated with secondary vocational education; slightly less than one-third are African American, affiliated with postsecondary education, and engaged in administration. Students, the second stakeholder group, are similar to educators in that most are female; however, the race/ethnicity and educational affiliation of students differ from educators. About one-half of the students are minority, either African American (39%) or Hispanic (11%), and nearly one-half are attending postsecondary education. The third group, employers, is dominated by males and Whites/Caucasians. About one-half of the employer group is comprised of persons working as corporate educational coordinators or human resource directors. Another one-third of the group is composed of independent business owners or business managers, and a smaller proportion is made up of persons working for public or community-based organizations (CBOs).

Instrumentation and Procedures

An attempt was also made to draw ideas from the education reform literature such as the Project 2061 report published by the American Association for the Advancement of Science in 1989; the Curriculum and Evaluation Standards for School Mathematics also published in 1989; and the Standards Project for English/Literature Arts sponsored by the Center for the Study of Reading at the University of Illinois, the International Reading Association, and the National Council of Teachers of English. Other sources reporting academic outcomes were reviewed as well, including Kulm and Malcom (1991); Steffy (1993); and White (1994). In addition, planning documents and reports submitted by the NCRVE Urban Schools Network sites were reviewed for specific outcomes statements (e.g., NCRVE, 1993).

Pilot Testing the Instruments and Data Collection Procedures

After conducting the concept mapping procedure, the experts also provided suggestions for improving the clarity and content of the statements. These suggestions were made verbally and/or in writing. The experts were also asked to help reduce the number of statements to a smaller number because of restrictions of the computer program for no more than 98 statements total. However, although the experts offered valuable comments regarding the content of statements, most did not eliminate statements. Consequently, the number of statements was reduced from 118 to 98 by randomly eliminating 20 statements. A list of the final 98 items contained in the "Tech Prep Student Outcomes Rating Form" is provided in Table 4. (See Appendix B for a list of the 98 outcomes statements categorized according to their predominant location in the literature.)

Data Collection, Analysis, and Interpretation

To assist with interpretation of the data, preliminary results of the study were presented to a small group of NCRVE Urban Schools Network participants at the March, 1995 meeting of that group in Washington, DC. At that meeting, various concept maps were presented to the participants who were asked to assist in labeling the maps and suggesting alternative interpretations of their meanings. This interpretation session was helpful in determining how the stakeholder participants were likely to interpret the final results. It was also helpful in determining how to represent the concept maps visually and verbally in this concept paper.

As completed concept mapping materials were returned, data from the rating and sort forms were entered into the concept mapping computer program called "The Concept System" (Trochim, 1989a). This program was used to aggregate the sort and rate data provided by the stakeholder participants. It "uses a combination of multidimensional scaling (MDS) and cluster analysis techniques to represent conceptual domains underlying the data" (Caracelli & Riggin, 1994, p. 142). Information from the Background Form was compiled into a spreadsheet program. One map was generated to represent the collective view of all stakeholder participants. Additional maps were generated to represent the perspectives of the three stakeholder groups of educators, students, and employers. Further information about the computation and interpretation of these maps is presented in the next section.

FINDINGS AND DISCUSSION

The Perspectives of All Participants Toward
Tech Prep Student Outcomes

Based on this point map, the statements were grouped into clusters that show the domains associated with Tech Prep student outcomes. Ward's hierarchical cluster analysis (Everitt, 1980; Ward, 1963) was used by the "Concept System" program on the X-Y coordinate data obtained from the MDS (Trochim, 1989b). The MDS partitions the points on the map into cluster solutions where each solution shows the average priority ratings of each statement and cluster. The Concept System allows a researcher to impose solutions having from four to twenty clusters. Within each cluster, statements with average ratings approaching 5.0 are a high priority; those having average ratings nearer 1.0 are a low priority. The map also shows the relationships of the statements (shown as points on a cluster map) and clusters to each other, called bridging values. Statements with bridging values approaching zero are highly associated with surrounding statements, meaning these statements were sorted by many of the respondents with other statements in close proximity. In contrast, statements having bridging values approaching 1.0 show little association with surrounding statements. Average bridging values are also computed for clusters. Clusters with lower bridging values indicate more concise concepts (or constructs), while clusters with higher bridging values are less clear and interpretable. These clusters are called "bridging clusters" because they act as a link between other clusters in the map.

Based on a qualitative interpretation of the maps, a decision was made to select a nine-cluster solution as the best way to portray the domains associated with Tech Prep student outcomes.[7] Figure 2 presents the nine-cluster solution with the label developed for each cluster, along with the average cluster rating and average bridging value. (For example, for "communications" the average cluster rating is 3.61 and the average bridging value is .59.) Table 5 lists the statements within each cluster, their ratings, the average cluster ratings, and the average bridging values.

The nine-cluster concept map for all participants presents a great deal of information about how the entire group conceptualized Tech Prep student outcomes. First, all of the items and clusters received relatively high ratings on a 1 to 5 scale where 1 meant "very low" to 5 meant "very high" priority. All of the clusters representing the 98 Tech Prep student outcomes statements were given an average rating well above 3.00, indicating all of the clusters had at least a moderate level of priority for the respondents. Looking at the nine clusters, one was rated above 4.0 and two were near that level, indicating these clusters were perceived to be of highest priority to the respondents. The most highly rated cluster was "Personal Attributes, Attitudes, and Employability Skills" with a cluster average of 4.13. This cluster includes statements about honesty, integrity, dependability, and punctuality. The composition of this cluster bares a striking resemblance to the personal qualities described as foundational competencies by SCANS (U.S. Department of Labor, 1991). Another high priority cluster, having a cluster average of 3.96, was labeled "School-to-Work Transition" because of the parallel of the statements in the cluster to the key concepts portrayed in the federal STWO legislation. Finally, the third high priority cluster, with an average of 3.92, was labeled "Technology and Quality Management" because of the mix of statements contained therein having to do with such concepts.

The cluster of outcomes receiving the lowest average rating was "Democratic and Participatory Strategies," showing an average rating of 3.29. This cluster contained statements having to do with democracy, social awareness, diversity, and group processes. The next lowest rated cluster was "Math and Science" with a cluster average rating of 3.46. Within this cluster, outcomes statements having a more basic or applied focus received higher ratings than statements of a more abstract and advanced nature. For example, to apply basic algebra and geometry was given an average rating of 4.07 compared to the statement specifying advanced algebra, analytic geometry, and/or calculus which received a lower average rating of 3.15.

Once the importance of each cluster is understood, it is important to examine the relative location of the clusters, one to another, on the map. First, the two clusters located in the northwest and northern portion of the map represent the academic domains of "Communications" and "Math and Science." On the opposite side of the map, in the eastern and southeast portions, are "School-to-Work Transition" and "Personal Attributes, Attitudes, and Employability Skills." The fact that what appears to be vocationally and academically oriented outcomes show up on opposite sides of the map is an important finding, especially where the bridging values are low as in the case of "Math and Science" and "Personal Attributes, Attitudes, and Employability Skills" showing respondents rarely put such statements together and therefore did not see them as closely associated. Furthermore, the participants gave the "School-to-Work Transition" and "Personal Attributes, Attitudes, and Employability Skills" clusters higher average ratings than the "Communications" and "Math and Science" clusters. Although, as was suggested previously, it is important to remember that all of the clusters received a moderate to high priority rating.

The three clusters in the middle of the map labeled "Information Use and Decision-Making," "Technology and Quality Management," and "Work and Interpersonal Relationships" are important because they are located between the clusters of "Communications" and "Math and Science" and the clusters of "School-to-Work Transition" and "Personal Attributes, Attitudes, and Employability Skills." These clusters received average ratings between 3.75 and 3.92, showing they are a relatively high priority to the respondents. Within each of these clusters is a mix of items drawn from the literature used to create the instrumentation for this study. For example, the "Information Use and Decision-Making" cluster contains an item drawn from the science literature (i.e., "use computers and other electronic technology to gather, organize, manipulate, and present information"); an item from the communications literature (i.e., "construct meaning through reading for information, literary experience, and to perform a task"); and an item from the vocational/occupational literature (i.e., "select, use, and maintain appropriate tools, information, materials, and equipment"). Similarly, within the cluster labeled "Technology and Quality Management" are items found in the science, vocational, and management literature. Possibly, the mix of outcomes statements within these clusters provides a nucleus of concepts useful to the integration of vocational and academic education called for by Tech Prep.

With regard to this notion of vocational and academic integration, we are reminded that three clusters located around the parameter of the map had fairly high bridging values, meaning the statements within the clusters were not consistently sorted with other statements in the same clusters. These three clusters were "Communications" with an average bridging value of .59, "Democratic and Participatory Strategies" with an average bridging value of .58, and "School-to-Work Transition" with a slightly lower average bridging value of .46. The significance of this finding is that these clusters represent concepts that bridge other clusters, suggesting "Communications," "Democratic and Participatory Strategies," and "School-to-Work Transition" may be constructs that connect clusters, suggesting a different way of thinking about vocational and academic integration.

Figure 3 presents the average cluster ratings for each subgroup for the nine-cluster concept map created by all participants.[8] Several conclusions can be drawn from this map that confirm prior observations, but also provide new insights about Tech Prep student outcomes. First, with respect to several clusters there is virtually no difference in how the subgroups rated the clusters. This conclusion applies to such clusters as "Educational Attainment," "Work and Interpersonal Relations," and "Technology and Quality Management," showing there is substantial agreement among the subgroups regarding the level of priority that should be placed on these outcomes. However, in a few cases, students gave higher average ratings to clusters than other subgroups (e.g., see "School-to-Work Transition" and "Personal Attributes, Attitudes, and Employability Skills"). Two clusters are the exception to this conclusion: "Math and Science" and "Democratic and Participatory Strategies." With respect to the cluster labeled "Math and Science," students gave a lower rating than either educators or employers. With respect to "Democratic and Participatory Strategies," the average rating supplied by students was lower than educators, but higher than employers, although all subgroups gave this cluster a much lower average rating than any of the remaining clusters.

The Perspective of Each Stakeholder Group Toward
Tech Prep Student Outcomes

Educator Perspectives

There are five clusters with substantial similarities between the nine-cluster concept map created by the group of all participants (Figure 2) and the educators subgroup (Figure 4). The mean ratings attributed to these five clusters are similar. In addition, most of the items appearing in the five clusters received similar ratings.

Four clusters in the educators concept map labeled "Education and Career Attainment," "Analytic and Scientific," "Work Environments," and "Democratic Process and Career Awareness" do not appear under the exact same label in the map of all participants. The composition of outcomes statements in these four clusters have a qualitatively different focus (albeit sometimes only slight in some cases) from related clusters in the map of all participants. For example, educators sorted outcomes statements into a cluster labeled "Education and Career Attainment" containing eight outcomes statements such as to "complete secondary school"; "make a smooth transition from secondary to postsecondary"; "apply knowledge, skills, and learning strategies to career and life choices"; and "participate in work-based learning." Although the "Educational Attainment" cluster for all participants (containing seven statements total) had many of the same statements, no items linked to careers were present there. Overall, educators gave the "Education and Career Attainment" cluster a higher mean rating than the group of all participants gave the "Educational Attainment" cluster, 4.02 and 3.68, respectively. Educators gave higher mean ratings to outcomes statements within the cluster such as to "complete secondary school," "make a smooth transition from secondary to postsecondary," and "enter postsecondary programs without remediation." In contrast, educators rated the outcome "succeed in the transition from secondary or postsecondary education to a 4-year college" much lower than the group of all participants, 3.17 for educators compared to 3.48 for all. Both groups gave the outcomes statements "make academic progress on grade level" a rating over 4.0, meaning it is a high priority to all participants and to the subgroup of educators.

The cluster labeled "Analytic and Scientific" in the educators' map is similar to the cluster labeled "Math and Science" in the map of all participants. However, the educators' map adds other outcomes found in the math and science literature: "use computers and other electronic technology to organize, manipulate, and present information"; "organize information through the development of classification rules and systems"; and "communicate ideas by quantifying with whole, rational, real and/or complex numbers," giving this cluster a broader scope and more analytical character than the "Math and Science" cluster shown in the map for all participants. Both groups gave their respective clusters a lower mean rating than almost all other clusters, but still indicated the outcomes to be of a moderate to high priority. The educators' group gave the cluster a mean rating of 3.67 and the group of all participants gave it a slightly lower rating of 3.46.

The clusters labeled "Work Environments" and "Democratic Process and Career Awareness" were a unique blend of outcomes statements. These two clusters received the lowest mean ratings of the educators' subgroup, indicating both clusters were a moderate priority to educators. The cluster labeled "Work Environments" contains 12 outcomes statements including to "adapt to emerging technology and adjust to changing work environments" and "expand own knowledge by making connections with new and unfamiliar knowledge, skills, and experiences." Outcomes statements taken from the humanities, science, and fine arts literature also appear in this cluster. The other cluster labeled "Democratic Process and Career Awareness" contains seven outcomes statements, including several that appear in the cluster labeled "Democratic and Participatory Strategies" in the map of all participants. Outcomes statements appearing in both of these clusters include "prepare for direct participation in the democratic process" and "recognize and apply the democratic principles of justice, equality, responsibility, choice, and freedom." Two outcomes statements added to the "Democratic Process and Career Awareness" cluster giving it more of a career orientation are for students to "have awareness of and interest in technical careers" and to "know the history of a particular occupation."

Finally, similar to the concept map for all participants, three clusters on the educators' map received relatively high bridging values, although they were not the same three clusters. With respect to educators, the three clusters with fairly high bridging values are "Information Use and Decision-Making" with a bridging value of .52, "Work Environments" with a bridging value of .66, and "Democratic Process and Career Awareness" with a bridging value of .80. Recall that these high bridging values mean that the educators did not tend to sort the items consistently into the same categories, but, rather, grouped them more randomly. These clusters are not as distinct in the minds of educators as other clusters such as "Personal Attributes, Attitudes, and Employability Skills" and "Education and Career Attainment."

Employer Perspectives

In comparing the concept map created for employers to the map for all participants, only one cluster was labeled differently and it was "Education and Career Attainment"--the same label used in the educators' map. This label was used because, like educators, employers created a cluster that combined outcomes linked to both education and career attainment. Outcomes statements appearing in the "Education and Career Attainment" cluster in the employers' map are to "complete secondary school"; "participate in work-based learning"; "apply knowledge, skills, and learning strategies to career and life choices"; and "achieve certification of mastery in an occupation." For employers, this cluster contained several more outcomes statements than for either the group of all participants or for the educators' subgroup. In addition, the cluster received a mean rating of 3.61, showing it to be of lower relative priority to employers than other clusters such as "Personal Attributes, Attitudes, and Employability Skills," "Work and Interpersonal Relationships," and "School-to-Work Transition." (However, this mean rating is similar to the rating given by all participants and the educators' subgroup.)

A final result shows that several of the clusters in the employers' map have fairly high bridging values. In fact, only two of the clusters have extremely low bridging values and they are "Personal Attributes, Attitudes, and Employability Skills" and "Math and Science," meaning the outcomes statements in these clusters were sorted together consistently by persons in the employers' subgroup. Four clusters have quite high bridging values. They are "School-to-Work Transition," "Technology and Quality Management," "Communications," and "Democratic and Participatory Strategies," suggesting that the employers' subgroup did not sort statements within these clusters together consistently. Finally, the cluster receiving the lowest priority rating across all the concept maps was the cluster labeled "Democratic and Participatory Strategies." Employers gave this cluster a mean priority rating of 2.99, indicating student outcomes related to it are a moderate priority.

Student Perspectives

The five unique clusters created by students are "Work, Technology, and Information Use," "Career/Work Management and Initiative," "Technical Communications," "Communications and Democratic Process," and "Math, Science, and Communications." The first two of these clusters was given an average priority rating that was substantially higher than the ratings attributed to the other three clusters. "Work, Technology, and Information Use" received a mean priority rating of 3.98 and "Career/Work Management and Initiative" got a mean priority rating of 3.97, both very near the high priority level of 4.0. The "Work, Technology, and Information Use" cluster contained 11 outcomes statements that, taken together, seem to suggest students in Tech Prep programs are well-aware of the need to be competent at using technology and information to operate effectively in contemporary workplaces. For students, this cluster falls between two career-oriented clusters--"School-to-Work Transition" and "Technical Communications"--and this finding might provide valuable insight into ways to link and integrate these and other clusters of outcomes.

The cluster labeled "Career/Work Management and Initiative" had a similar focus on work, but it encompassed 13 items related to managing one's and others' work and careers. Similar to the previous discussion about students acute sense of awareness of careers and work, the overall character of this cluster suggests students in Tech Prep programs recognize the importance of developing management skills and personal initiative related to immediate and future work. They give high ratings to outcomes demonstrating the ability to show leadership and strong supervisory and management competencies.

The three other clusters unique to students are "Technical Communications," "Communications and Democratic Process," and "Math, Science, and Communications." All three of these clusters received mean priority ratings between 3.52 and 3.62, indicating they were not viewed to be as important as other clusters, but still were rated of moderate priority to students. Immediately apparent from the titles of these three clusters is the fact that students sorted outcomes statements related to communications into all three clusters. Not surprisingly, then, all three clusters have relatively high bridging values, ranging from .62 to .90, providing additional confirmation that the items in these clusters were sorted in different ways by different students.

In terms of the organization of the clusters, the cluster labeled "Technical Communications" appears in the northwest portion of the map with clusters labeled "Math, Science, and Communications," and "Communications and Democratic Process." Seven outcomes statements appear in the cluster labeled "Technical Communications," including "communicate ideas and information through speaking," "demonstrate oral and verbal proficiency in technical communications," and "understand the relationships between theory and practice in a technical area." The cluster labeled "Communications and Democratic Process" contains six outcomes statements, including "create meaning from messages communicated through listening"; "recognize and apply the democratic principles of justice, equality, responsibility, choice, and freedom"; and "recognize differences and commonalties in the human experience through productions, performances, or interpretations." Within this cluster are outcomes taken from the English/communications, humanities, and fine arts literature, suggesting a potential area where various outcomes are interrelated from the perspective of students.

Finally, the cluster labeled "Math, Science, and Communications" contained 13 outcomes statements related to math, science, and English/communications. The cluster appears to be a mixture of outcomes that might traditionally be expected of high school students. Included among these outcomes are the following statements: "apply the English language correctly (spelling, grammar, structure)"; "use critical thinking skills in a variety of situations"; "apply basic algebra and geometry to solve technical and work-related problems"; and "use scientific methods to acquire information, plan investigations, use scientific tools, and communicate results." Several outcomes statements specifically related to advanced mathematics and science were present in this cluster and given relatively low priority ratings by students, ranging from 3.33 for "use scientific methods to acquire information, plan investigations, use scientific tools, and communicate results" to 2.56 for "use the metric system and convert between metrics and traditional systems." These results suggest students in Tech Prep programs may not appreciate these academic outcomes, particularly those related to math and science. It is clear that students undervalued these outcomes relative to the educators and employers who participated in this study.

IMPLICATIONS FOR POLICY AND PRACTICE

Formal program evaluation and outcomes assessment for Tech Prep has been limited, but when evaluations have been conducted they have tended to focus on compliance-oriented measures required by governmental units. Outcomes measures linked to enrollments, program completion, and job placement have been typical of the kinds of measures demanded by state and federal agencies. The national evaluation sponsored by the U.S. Department of Education concentrates much of its attention on having local Tech Prep coordinators estimate the number of students who reach specified points in the educational and employment system, such as high school completion, matriculation into two-year postsecondary education, two-year postsecondary completion, and job entry or matriculation into four-year postsecondary education. Such estimates may be useful in terms of understanding the potential scope and scale of the nation's emerging Tech Prep system, but they are not as helpful to understanding the way programs should operate and benefit students on more personal and consequential levels.

When local coordinators have been asked to specify outcomes they believe to be appropriate for students in vocational-technical programs, typically they identify educational, economic, and psychosocial outcomes (McCaslin, 1990). In 1992, Hammons surveyed local Tech Prep coordinators (educators), and found they supported a wide range of performance indicators for Tech Prep programs, including outcomes in all three of the categories described by McCaslin. Later, in 1994, many of Hammons' findings were supported when a national sample of local coordinators indicated a wide range of academic, vocational, and employment-related outcomes were a high priority for students (Bragg et al., 1994). Now, results of this concept mapping study show the three stakeholder groups of educators, students, and employers also give high priority to a wide array of student outcomes. All three stakeholder groups rate nearly all of 98 student outcomes statements at a moderate or high priority level. (For a summary comparison of how the three stakeholder groups rated the outcomes statements by clusters, see Appendix B.) This finding indicates it would be a mistake to limit assessments of student outcomes to only a few outcomes measures. Rather, multiple measures addressing a wide range of outcomes are necessary to determine how students benefit from educational and employment-related experiences. Unfortunately, assessment measures and methodologies are not available to conduct wide scale assessments of many of the student outcomes identified by the stakeholders in this study, heightening the importance of the need to create valid, reliable, and meaningful outcomes assessments for Tech Prep programs.

Are certain Tech Prep student outcomes grouped together in a logical, consistent pattern? Do the stakeholder groups perceive of the groupings (clusters) of Tech Prep student outcomes in similar and/or different ways? Are particular clusters of student outcomes more important than others to subgroups and to the group as a whole? By looking at the concept maps organized into nine-cluster solutions by the three stakeholder groups and the entire group of participants, it is possible to answer these important questions. Indeed, results show there are similarities in how the three groups conceptualize and prioritize Tech Prep student outcomes. All three stakeholder groups sorted many of the same student outcomes into three clusters labeled "Personal Attributes, Attitudes, and Employability Skills"; "School-to-Work Transition"; and "Work and Interpersonal Relationships." All of these clusters were given a rating near or at a high priority level of 4.0 (out of 5.0) by the subgroups, showing a high degree of consensus in how the stakeholders conceptualized these sets of student outcomes for Tech Prep (see Table 9). These results also suggest that student outcomes linking school-based education to effective attitudes and behaviors in the workplace are a high priority to all three stakeholder groups. Participants organized these outcomes into three distinct groupings: one dealing with personal attitudes and behaviors at work; a second focusing on interpersonal attitudes and behaviors at work; and a third concentrating on more transferable attitudes and behaviors between the school and work environments.

Three additional clusters were created by two of the subgroups. These clusters were "Information Use and Decision-Making," "Education and Career Attainment" and "Communications." Educators and employers were in agreement with respect to how they grouped and prioritized the first two of these three clusters of student outcomes; however, a notable exception was in "Education and Career Attainment" where the average cluster rating given by educators exceeded that of employers by a wide margin, 4.02 compared to 3.61. Although students did not link concepts associated with careers to educational attainment in the same way as educators and employers, the priority placed on the "educational attainment" cluster was also quite high (3.97). These results show outcomes associated with advancing within the educational system are a high priority to educators and students within the system, but not as much to employers outside of it. Consistently, employers give student outcomes associated with school-to-work transition and employment a higher priority than outcomes more closely associated with the educational system itself. This finding raises the question of what level of priority to place on educational attainment outcomes that address whether students are making progress on grade level, graduating from high school, matriculating to the two-year postsecondary level, and so forth. The question is particularly pertinent with regard to a program such as Tech Prep where a school-to-work focus is important to all stakeholders. Given that position, how far should Tech Prep programs go to accommodate the perspective of employers who know the workplace best? How much weight should be given to their preference for vocationally oriented outcomes over educational outcomes?

Yet, the issue is not simply one of educators and students giving greater priority than employers to educational outcomes. Indeed, the situation is much more complex. In fact, all the subgroups rated many outcomes statements associated with traditional academic subjects such as mathematics, science, English, humanities, social studies, and the fine arts lower than outcomes aligned with school-to-work transition and employment. Although all three groups organized outcomes statements into distinct clusters aligned with these academic concepts, most rated these clusters lower than the ones having a work or career orientation. Furthermore, within the clusters of "Math and Science" and "Analytic and Scientific," the stakeholders rated outcomes at the basic level more highly than those at the advanced level, showing a preference for students' mastery of more fundamental academic concepts over more advanced. Also, the academic clusters were segregated from vocational clusters on all the concept maps, with the academic concepts placed on the west side of the map and vocational concepts on the east. This result gives the impression that sets of outcomes associated with vocational and academic education may be both distinct and independent from one another. However, some clusters do not appear to fit this conclusion. In all of the concept maps, stakeholders created one or more clusters containing outcomes having to do with technology, information use, decision-making, work, and management. The outcomes within these clusters were drawn from across the disciplines such as humanities, social studies, science, and vocational-technical education. Typifying this kind of cluster is the one created by students labeled "Work, Technology, and Information Use" and one developed by employers labeled "technology and quality management." Within each of these clusters is the nucleus of outcomes taken from a wide range of vocational and academic subject matter, potentially providing ideas for integrating Tech Prep instruction.

An additional observation should be made about the nature of the three subgroups' conceptualizations of vocational and academic outcomes. Consistently, vocationally oriented outcomes received high or nearly high priority ratings, while academically oriented outcomes received lower (albeit not low but moderate) ratings. Within the clusters of academically oriented outcomes, statements linked to the academic areas of social studies and humanities received the lowest ratings. All three stakeholder groups created clusters with outcomes statements linked to the "Democratic Process and Career Awareness" or "Democratic and Participatory Strategies" and all three gave these clusters low mean ratings relative to the other clusters. In fact, the cluster labeled "Democratic and Participatory Strategies" created by employers rated the lowest of all clusters with an average rating of 2.99. Is this pattern a random occurrence or is there something about Tech Prep that suggests democratic outcomes should receive a lower priority than other outcomes? Public policy specifies that Tech Prep curriculum should be comprised of mathematics, science, English/ communications, and vocational-technical education. Rarely is the area of social studies or humanities mentioned as central. In addition, many local consortia and state agencies profess a primary purpose of Tech Prep is to "eliminate the general track." These constituents are attempting to improve education for the neglected majority of students who have been engaged in the general track, but in so doing may shift priorities away from some of the more traditional social and democratic functions of public education. Is this shift actually occurring? We could not identify data to suggest that such a curricular shift is occurring; however, it is an important issue to monitor. What are the consequences of shifting priorities away from traditional academic subjects to education more highly focused on school-to-work transition, technologies, and vocations? Without more attention paid to formal evaluation, this question will remain elusive.

In summary, this study attempted to better understand Tech Prep student outcomes from the perspectives of educators, students, and employers actively engaged in implementing Tech Prep. Knowing how these groups conceptualize student outcomes has important implications for understanding the fundamental objectives of Tech Prep, for planning and implementing programs, and for assessing outcomes in the future. Also, by uncovering various conceptualizations of Tech Prep, it may be possible to identify conflicting perspectives held by disparate stakeholder groups. Having information from still more stakeholder groups from other localities such as rural and suburban areas would help to illuminate the ways other constituents think about Tech Prep. In addition, obtaining information from policymakers, parents, counselors, and still other groups could help in the development of outcomes assessments. As Tech Prep implementation continues, more attention must be devoted to student outcomes, and this study takes an important next step in that direction.

REFERENCES

Bragg, D. D. (Ed.). (1992, December). Alternative approaches to outcomes assessment for postsecondary vocational education (MDS-239). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

Bragg, D. D., Layton, J., & Hammons, F. (1994). Tech Prep implementation in the United States: Promising trends and lingering challenges (MDS-714). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

Bragg, D., & Layton, J. (1995, May). The role of the urban community college in educational reform. Education and Urban Society, 27(3), 294-312.

Brown, J. M., Pucel, D. J., Johnson, R., & Kuchinke, P. (1994). The Tech Prep consortia evaluation system 1993/94 cohort summary preliminary report. St. Paul: University of Minnesota, Department of Vocational and Technical Education.

Campbell, J. R. (1995, March). Delaware consortium on technical preparation programs: Evaluation model. Dover, DE: State Consortium on Technical Preparation Programs.

Caracelli, V. J., & Riggin, L. J. C. (1994, June). Mixed-method evaluation: Developing quality criteria through concept mapping. Evaluation Practice, 15(2), 139-152.

Connell, T., & Mason, S. (1995, April). School to work transition: Issues and strategies for evaluation and program improvement. Paper presentation at the annual meeting of the American Educational Research Association in San Francisco, CA.

Decision Information Resources, Inc. (n.d.). Evaluation of Tech Prep system development and implementation in Texas public schools and institutions of higher education. Houston, TX: Author.

Dornsife, C. (1992). Beyond articulation: The development of Tech Prep programs (MDS-311). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

Everitt, B. (1980). Cluster analysis (2nd ed.). New York: Halsted Press.

Grayson, T. E. (1992). Concept mapping. In D. D. Bragg (Ed.), Alternative approaches to outcomes assessment for postsecondary vocational education (MDS-239) (pp. 65-93). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

Hammons, F. T. (1992, April). The first step in Tech-Prep program evaluation: The identification of program performance indicators. Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University.

Hammons, F., & Pittman, L. (1995, March). New Hampshire statewide Tech Prep evaluation. Miami Beach: Hammons, Pittman & Associates, Inc.

Harman, H., & Stowers, P. (1995, February). Status of the Tech Prep associate degree initiative in West Virginia. Charleston, WV: Appalachia Educational Laboratory.

Hershey, A., & Silverberg, M. (1994, September). The emergence of Tech-Prep at the state and local levels. Washington, DC: U.S. Department of Education, Office of the Under Secretary Planning and Evaluation Service.

Hershey, A., Silverberg, M., & Owens, T. (1994, October). The diverse forms of Tech Prep: Implementation approaches in ten local consortia. Washington, DC: U.S. Department of Education, Office of the Under Secretary Planning and Evaluation Service.

Hoachlander, E. G., & Rahn, M. L. (1992, March). Performance measures and standards for vocational education: 1991 survey results (MDS-388). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

Hoachlander, E. G., Levesque, K., & Rahn, M. L. (1992, July). Accountability for vocational education: A practitioner's guide (MDS-407). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

Hull, D., & Parnell, D. (Eds.). (1991). Tech Prep associate degree: A win/win experience. Waco, TX: Center for Occupational Research and Development.

Jurgens, C. (1995). Nebraska Tech Prep: Executive summary fiscal year 1995. Lincoln: Nebraska Department of Education.

Keller, R. (1995, January). Evaluation of Colorado Tech Prep educational initiative: FY 1994 report. Aurora, CO: Mid-Continent Regional Educational Laboratory, Inc.

Key, C. (1994). Synthesis of literature related to Tech-Prep outcomes. In G. Baker III (Ed.), A handbook on the community college in America: Its history, mission and management (pp. 137-150). Westport, CT: Greenwood Press.

Kulm, G., & Malcom, S. M. (Eds.). (1991). Science assessment in the service of reform. Washington, DC: American Association for the Advancement of Science.

Layton, J., & Bragg, D. (1992). Initiation of Tech Prep by the fifty states. In D. D. Bragg, (Ed.), Implementing Tech Prep: A guide to planning a quality initiative (MDS-241) (pp. 4-1 to 4-18). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

MGT of America, Inc. (1995, November). An evaluation of Tech Prep in Ohio: Year one final report. Tallahassee, FL: Author.

McCaslin, N. L. (1990). A framework for evaluating local vocational education programs. Columbus: ERIC Clearinghouse on Adult, Career, and Vocational Education, Ohio State University.

McCaslin, N., & Headley, S. (1993). A national study of approved state systems of performance measures and standards for vocational education. Columbus: Ohio State University, Comprehensive Vocational Education Program.

McKinney, F., Fields, E., Kurth, P., & Kelly, F. (1988). Factors influencing the success of secondary/postsecondary vocational-technical education articulation programs. Columbus: National Center for Research in Vocational Education, Ohio State University.

National Center for Research in Vocational Education. (1993, November). Establishing integrated Tech Prep programs in urban schools: Plans developed at the NCRVE 1993 national institute (MDS-770). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

National Commission on Excellence in Education. (1983). A nation at risk. Washington, DC: U.S. Department of Education.

National Commission on Secondary Vocational Education. (1984). The unfinished agenda. Columbus: National Center for Research in Vocational Education, Ohio State University.

North Carolina Department of Public Instruction & North Carolina Department of Community Colleges. (1994, November). 1993-94 final report: Planning and implementation grants. Raleigh: Author.

O'Neil, S. L. (1976). Worker perceptions of skills necessary for survival in the world of work. Unpublished doctoral dissertation, University of Illinois at Champaign-Urbana.

Oregon Department of Education. (n.d.). The certificate of advanced mastery. Work in progress planning document. Salem: Author.

Oregon Department of Public Instruction. (1993, January). Excerpts from Oregon certificate of initial mastery--Task force report. Salem: Author.

Owens, T. R. (1995, April). Washington year two: Tech Prep planning and implementation survey summary. Portland, OR: Educational and Work Program, Northwest Regional Educational Laboratory.

Owens, T., Lindner, F., & Wang, C. (1994, December). Tech Prep in Washington state: Case studies report. Portland, OR: Northwest Regional Educational Laboratory.

Owens, T., Lindner, F., & Wang, C. (1995, June). New learnings from Tech Prep in Washington state: A second-year case studies report. Portland, OR: Northwest Regional Educational Laboratory.

Parnell, D. (1985). The neglected majority. Washington, DC: Community College Press.

Pearce, K., Pease, V., Copa, G., & Beck, R. (1991, July). Educational outcomes: Past, present, and future. Working paper of the Design Group for the project "New Designs for the Comprehensive High School." St. Paul: National Center for Research in Vocational Education, University of Minnesota.

Peasley, D., & McCaslin, N. (1995, April). An exploratory examination of the framework for evaluating vocational education programs. Paper presented at the American Educational Research Association meeting, San Francisco, CA.

Rhode Island Tech Prep Associate Degree Program. (No author or date given.)

Roegge, C., & Evans, J. (1995, July). Illinois Tech Prep evaluation study: Final report draft FY1995. Urbana: Department of Vocational and Technical Education, University of Illinois at Urbana-Champaign.

Roegge, C. A., Leach, J. A., & Brown, D. C. (1992). Stakeholder's perceptions of major themes and priority areas of Tech Prep in Illinois. Journal of Vocational Education Research, 18(3), 77-96.

Rubin, M. (1994). Evaluation of California's Tech Prep education program: Year two evaluation report (Draft). San Diego: Evaluation and Training Institute and San Diego Community College District.

Ruhland, S., Custer, R., & Stewart, B. (1994). Status of Tech Prep in Missouri, 1993-94. Jefferson City, MO: Division of Vocational and Adult Education, Department of Elementary and Secondary Education.

Secretary's Commission on Achieving Necessary Skills (SCANS). (1991, June). What work requires of schools. Washington, DC: U.S. Department of Labor.

Seidman, P., & Ramsey, K. (Eds.). (1995, May). Vocational education and school reform: New connections from school to work. Education and Urban Society, 27(3), 235-243.

Stecher, B., Hanser, L., & Hallmark, B. (1995, January). Improving Perkins II performance measures and standards: Lessons learned from early implementers in four states (MDS-732). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

Steffy, B. E. (1993). The Kentucky education reform: Lessons for America. Lancaster, PA: Technomic Publishing Co.

Tennessee Board of Regents & Tennessee Department of Education. (1994, December). Tech Prep annual performance report, July 1, 1993 to June 30, 1994. Nashville: Author.

Trochim, M. K. (1989a). The concept system. Ithaca, NY: Concept System.

Trochim, M. K. (Ed.). (1989b). An introduction to concept mapping for evaluation and planning. Evaluation and Program Planning, 12(1), 1-16.

Trochim, M. K., & Linton, R. (1986). Conceptualization for planning and evaluation. Evaluation and Program Planning, 9(4), 289-308.

U.S. Congress. (1990). Carl D. Perkins Vocational and Applied Technology Education Act Amendments of 1990. Public Law 101-392. Washington, DC: U.S. Government Printing Office.

U.S. Department of Labor. (1991). What work requires of schools: A SCANS report for America 2000. Washington, DC: Author.

Ward, J. H. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58, 236-244.

White, S. (1994). Overview of NAEP assessment frameworks. Washington, DC: National Center for Educational Statistics.

NOTES

[2] On May 4, 1994, the U.S. Congress passed the School-to-Work Opportunities (STWO) Act which has a primary goal of encouraging states to plan and implement coordinated school-to-work systems using a variety of models including Tech Prep to assist youth to obtain employment after completing secondary or postsecondary education.

[3] The study sample was comprised of educators, students, and employers actively engaged in Tech Prep implementation as a part of the National Center for Research in Vocational Education's Urban Schools Network.

[4] A national-level evaluation is mandated by the Carl D. Perkins Vocational and Applied Technology Act Amendment of 1990 (Perkins II) and this evaluation is described later in this section.

[5] Perkins II has a primary objective of developing improved accountability systems that require each state to measure student learning gains in basic and more advanced academic skills and student performance in competency attainment. States must also implement measures in one or more of the following areas: job or work skill attainment or enhancement, retention or completion, or job placement.

[6] A targeted follow-up of about fifty Tech Prep consortia indicated by the NCRVE survey to be the most advanced at Tech Prep evaluation in the nation produced disappointing results. Very few formal evaluation plans, instruments, or reports were produced by these sites.

[7] Since it was a primary goal of the study to compare the maps generated by the three stakeholder groups, it was important to maintain the same number of cluster solutions for all the maps. The nine-cluster solution was selected because of its meaningfulness in representing all of the participants' perspectives and each of the subgroups' perspectives in concept maps.

[8] The average cluster ratings for subgroups presented in Figure 3 differ from those presented later in this report because the calculation of average cluster ratings are dependent upon the composition of outcomes statements in the clusters obtained from each particular cluster map. As each new map is calculated, producing more or less difference in the clusters, the average cluster ratings change to reflect the different composition of the clusters. This is why it is important to interpret the results in several different ways to obtain a more complete understanding of results.

APPENDIX A
LIST OF STUDENT OUTCOMES BY CLASSIFICATION IN THE LITERATURE

Communications (7)

• communicate ideas and information through speaking
• construct meaning through reading for information, literary experience, and to perform a task
• construct meaning from messages communicated through listening
• communicate ideas and information through writing
• apply the English language correctly
• understand nonverbal communication
• demonstrate oral and verbal proficiency in technical communications

Fine Arts (4)

• appreciate own and others' artistic products and performances
• communicate ideas and emotions through the fine arts (e.g., art, music, dance)
• recognize differences and commonalities in the human experience through their productions, performances, or interpretations
• understand and communicate in a second language

Group Skills (9)

• evaluate others' performance and provide feedback
• demonstrate consistent, responsive, and caring behavior
• exercise leadership in a variety of situations
• get along with a variety of people
• participate as member of a team
• serve clients/customers
• teach others new skills
• resolve conflict based on divergent interests and perspectives
• plan and work together in meetings

Integrated Knowledge Skills (2)

• expand own understanding of existing knowledge by making connections with new and unfamiliar knowledge, skills, and experiences
• recognize the need for lifelong learning to enhance skills and learn new skills

Math Skills (7)

• apply basic algebra and geometry to solve technical and work-related problems
• communicate ideas by quantifying with whole, rational, real, and/or complex numbers
• demonstrate own ability to calculate through ratios, proportions, and percentages
• use division, multiplication, addition, and subtraction with real numbers, decimals, fractions, integers, roots, and powers
• use the metric system and convert between metrics and traditional systems
• apply advanced algebra, analytic geometry, and/or calculus to solve technical and work-related problems
• read and create charts, tables, and graphs

Personal Skills (8)

• demonstrate self-control and self-discipline
• demonstrate the ability to be adaptable and flexible
• demonstrate motivation to learn
• make ethical decisions
• know own abilities, strengths, and weaknesses
• maintain good physical, mental and emotional health
• build own self-esteem
• use initiative, imagination, and creativity

Educational Attainment (10)

• earn college credit in high school
• make academic progress on grade level
• demonstrate a positive attitude toward school
• attend school regularly
• complete postsecondary school
• enter postsecondary programs without remediation
• complete secondary school
• make a successful transition from education to employment
• make a smooth transition from secondary to postsecondary education
• succeed in the transition from secondary or postsecondary education to a 4-year college

Science (6)

• organize information through the development and use of classification rules and systems
• use appropriate and relevant scientific skills to solve specific problems in real-life situations
• use models and scales to explain or predict the organization, function, and behavior of objects, materials, and living things
• understand how technology affects quality of life
• use the scientific method to acquire information, plan appropriate investigations, use scientific tools, and communicate the results
• use computers and other electronic technology to gather, organize, manipulate, and express information and ideas

Social Studies (9)

• articulate personal values and beliefs as they relate to a particular occupation
• be critically aware of social issues involved in a field of interest
• know the history of a particular occupation
• observe, analyze, and interpret human behaviors to acquire a better understanding of self, others, and human relationships
• recognize and apply the democratic principles of justice, equality, responsibility, choice, and freedom
• recognize the geographic interaction between people and their surroundings and make responsible decisions for the environment
• recognize varying forms of government and address issues of importance to citizens in a democracy
• appreciate the diversity of values and cultural differences among people
• prepare for direct participation in the democratic process

Thinking and Problem Solving (11)

• design, maintain, and improve systems
• know how social, organizational, and technological systems work
• monitor and correct own performance
• use goal-relevant activities, rank them, and allocate time for them
• prepare and follow schedules, and manage time efficiently
• use decision-making processes to make informed choices among options
• use critical thinking skills in a variety of situations
• prepare and use budgets, make forecasts, keep records, and make adjustments to meet objectives
• use research tools to locate sources of information and ideas relevant to a specific need or problem
• apply group problem-solving strategies
• apply logical reasoning to develop solutions to complex problems

Vocational/Occupational (25)

• show appropriate personal appearance and attitude
• acquire, store, allocate, and use materials and space efficiently
• have awareness of and interest in technical careers
• be honest and demonstrate integrity
• adapt to emerging technology and adjust to changing work environments
• apply knowledge, skills, and learning strategies to career and life choices
• be dependable and punctual
• be loyal to employer
• achieve certification of mastery in an occupation
• achieve and maintain employability in high-wage jobs
• know how to give and take instructions
• apply appropriate safety and environmental measures
• develop and follow through on individual career plans and goals
• demonstrate experience in all aspects of an industry
• know employer expectations for job performance
• select, use, and maintain appropriate tools, information, materials, or equipment
• work under tension or pressure
• work without close supervision
• show good working relationships with superiors and coworkers in an occupational role
• demonstrate awareness of workforce and societal trends
• understand the relationships between theory and practice in a technical area
• recognize and apply quality standards
• understand the norms and values of the work culture
• understand the principles of competition, cooperation, and leadership in a work environment
• participate in work-based learning

APPENDIX B
A COMPARISON OF MEAN RATING AND BRIDGING VALUES
FOR OUTCOMES STATEMENTS BY CLUSTER AND STAKEHOLDER GROUP