Appendix A

---

CAREER-TECHNICAL ASSESSMENT PROGRAM

---

• DESCRIPTION AND PURPOSE
• RELATIONSHIP TO OTHER PROGRAMS
• IMPLEMENTATION AND ADMINISTRATION
• TECHNICAL QUALITY
• CONSEQUENCES AND USE OF ASSESSMENT RESULTS
• APPLICABILITY TO VOCATIONAL EDUCATION

The Career-Technical Assessment Program (C-TAP) is a standards-based assessment system designed to support instruction of important career skills and assess the preparedness of California students for entry-level jobs and postsecondary educational training. The program was developed by WestEd, under the direction of the California Department of Education (CDE) and the Sacramento County Office of Education (SCOE), for use in vocational programs in California's high schools and regional occupational centers/programs. C-TAP assessments are being implemented in five career areas: agriculture, business, health careers, home economics, and industrial and technology education. Within each area, C-TAP is targeted either to clusters of occupations or a core program that teaches basic information relevant to a wider grouping of occupations. There are currently C-TAP assessment materials available for two clusters or core programs for each career area:

1. Agriculture: agriculture core, animal science

2. Business: marketing, computer science and information systems

3. Health Careers: health careers core, advanced core course

4. Home Economics: child development and education, food services and hospitality

5. Industrial and Technology Education: tech core, construction

DESCRIPTION AND PURPOSE

Originally, in 1990, C-TAP was planned as a set of specific occupational tests for over twenty-nine occupations. The tests were to be made up primarily of multiple-choice questions and some performance items measuring specific skills for entry-level jobs. C-TAP's primary purpose was to be a standardized statewide student certification system; its secondary purpose was to be a program evaluation tool.

C-TAP's purpose and content have changed over time. Early on, the focus switched from specific occupations and job skills to clusters of related occupations and broader skills, and expanded to include cumulative assessment components. Both shifts were consistent with other state and national initiatives in vocational education and assessment. C-TAP is currently being used primarily as a teaching/
learning tool and as an assessment contributing to grades in vocational education programs. It is not yet widely used in any standardized fashion; rather, teachers tend to adapt the materials for use in their own classrooms. WestEd estimates that several hundred teachers are using C-TAP.

The C-TAP assessment system includes three components, each of which addresses academic skills, general workplace skills, and job-specific skills: (1) the portfolio, (2) the project, and (3) the written scenario. C-TAP is currently being expanded to include additional on-demand assessments (multiple choice and short written-response items) that will be administered by CDE. Each of the C-TAP assessment components is linked to California's career-technical model curriculum or challenge standards, as well as to related academic and career preparation standards. Scoring for the portfolios, projects, and written scenarios is done by teachers, using scoring rubrics designed by WestEd. Readers assign scores on specific dimensions of performance as well as an overall holistic score. There are four possible scoring levels: (1) Low Basic (incomplete and/or very unsatisfactory), (2) Basic (unsatisfactory), (3) Proficient (very good), and (4) Advanced (excellent).

The portfolio is a collection of student work that shows important knowledge, skills, and achievements. It serves as a vehicle for organizing and presenting students' work for assessment purposes, as well as for presentation to prospective employers or advanced training institutions. The portfolio contains five parts: an introduction (table of contents and letter of introduction), a career development package (an application for employment or college, a letter of recommendation, and a resume), work samples (at least four products in which the student illustrates mastery of specific skills as reflected in the model curriculum standards [MCS]), a writing sample related to the student's career area, and a supervised practical experience evaluation (encouraged but not required, since career programs vary in their requirements for practical experience). Figure A.1 describes some portfolio work samples.

Theproject is a major piece of hands-on work requiring students to plan, develop, and evaluate an important product or event related to their career interests. The project includes four parts: (1) a planoutlining

the process a student will go through to design and complete the project; (2) evidence of progress (a minimum of three pieces) showing the student's progress toward developing the final product; (3) a final product/event; and (4) an oral presentation in which the student describes his or her project, explains the standard-related knowledge and skills used, evaluates his or her own work, and explains what he or she learned. Figure A.2 describes some sample projects.

Originally, C-TAP did not include a project but instead required an on-demand performance task (chosen centrally for all schools) and a
separate oral presentation. When pilot tested, the on-demand task met with some opposition from teachers who felt it did not necessarily accommodate instructional and curricular differences among schools. As a result, the task and oral presentation were merged to create the project, which allows for some student choice and can be adapted to fit with the teacher's curriculum.

The written scenario is a forty-five-minute written response test that presents students with a "real-life" problem in their career area. Students must evaluate the problem and propose a means of addressing it. Their ability to demonstrate content knowledge is evaluated, as well as their problem-solving and communication skills. Scenarios were included in C-TAP to directly assess students' problem-solving skills. A scenario from a veterinary science class provides one example. Students read a description of an unhealthy cow's symptoms and living conditions. They then identify and write about the illnesses the cow may be suffering from, their causes, and possible treatment options.

C-TAP was recently expanded to include multiple-choice and short written-response questions. These on-demand components are designed to measure the breadth of students' career-technical knowledge. They will be administered and scored by CDE.

RELATIONSHIP TO OTHER PROGRAMS

In the future, C-TAP may continue its current form of usage while also fulfilling its originally intended role as part of a certification program. California has already incorporated it into certain reform initiatives. For example, California requires that eighty sites with tech-prep programs use C-TAP for assessing student progress. The state also is considering the use of certificates of initial and advanced mastery; if this initiative is adopted, C-TAP may become a requirement for the receipt of one or both certificates. Schools and programs that regularly use and score most C-TAP components can use it to develop their own certification system.

C-TAP is linked to several sets of state standards. The portfolio, project, and written scenario are tied to both career-technical model curriculum (content) standards (MCS) and career preparation (general workplace readiness) standards. There are separate MCS for each occupational cluster. The seven career preparation standards apply to all five career areas. The supervised practical experience evaluation in the portfolio specifically asks for students to be evaluated on these standards. The newly developed multiple-choice and short written-response items are linked to the draft interim content and performance standards, "Challenging Standards for Student Success," developed by CDE. There is substantial overlap between these standards and the career-technical MCS.

IMPLEMENTATION AND ADMINISTRATION

All of the teachers interviewed used the portfolio, which was generally deemed the strongest component of C-TAP. In turn, teachers saw the work samples as the most valuable element of the portfolio. Work samples require the student to document and explain what he or she has learned, which reinforces the material while also giving the student a record of his or her achievement for the future. This technique both bolsters students' self-esteem and provides evidence of knowledge and skills that can be shown to others.

Teachers believe that three types of skills are required for the portfolio: applied academic skills (especially writing), work readiness skills (demonstrated in the supervised practical experience evaluation and in how well the student communicates in the work samples), and specific technical skills. The requirements for the portfolio vary by program. For example, some teachers ask students to submit more work samples than the recommended four. The level of detail in work samples may vary greatly, e.g., from a half-page description of how to use a fax machine to several pages with photos describing how to vaccinate a lamb. The requirement for the writing sample also can take various forms, including a research paper written for a career-technical course, a paper related to a career-technical subject written in another course, and an expository paper describing a personal experience.

Teachers also differed in the weight they placed on C-TAP when evaluating students. All the teachers surveyed graded the subcomponents of the portfolio as well as the portfolio as a whole, placing primary emphasis on the work samples and writing sample. The contribution of the portfolio to students' final grades varied by teacher but reached as high as 80 percent. The majority also required the submission of a completed portfolio in order to pass the course or program, and one teacher tied a completed portfolio to receiving the career certificate plus college credit for the high school course. Where used, projects were graded. Scenarios were or were not graded, depending on the teacher.

The scenario and project components of C-TAP were used less consistently than the portfolio. Teachers gave several reasons for not using scenarios or projects, including the following: they created too much work when combined with portfolios; projects were already being assigned and evaluated, so C-TAP project materials added nothing new (or not enough that was new); and there were gaps between when particular material was taught and when the relevant scenarios arrived from WestEd. When teachers used projects, they often simply used the ideas provided in C-TAP to give more structure to an existing assignment. Use of the scenarios varied from once or twice during a course, to more often for practice, to once at the end of a unit.

TECHNICAL QUALITY

WestEd's attempts at determining the reliability of C-TAP have focused primarily on developing consistent scoring procedures. So far, this work has entailed selecting benchmarks and scoring student work, primarily for portfolios and scenarios. Benchmarks are examples of student work at the different performance levels (Low Basic, Basic, Proficient, and Advanced). They are used, along with training and calibration examples, to train teachers in how to score the products.

Portfolio Benchmarking and Scoring

As originally designed, the portfolio scoring process entailed two steps: (1) evaluating individual portfolio entries as students completed them and (2) reviewing and rating completed portfolios. Teachers were encouraged to give students feedback on the quality of their work throughout the portfolio development process to ensure they produced high-quality entries. Once all required entries were present, the portfolios were evaluated and assigned one holistic rating of Basic, Proficient, or Advanced.

The original C-TAP rating guide comprised four dimensions identified as critical for demonstrating competence in career-technical areas. These dimensions included content (breadth, depth, and application of knowledge and skills related to the career-technical MCS), career preparation (understanding of career preparation and personal employability attributes), analysis (ability to apply analytical skills to the gathering of information and the evaluation of one's own work), and communication(effective use of communication). The rating guide provided qualitative descriptions for each of the dimensions for the three performance levels (Basic, Proficient, and Advanced). Raters were asked to review a student's work, considering each of the evaluation dimensions. They were then required to assign one holistic score (Basic, Proficient, or Advanced) based on their overall impression of the student's work, all dimensions considered.

Benchmarking sessions were held in the summer of 1993 using the above process. Basic and proficient benchmarks were identified for the career-technical areas of animal science, child development and education, computer science and information systems, construction, and health careers core. The benchmarking session provided invaluable information that was used to refine the portfolio entries and requirements. However, as a result of these refinements, the benchmarks identified during the 1993 benchmarking session were not valid for subsequent years.

To give teachers and students time to adjust to the changes that were made, whole portfolios were not reviewed in 1994. In the summer of 1995, portfolios were evaluated by subject area. Sample portfolios that reflected proficient work were identified. Because these portfolios were to serve as exemplars for training, minor edits were made to assure clear adherence to portfolio requirements and performance standards.

During 1996, the holistic scoring method previously used to evaluate portfolios was reconsidered. Following a review of the results of several similar assessment efforts around the nation, a modified analytic approach was adopted to help ensure acceptable levels of reliability. The modified method requires scorers to provide actual ratings for each of the dimensions before providing an overall holistic score, so a portfolio now receives multiple scores rather than just one. WestEd added this step to address two possible sources for error in the rating of student portfolios--variations due to student performance and scorer variability. The quality of student performance can vary across dimensions. By evaluating each of the dimensions separately, raters can note this uneven student performance. Raters can also show variability in the scores they assign to individual portfolios. Requiring separate dimensional scores allows for the comparison of scorers at the dimension as well as the holistic level.

The developers of C-TAP also hoped that the use of dimensional scores would yield instructional benefits. Holistic scores alone do not provide teachers or students with specific information about relative strengths and weaknesses. The dimensional scores help support and explain the overall holistic rating by identifying those dimensions in which a student is showing satisfactory or exemplary performance as well as those areas needing improvement. Dimensional scores also can be compared with other instructional and assessment indicators, e.g., classroom grades, test scores, and standardized test and subtest scores.

The new dimensional scoring guide for portfolios includes the same four dimensions described earlier with minor revisions, and adds a fifth dimension by separating knowledge and application of content knowledge into two distinct dimensions. Previous raters indicated that students may demonstrate technical knowledge, for example in a writing sample, but are not always able to demonstrate the ability to accurately apply that knowledge. This lack of continuity between knowledge and application led to the separation of these two dimensions.

In addition, a fourth performance level (Low Basic) was added for the dimensional ratings and the overall holistic judgment, so each of the dimensions is now delineated into four performance levels rather than three. The new performance level was included in direct response to requests from teachers who found that three levels did not adequately represent the range of student performance. They considered it important to distinguish students who failed to make a full attempt to meet the requirements of the portfolio (Low Basic) from students who extended the effort but did not achieve at the proficient level (Basic).

Project Scoring

Because of their somewhat limited use, projects did not require extensive scoring sessions like those held for portfolios and written scenarios. Instead, focused review and benchmarking sessions were conducted for the C-TAP project in 1994 and 1995. Both sessions involved teachers who had implemented the C-TAP project in their classrooms. In 1994, teachers were brought together with WestEd staff and CDE representatives to review projects and select benchmarks for each content area. In the summer of 1995, teachers, WestEd staff, and CDE representatives reviewed student projects and identified student work to serve as exemplars in the following career areas: agriculture core, animal science, computer science and information systems, marketing, health careers core, child development and education, food service and hospitality, technology core, and construction technology. These samples were edited as needed to assure strict adherence to the project guidelines and were disseminated statewide as examples of student work meeting the C-TAP project requirements for each content area.

In 1996, as with portfolios, a decision was made to move toward dimensional scoring of projects. New dimensional scoring guides were developed for the project and the associated oral presentation following a process similar to that used for the portfolio. WestEd is in the process of soliciting feedback on the revised scoring guides from teachers and others actively involved with the C-TAP project.

Scenario Scoring

Until the 1995-1996 academic school year, written scenarios were scored holistically, using a generic written scenario rating guide that was the same for all scenarios regardless of career area or occupational cluster. The rating guide consisted of three scoring dimensions: content (the breadth, depth, and application of knowledge of major ideas and concepts related to career-technical MCS), analysis (the evaluation of evidence presented in the scenario and quality of response to all instructions and requirements in the written scenario assessment task), and communication (the effective use of written communication). Using the holistic rating guide, scorers were asked to review student responses to a scenario, considering all three dimensions, and then assign a holistic score. When determining holistic scores, raters were told to give less weight to written communication than to the other two dimensions because the scenario was not designed to focus too heavily on written communication skills.

In 1996, a decision was made to use both a generic guide and item-specific guides for the scenario. The generic scoring guide used previously is now intended to facilitate the development of item-specific guides tailored to the individual requirements of each written scenario. This change was made because it was generally agreed that item-specific guides would provide more specific information to scorers, making judgments more objective and reliable.

Just as the portfolio and project moved toward dimensional scoring and a four-point scale in 1996, so did the written scenario. In the process of refining the generic and new item-specific guides to reflect these changes, project staff conducted a review of written scenario responses across various content areas and prompts. During this review, it became clear that specifying separate criteria for the dimensions of content and analysis was not meaningful. That is, staff found it difficult to tease out evidence of content knowledge/application versus analysis in a written scenario response. As a result, the written scenario guides were refined to encompass only two dimensions--content and analysis and written communication--instead of the original three.

To address content validity, WestEd established two committees of experts to advise on the type of performance assessments to include in the C-TAP system. These committees were influential in the initial decision to limit the use of multiple-choice questions. They also provided input into the structure of the portfolio and its contents.

The advisory committees were replaced by development committees, each focusing on a specific occupational cluster. These committees were made up primarily of career-technical teachers but also included academic subject teachers, postsecondary faculty, and employers. For the portfolio, these committees produced lists of topics for the writing sample and ideas for work samples. For the project, they generated sample project aims to reflect appropriate content and scope. The development committees also drafted written scenarios that WestEd staff revised and field tested.

Content validity was also examined using focus groups. These groups reviewed completed student work to determine whether it demonstrated the skills desired in the workplace. Although this process has been somewhat informal, the results have been positive. WestEd plans to establish more formal focus groups made up of employers, industry practitioners, teachers, and postsecondary educators in the relevant career fields. These focus groups will consider whether the current form of each component of C-TAP and its scoring process are valid assessments of career-technical knowledge and skills. No other validity work has been done, such as correlating C-TAP ratings with other student ratings (such as test scores or teacher ratings) or using C-TAP ratings to predict student performance based on particular criteria.

CONSEQUENCES AND USE OF ASSESSMENT RESULTS

Because C-TAP has not been used statewide as a formal student certification system, it has not been implemented in a standard manner at all schools. Teachers differ in their use of C-TAP components and in the importance they place on C-TAP work when evaluating students.

C-TAP, especially the portfolio, is intended to document students' knowledge and skills and has been promoted as a means of impressing employers and postsecondary institutions through evidence of student achievement. Some teachers are somewhat skeptical of the usefulness of C-TAP for these ends. They believe that postsecondary institutions are generally unwilling to use portfolios alone for making admissions decisions. One teacher did note, however, that colleges were more willing to give college credit for her high school course upon reviewing her students' portfolios. Perceptions of C-TAP's usefulness for job seeking were more mixed. Anecdotal evidence suggests that some employers are impressed by portfolios while others are not. Overall, teachers found it challenging to get employers to consider portfolios when hiring. Teachers, however, argue that having portfolios gives students examples of their work and accomplishments that can be shared and discussed with potential employers who show interest. Where students do internships while taking classes, C-TAP may also generate employer interest. For example, a health teacher noted that several dentists asked for student interns who would be producing work samples as part of their internship.

The purpose of C-TAP is still evolving from the original idea of a student certification program. To different degrees in different schools, it has modified instruction, become a source of information for grading or a requirement for passing a class, and created a record of student work that may convince both a student and a prospective employer/admissions officer that the student has learned useful skills. WestEd is in the process of evaluating C-TAP's impact on instruction and curriculum by surveying and interviewing teachers.

APPLICABILITY TO VOCATIONAL EDUCATION

C-TAP is a vocational assessment system, so the key question is whether it could be implemented on a widespread basis. This question is difficult to answer for two reasons. First, C-TAP has been adopted schoolwide in only a limited number of cases. Second, not all teachers surveyed had adopted all three components of C-TAP. Some argue that there is not enough time to do all three, and others say they will adopt the project and scenario only after they have become comfortable with the portfolio. There has also been some concern that scenarios would be used to evaluate teachers, because they were originally designed to be centrally scored. For these reasons, the focus here is solely on the feasibility of implementing C-TAP portfolios. Feasibility includes the effects of portfolio use on teacher time and responsibilities, plus the reactions of teachers, parents, students, and schools.

Teachers agreed that using the portfolio took substantial amounts of class time, especially in the first year. Much of the time went to explaining what was expected and giving students adequate time to carry out certain parts of the portfolio that were not traditionally part of the class (e.g., resumes and letters of application). For most teachers, the work samples came from existing classroom activities, but extra time was still required for students to write summaries. Teachers did give up teaching some material to make time for the portfolios.

Additional instructional demands are placed on teachers who use the portfolios. They have to focus more on writing, especially specialized writing such as resumes and job/college applications. Additionally, class management skills play a role, because students complete work samples at different speeds. Grading does not appear to pose an additional burden when the portfolio is used to supplant traditional tests or practicals. In fact, teachers said portfolio grading was easier than the traditional type because it is obvious from work samples whether students understand the material. If traditional assessments are continued, however, C-TAP requires additional grading time.

A further demand on teachers is the need to create storage space for the portfolios. Portfolios can take up substantial amounts of classroom space, and they must be easily accessible to students. Some teachers want to keep all the work samples students have done in class rather than just those selected for the portfolio. Students can then change their portfolio contents as needed. Other teachers are trying to determine where to store portfolios that will be maintained over all the years students are in the program.

The teachers surveyed believe the portfolio is a valuable approach and see a reduced learning curve for other teachers who subsequently adopt it with some help. Some teachers have been involved in training large numbers of their colleagues in the portfolio's use and believe that teachers with different levels of experience will vary in their willingness to use it. New teachers, for example, may not have time to adopt portfolios, while older teachers, especially those near retirement, may not want to invest the time. They do not yet see strong support from school administrators for using the portfolios. Students seem to resist the additional work requirements of the portfolio, but over time they come to accept them. Some students value having the record of work when they approach employers. Teachers have given portfolios a fairly low profile as they learn to use them. For this reason, there has been little community response to them up to this point. Parents are generally not familiar with portfolios, but usually react favorably when teachers explain them.

In conclusion, widespread adoption of C-TAP portfolios by individual teachers appears to be feasible, if teachers can find time to learn how to use them and agree that substantive course material may have to be dropped, at least initially, to free up time for their use. WestEd is distributing the cluster-specific supplements (which contain the career-technical MCS, ideas for projects and writing samples, and an example of a work sample write-up) and the examples of proficient student work (each containing one portfolio or project and together covering the occupational clusters). Teachers and students can use these to gain a better understanding of how to implement C-TAP and what completed projects and portfolios should look like.