Our project was undertaken to investigate the utility of alternative assessments for vocational education and to provide vocational educators with guidance in evaluating different strategies for assessment. The case studies provide a rich set of illustrations of the range of constructed-response measures that are available to vocational educators and the purposes they might serve. However, the cases do not identify a set of best practices or a simple formula for choosing among alternatives. Instead, our research suggests that vocational educators must make their own choices from a growing range of options. That information may be interpreted as either bad or good news. Some may long for a simple all-purpose solution; for them, the results of this study will be disappointing. Others may be excited to learn that they have considerable freedom to craft assessment systems to meet their needs.
Our project will have value if it helps vocational educators make better assessment choices. To that end, we have discussed a number of elements that need to be factored into assessment decisions. The choice of an assessment strategy should depend on the purposes to be served, the quality of the information desired, and the feasibility of different alternatives within the local context. Our six case studies illustrate how different programs have crafted assessment systems to meet specific needs. In addition, these case studies suggest a number of other factors educators must address when thinking about assessment systems.
The goal of the first two sections in this chapter is to illustrate how the information from the previous chapters might be used to address the needs of vocational educators. Two common situations--assessment for program improvement and assessment for certifying student mastery--are used to show how assessment planning can be informed by the results of our study. The last section then provides conclusions.
Dale McIver teaches in the office automation program at Watson Tech, an area vocational technical school in Dade County, Florida. She teaches a three-course sequence leading to a certificate in office machine operation, but few of her students complete the full sequence. Instead, her classes are primarily composed of students wanting to gain some initial familiarity with computerized text and data processing or wanting to upgrade their skills in particular ways. Dale is frustrated with her current grading system, which is based on unit tests from the class workbooks. The test scores do not give her or the students enough information about the students' abilities to respond to realistic office demands that involve automated equipment. She is looking for an assessment system that will engage her students more, help them understand their own strengths and weaknesses, and provide her with information to improve her instruction. She believes there is too much emphasis on rote learning of commands and functions. She wants her students to be better problem solvers when it comes to using computers in the office environment.
Dale's situation should be familiar to vocational educators because the changes she is experiencing are widespread. In fact, part of the motivation for this project was to address these new challenges facing vocational educators. Our approach to solving Dale's problem was to review the various assessment options in relation both to her purposes and to the quality and feasibility issues discussed earlier.
In this situation, the broad purpose for the assessment is unambiguous. Dale's interest is in improving the courses she teaches. She wants information to help students focus their efforts and to help her determine which skills need additional emphasis. She also recognizes there have been changes in the nature of the skills to be taught and the needs of the students who are enrolling, and she hopes the assessments will be responsive to these conditions. In particular, she wants information about how well students would respond to realistic problems.
All of the alternative assessment methods we have described could be used in this situation. Students could write extended descriptions of the procedures they would use in a particular office situation. Dale could develop realistic office tasks for students to perform and then judge their products. A course-long project culminating in a formal presentation is a possibility, but it seems less suitable than the other alternatives. A portfolio containing a collection of student products would work well. If part of Dale's goal is for students to build a repertoire of solution strategies and a command of the technology, then a collection of successful products might be a good contribution.
It is important to recall Dale's purpose and to consider how well each type of assessment would help the students and Dale to improve. We cannot analyze the situation completely without a clearer understanding of the type of problem-solving skills Dale hopes to foster and how well they can be embodied in the assessment. For example, does she expect students to be able to revise a document based on editor's marks, produce a presentation-quality organizational chart based on a sketch, or compile a report that includes text, tables, and a graph? Does she expect students to find a file without knowing its filename, recover a document after a power outage, or repair a faulty disk drive? These are all problems one might encounter in an office setting, but their solution requires different knowledge and skills. Some problem-solving abilities might be measured adequately with written questions, others with performance tasks, and others through an extended project.
Dale's plans for grading also play a role in selecting an assessment. Is the manner in which students produce the product important, or just the quality of the result? Some grading standards may be more helpful than others, just as certain types of feedback may be more informative than others. Dale knows (or can find out) the answers to these questions, so she can factor them into her evaluation of alternatives.
One of Dale's purposes for the assessment is program improvement. If it is to improve her courses, the assessment must provide information that is easy to link to instruction (either to particular units or to behaviors). Will Dale know what to do if students do poorly on one aspect of the assessment?
In Dale's case, it is not necessary to place a premium on technical quality. Students will have multiple opportunities to perform in class, and the assessment results will not be used for critical decisions. Dale need not be overly concerned about the accuracy of scoring. Teachers make judgments about student performance all the time, and there is no reason to think Dale will be unable to judge the responses to the assessment tasks fairly. In addition, if the results are going to be used to help students judge their own skills and to inform changes in lessons, validity is not of great importance. However, if Dale wants to draw inferences about broader behavior, such as problem solving in the office environment, she will need to collect far more information to test their validity.
All the alternatives are feasible, but each will increase the burden on Dale's time compared to her current assessment methods. None of the options involves great financial costs, but Dale will have to be willing to bear added preparation burdens. The portfolio appears to be the most demanding because of the time needed to organize it and assess the products at the end of the term. None of the alternatives appears to be so complex that Dale will need specialized help, or so unusual that she will encounter resistance from students or faculty.
Administrative issues are less germane to this situation. The vignette does not clarify whether Dale's problem is unique to her or shared by other teachers in the district. It might be possible to share the development effort with other office-automation teachers, so long as all remained engaged and committed to the work.
Some of the other factors discussed in Chapter Five come into play as well. The choice of single versus multiple types of assessment should probably be made after clarifying the kinds of problems students need to solve. It may be that a series of similarly organized scenarios can capture the desired problem-solving skills well. If the assessments reflect the course goals, then Dale will probably want to make them mandatory and give them considerable weight in determining course grades. Again, because the assessment is supposed to be closely linked to course expectations, it might be possible to embed it into class lessons in a relatively seamless way.
J.C. San Martino is the coordinator of the automotive repair program for the Fort Meede School District. He supervises seven teachers in five high schools and one vocational school. Historically, the program has been very successful in preparing students for entry-level jobs in service stations and repair shops. In the past two or three years, however, local employers in various parts of the district have complained that graduates are not as good as they used to be. It has been hard for J.C. to respond, because the complaints are all different, but the common thread seems to be that students are very good at some things but have gaps in their training. Employers are losing confidence in the district's program, and they are growing cautious about hiring graduates. Although every school uses the same curriculum and teaches to the same set of competencies, each instructor is responsible for his or her own testing and grading. J.C. thinks that a common assessment system might help him raise standards in the program, assure employers that graduates are competent, and encourage instructors to provide more consistent training. He has considered offering employers a guarantee that graduates from the auto repair program will meet agreed-upon standards, but is not certain he could develop a system that would support this claim. He also wants to be sure that the assessment system provides information to help teachers improve their programs.
Comments from employers have led J.C. to question the quality and consistency of the training being provided in the district. He wants to use assessment to certify student competence, but he realizes that he must also provide information for program improvement or he will never achieve the certification goal. The problem is complicated by the fact that there appear to be problems with the individual schools, as well as with the overall program. So the assessment system needs to provide information to certify mastery and to identify shortcomings in the instructional program at each location.
More than one type of assessment can address J.C.'s concerns, but not all the approaches described in Chapter Three are equally helpful. Because the auto repair program is organized around a set of competencies and because employers have expressed concerns about specific skills, the assessment should relate to these specific program elements. Written tests and performance tasks can both be useful in this regard. Senior projects are less helpful for providing information about specific learning outcomes. Portfolios might be useful if constructed such that they contain information linked to core competencies and skills.
One issue J.C. must address before designing the assessment system is whether the course curriculum is aligned with the needs of local employers. It might be that the demands of entry-level auto repair jobs have changed in the past few years and the curriculum itself needs to be updated. Students may need to acquire new skills, such as using new finishes or new application procedures, or they may need to be prepared to work in different arrangements, such as multiperson teams. Reaffirming the link between course content and employer needs is an important first step; it also provides an opportunity to involve business and industry representatives in the program.
There are many ways to measure the competencies students are being asked to master, and J.C.'s biggest challenge will be choosing methods that strike the right balance between quality and feasibility. Strictly speaking, the "guarantee" he hopes to give to employers is not a binding contract, and there is no reason to apply the same quality standards one would apply to a licensing examination designed to protect public health and safety. Nevertheless, J.C. wants the effort to have merit, and he is particularly concerned about the consistency of performance across schools. It is not enough for each instructor to check off each competency as it is mastered; J.C. wants to impose some common external measures that have credibility with employers. Therefore, he may want to use common written exams to test knowledge, coupled with common performance tasks to measure some applied skills. Using the same scoring procedures for all schools will provide the comparability he desires, and assessment certainly should be mandatory for all students. This may also be an excellent opportunity to use industry representatives as judges to review student performance and provide external validation for the quality of the work. He should certainly investigate existing assessments that have been developed in conjunction with automotive industry organizations; it may not be necessary to develop new assessments if credible ones exist.
He must also keep in mind, however, that standardized measures (written or performance) require time and effort, and too much testing takes away from learning time and may annoy participants. Because there are many ways to measure competencies, it may make sense to measure some skills using quick and less intrusive methods (such as a checklist initialed by the instructor) and to measure other, more difficult or important skills using more time-consuming methods (such as standardized written tests or standardized performance assessments). In some cases, improvement may come from merely requiring students and instructors to monitor their progress against a master list of competencies. Requiring a portfolio in which students compile evidence of mastery of all course competencies is one alternative J.C. might consider.
It probably would be wise for J.C. to involve all the auto repair instructors in the development of the assessment system. This will help them understand the need for the system and increase their commitment to using it. More importantly, the instructors will have useful insights into ways to measure various skills, including those that may be difficult to measure (e.g., working in multiperson teams). For example, it may be possible to have embedded components in which existing class projects become the measurement tool for certain skills. Teachers also may have a better sense of the demands that certain choices will place on their time and the students' time. If the system is meaningful to instructors, it will increase the likelihood that they will participate enthusiastically. If instructors incorporate performance on the assessment into students' course grades, the system will have more meaning for students as well.
The results of our study indicate that alternative assessments can be useful tools for vocational education, but that vocational educators must learn to be wise consumers and users of assessments. Our examination of cases illustrates the breadth of assessment options available, from open-ended written assessments to performance tasks to portfolios. Each has been used effectively on a large scale in at least one location, and all appear to have potential for vocational education.
For vocational educators who do not know where to begin, we have suggested an approach for choosing among alternative assessments. The first step is to clarify the purposes of the assessment and the specific conditions of the vocational context. These conditions might include the needs of constituents, the demands for accountability, and the nature of the skills to be assessed. Next, one should consider a wide range of assessment options. The case studies illustrate a few alternatives that have been used in practice. Reading the complete descriptions in the appendices conveys a fuller picture of the demands of each situation and the strengths and weaknesses of the approaches taken. A thoughtful educator would supplement our findings with information from colleagues and professional organizations about other assessment methods that might be used or adapted.
Educators must also consider the issues of quality and feasibility. The manner in which the information will be used determines the level of technical quality that needs to be achieved. In general, the more importance attached to the use of the information, the higher the necessary levels of reliability and validity. However, quality concerns should be balanced against practical realities. Cost, time burdens, and acceptability by stakeholders are also important considerations in selecting assessment methods. Some approaches are cheaper, less intrusive, and more familiar than others.
In the end, no single assessment approach is best for all situations. However, fairly simple considerations can guide assessment planning. Our study illustrates the breadth of the existing alternative approaches, their utility in the vocational context, and some of the trade-offs vocational educators face when making choices among them.
[4]The individuals, schools, and programs in this example and the one that follows are fictitious.