Vocational/Industrial Clubs of America (VICA) is a national organization for secondary and postsecondary students in vocational/technical fields. The occupationally oriented skills tests that form the centerpiece of VICA's national conference, called the Skills USA Championships, cover a broad range of vocational fields, test generic and job-specific skills, and use several different forms of assessment. VICA, a nonprofit organization, has obtained assistance from corporate leaders and practitioners in some sixty fields to develop its tests, which are designed to measure skills required in those fields anywhere in the nation. The main purposes of VICA's contests are to document students' skill mastery, encourage excellence, and increase the competence of entry-level workers. Indirectly, VICA also aims to improve instruction and curriculum.
There are two main types of contests, the job skills contests, in which individuals compete in performing job-related skills and applying relevant knowledge from the vocational field they are studying, and the leadership development contests, which consist mainly of demonstrating generic and employment-readiness skills, but also include more academic skills, such as speech making and knowledge of parliamentary procedure. The national competition is the culmination of local, district, and state contests in which winners proceed to the next higher level.
The VICA assessments focus mainly on demonstrating hands-on occupationally specific skills. Many, however, also call for cluster-focused or more general, industrywide knowledge (such as fundamentals of electronics, in the electronic products servicing contest); generic-thinking, decision-making, or troubleshooting/diagnostic skills (e.g., in automotive repair technology); or specific academic skills (e.g., math skills in cabinetmaking).
In many of the contest areas, a written exam is included; items are usually multiple choice, but sometimes are constructed response. The different components are designed to test the skills and knowledge a person needs to work in a specific occupation or occupational group, so the parts are intended to complement one another. Figure F.1 gives detailed descriptions of several job skills tests.
Test
results are used to determine first-, second-, and third-place winners in each
competition among secondary students and then among postsecondary students.
[11]
The winners are determined by the total scores on all components, or contest
stations, from all judges. Many times, the scores are so close that a fraction
of one point (out of a score typically in three digits) determines who wins.
The
top three scorers in each category (secondary and post-secondary)
win medals and prizes such as scholarships and equip-ment. (Other participants
are given certificates.) Each first-place winner in a skill area that is also
part of the international competition in which VICA participates gets to
compete in a runoff against the winner in that skill area from the previous
year. Runoff winners then receive additional in-depth industry-sponsored
training to prepare for the international competition.
The contests are judged using criterion-referenced scoring. In most cases, contests combine objective questions that have a single right answer (e.g., a multiple-choice, yes/no, or mathematical question) with aesthetic or subjective judgments about student progress (e.g., in cosmetology, advertising design, culinary arts, and even in skill areas such as cabinetmaking and precision machining). In some cases, points are awarded for overall quality of a project, introducing a holistic element to the scores.
The vocational-technical areas represented in the skills contests range from trade and industrial crafts (auto mechanics, brick masonry, carpentry, printing) to home economics-related or service occupations (culinary arts, commercial sewing, cosmetology, practical nursing, advertising design) to emerging or rapidly changing technology-based occupations (automated manufacturing technology, robotic workcell technology). Some leadership development assessments pit teams of students against each other, but most require strictly individual performances. Brief descriptions of the specific tasks assigned for a few of the assessments follow. Electronic Products Servicing
Law Enforcement
Culinary Arts
Precision Machining Technology
Advertising Design/Commercial Art
Carpentry
Practical Nursing
Job Skills Demonstrations (in Leadership Development contests)
|
The leadership development competition goes on for several days and includes the opening and closing ceremonies (group competitions that test public speaking, organization, synchronized movement, and memorization). The job skills contests take place during a single day. Competitors do not find out who won until the last night. At the awards ceremony, first-, second-, and third-place winners are announced in front of the thousands of conference participants, family members, and guests; they then go to the podium to receive their medals (gold, silver, and bronze; respectively). All contestants receive a certificate for having competed, and the other top-ten finishers (those who place fourth through tenth in each contest) receive special recognition on their certificates.
The tests are all developed by technical committees. For the leadership development contests, these committees consist mainly of vocational-technical teachers; for the job skills tests, committee members are current or former professionals in the particular occupation or industry, along with some people who work for a relevant industry association or labor union. Only these industry representatives set standards and design assessments, but other committee functions (such as running the contests) may be performed by teachers as well.
The technical committees develop the tests through a series of meetings and conference calls; they solicit feedback from teachers about the feasibility of new assessment ideas. The process is somewhat informal and varies considerably from one skill area to another. In a very few areas--automechanics, electronics, precision machining, and technical drafting--the committees have used the standards produced by national skill standards development groups to guide their test content. The committees may draw on task lists for an occupation or on tests developed at the state or local level, changing test questions substantially to prevent unfair advantage going to one or another state's competitors.[12]
The people who participated in the job skills contests as competitors, judges, technical advisors, education team members (for liaison and communication), and advisors/chaperones were in general quite positive and enthusiastic about the VICA program and the system of job skills contests, including the local, state, and national competitions. While it is not surprising that students who had won local and state competitions were excited about attending the national conference, it is more telling that high-level managers and other employees in the industries VICA serves were strong supporters of the program. Many have been involved year after year in designing the tests, working out logistics, or judging the competitions, and many were taking personal time off from their jobs to do this work (entailing several days to a week of very long hours). If there are any downsides to VICA's contests, they were not apparent at the Skills USA Championships.
In some cases, this dedication may reflect appreciation for VICA's having helped them at the start of their careers, but it also attests to positive impressions of VICA's ongoing work; it seems unlikely that so many professionals would continue to work on the conference and persuade their companies to donate equipment and materials if they were not consistently recruiting good employees through VICA. Industry representatives, mainly managers or laborers in the skilled trades, are complemented by industry association employees, vocational teachers (secondary and postsecondary), labor union representatives, and vocational administrators (school or local/state agency) in putting together this wide-ranging conference.
The contests contribute significantly to VICA's overall goal, which is to prepare qualified and highly motivated workers, mostly for occupations in the skilled trade and industrial sectors, some of whom go on to become managers and leaders in these sectors. In preparing for the contests and working with teachers who have access to VICA's other offerings, students should gain not only the practical knowledge demanded by their industry, but generic workplace qualities and skills needed for entry to and success in any industry (such as dependability and integrity, and teamwork, decision-making, and communication skills). The contests are just one part of VICA's overall program, which also includes teacher training workshops, publications, and access to personal networks in industry.
VICA aims for its contests to be closely related to instruction in classes taught by VICA members, though these ties undoubtedly vary across competition fields and across instructors. The tests seem to drive the curriculum content more than they respond to it. The skills and knowledge tested by VICA's Skills USA Championships should in most cases also have been tested earlier, in the classroom, though the national contests may serve as a kind of final or comprehensive exam, not just for one course but for a program or group of related courses. VICA's contests bring together more content elements than a typical school test does, and may use more current or cutting-edge technology than some schools have. (This can be a problem, be-cause some students will be less prepared than others through no fault of their own.) VICA distributes the topics for next year's leadership contests in advance to teachers and student members, so they have a good idea what to emphasize. Previous year's project plans are released to familiarize teachers with contest procedures and to aid the planning of local and state contests. Moreover, the specific competencies that will be tested are listed in the official technical standards, which are revamped every three years.
In most of the skill areas, training seminars are offered to teacher members through the state VICA offices on a regular basis. These seminars serve at least two purposes: keeping teachers informed about changes in the industry or occupation and new curricular or teaching materials, and gathering data from teachers about topics on which they need more information. At these sessions, teachers can discover tasks that may be useful in their own classroom tests and discuss curriculum and instructional issues with others from around the state (or country, since seminars are also given at the national conferences). Teachers also have opportunities to make contacts with industry representatives for field trips, internships, and mentorship opportunities. Teachers may even be able to earn college or inservice training credits by taking a lead role in running the state VICA competition or teaching continuing education seminars.
The links between classroom/workshop instruction and these national contests mean that students at the national competition should be familiar with the format of the tests and should know in advance what content areas will be tested. For one thing, those who make it to the nationals have usually won in the same area earlier in the school year at their state competition.[13] In addition, teachers are supposed to have the official technical standards, which provide specific information about the competencies that the national contests may test. However, in practice many teachers do not have the current version. This problem was especially common among teachers who taught contestants in the leadership development areas. This lack of initiative on the part of some teachers is mystifying, given that the offical standards cost only $10 and are updated only once in three years.
The VICA contests started in the mid-1960s with only three contest areas (all in leadership development) and have grown steadily since then. The national job skills and leadership development events now involve thousands of students in dozens of competitions. These assessments are implemented annually from the local to the international level. (Only certain areas are competed in internationally, the choice depending on the presence of strong programs in enough participating countries. Skill areas differ among the other levels of competition as well.) VICA members in the state hosting the national conference, especially the teachers, play a stronger role than members from other states.
The specific content of tests is changed every year, though most of the subject material covered remains consistent from one year to the next. Students can compete in events for more than one year so long as they still meet the eligibility requirements, so VICA has to change the tests to keep things fair. Students must be active VICA members to compete, and must compete during the school year in which they are enrolled in a course or program in the same subject area as the competition (for local, regional, and state competitions) or at the end of that year, in June (for the national competition). Tests used at the national level are released for use by state, regional, and local competitions in subsequent years; these are usually modified and shortened to suit the state and local contests, which generally are finished within one day. Teachers can also obtain these tests for use in their classrooms.
Judges for the job skills tests are (or recently were) professionals in the relevant field, i.e., they perform or supervise the tasks being tested or set standards in the occupational field. Judges for the leadership development contests are teachers or professionals in the relevant field. All judges receive training in what to look for, how to score, how to strategize in assigning scores so that there is room above a strong performer for an even better or perfect score, and so on. To encourage consistency in judging, new judges are always put on a team with experienced judges and encouraged to ask questions; the new judge's work is carefully monitored by the experienced judges. All judges attend a "familiarization session" for contests that have new equipment, computers, software, or a dramatically different test element; these sessions can last up to a full day. The extensiveness of the judges' training varies from one field to another, but no one interviewed suggested that the judges received less than adequate training.
As to the fairness of the judging, people interviewed thought that each judge was fairly consistent across the projects he or she judged, even though there could be a wide variation across judges' scores for one contestant or on one test component. Since the scores used for ranking contestants are the totals from all judges, one judge being consistently lenient should not affect the relative position of any contestant. In some contests, one highly experienced judge scores all contestants' products or papers for a component of the test. One line of thinking is that VICA student members should get used to judging that may be subjective, since such judgments occur on the job. National VICA staff say that there has been general acclaim for the fairness of the contest judging--but there are no hard data, such as external reviews from unbiased experts, to confirm this.
No research has been done on the validity, reliability, or equity of VICA's assessments (either test content or methods). VICA has kept electronic records of the score results from the national contests for the last five to six years, so some of the data needed to do such research exist. However, there has not been sufficient demand for it to be done.
Those involved with VICA believe that the tests are highly valid because industry is extensively involved in designing them. Industry representatives work hard on many different tasks at the conference and on activities during the year leading up to it; they also secure funds and equipment. Their motivation is to produce future workers who may contribute to their company's success, so they want to make the tests fair, challenging, up-to-date, and authentic. Teachers also work hard, because they want to improve their students' chances for success. The result is that everyone involved has strong, consistent motivations.
There are contests in which logistical or other difficulties tend to obstruct the fairness of test administration, however. These are probably fairly rare occurrences that usually happen when a skill area is completely new or a new element is introduced to an existing contest. For example, in this year's advertising design contest, four problems led to unfair advantage for certain contestants in the view of one teacher/advisor. First, the technical committee did not allow adequate time or opportunity for competitors to clarify the meaning of terms they had not heard before (they may have been familiar with the concept but by another name). Allowing such discussion adds authenticity to the experience, since an ad agency employee would normally be able to clarify what the client wants before proceeding.
The other three problems had to do with the fact that part of the competition was done on computers at a local school. First, and perhaps most significant, students who were familiar with the software used at that particular school had a distinct advantage, especially since it was a timed project. [14] Second, some of the printers failed to work properly, wasting time for some students but not others. Third, a class was being conducted in another part of the room where some contestants were working, causing distraction--but for only some students.
Informants believed judges' scores were adequately reliable. Whenever possible, judges are given objective criteria to use in making decisions about correctness of responses or performances. In cases where subjective judgment enters into the score (particularly in aesthetic areas such as how pleasing or effective a graphic design or advertising concept is, or how food looks and tastes), there is probably more variation among the judges' scores. However, these differences likely even out in the total scores (assuming that judges are consistent across the different contestants). Informants did not feel a need to formally measure the reliability of the scores.
People running the contests strive for fairness. On one troubleshooting component of the electronic product servicing event, some of the contestants had a faulty reading on a diagnostic tool and thought this was the problem they were being asked to identify. Although the person who designed this part of the test thought they should have known this was not the intended problem, and said so at the debriefing, the technical committee eliminated that part of the test after several contestants filed a grievance to challenge their scores.
Debriefings are held in many contest areas to go over the correct answers, section by section, and discuss how contestants performed in the aggregate. At these sessions, students can learn from what they did wrong (and, to some degree, satisfy their curiosity about what their chances of winning are). Some of the debriefings stuck to the points raised in the contests; others involved long-winded presentations of new materials and equipment, amounting to endorsements/advertising for a company's products (in exchange for the items and/or labor the company donated to the conference). This by-product of the contests' corporate sponsorship is unfortunate, but it is of minor importance compared with what the contests offer in opportunities to excel, to learn from others, and to discover more about employment and postsecondary training options.
Everyone asked about this issue said that it was not possible for a person's gender or race/ethnicity to influence judges of the job skills tests, since in many cases judges fill out the scoring sheets for the whole group of products after contestants have departed, and these are identified only by contestant number. However, in most of the job skills areas, the vast majority of the contestants were either white males or white females. In such a setting, the one or two contestants who do not fit the pattern (e.g., the one female welder among a group of males) may in fact stick in the judges' minds, since judges usually observe the performances as well. Thus, there may be subconscious bias either favoring or disfavoring students who stand out from the group. In some trade areas, the groups were more mixed, especially by gender (e.g., advertising design and culinary arts).
In all of the leadership development tests except those few that involve teams (such as parliamentary/chapter business procedure), judges observe the contestants one by one as they perform, so one cannot argue that the procedures preclude bias. It is therefore possible for unfair judging based on gender or race/ethnicity to occur, though no one asked about this was aware of any such instance. It is also quite possible for judges to take into consideration additional factors beyond the skills and attributes tested, such as appearance or voice characteristics, whether consciously or not.
For the most part, results are used only informally by students, teachers, and employers. The following discussion addresses these groups separately.
Though the goal is to share students' results with them, the students may have graduated from high school or their postsecondary program just before competing in June and may not be in touch with their teachers during the summer. Although scores and rankings are given to all state VICA advisors at the close of the awards session, in some cases they are not disseminated further to teachers or students. The actual results are not used for formal decisions about a competitor's schooling or employment. However, individual student competitors use the experience of preparing and participating in various ways: to make contacts for jobs or further training, to help decide on particular avenues to pursue within an industry or job category (e.g., whether to strive to be a charge nurse, who manages other nurses in a unit), to bolster his or her resume, to learn to handle pressure, and to learn skills or helpful tricks from other competitors' performances.[15]
Student participants reported increased enthusiasm for schoolwork, higher self-confidence and aspirations, more assertiveness, gains in teamwork skills (for some events), and better decision-making ability. Results mentioned by students were generally quite positive. On the downside, some students experience anxiety or disappointment if they do not win, and a small minority even show temporary symptoms of illness from the stress. A small percentage of students are unpleasantly surprised to discover that industry standards represented at the VICA conference are higher (or cover different content) than standards at their school or at the state competitions. A small proportion of students interviewed mentioned that they found it difficult to deal with strains on friendships when they won a contest but a good friend did not, especially when they were in direct competition with the friend (this is far more likely at the state and local competitions). These negative effects are minimal, however, compared with the substantial positive effects cited by all students interviewed.
Students from different backgrounds may gain different benefits from participating in the contests. An outstanding high school student in a college-prep or other highly rigorous program may sharpen his or her interpersonal, communication, and public-speaking skills through contest preparation or other chapter activities. An average student in a general or vocational track may get involved in electronics or machining through VICA, do well and gain confidence, take more math classes, and decide to enroll in a technically oriented college program. There are even limited roles for developmentally disabled or other special-needs students: the custodial services competition is restricted to students with individual educational plans. Several student competitors at the 1995 competition had a physical disability. Teachers thought that most students possessed a high degree of motivation before deciding to participate, but in a few cases the rewards offered by the VICA program may have reached out to at-risk students. For example, one teacher of law enforcement/criminal justice issues from a poor, small-town school in Texas said he had used the lure of a steady job and higher chances of entering postsecondary school to turn around some potential gang members.
Teachers are supposed to receive the individual scores for the students competing from their program (one per skill area). They also should receive scores for each state's national competitors and the national average and standard deviation of the national scores. Some teachers reported that they received information from their state director in a form that was difficult to understand; others never received the scores. (What is reported to teachers after the state and local competitions undoubtedly varies too.) Teachers who receive readable results can see which skills they are teaching well or not so well, based on how their students compare nationally.
However, to make sound judgments about how they may need to change their curriculum, teaching methods, or equipment, teachers really need additional information--for example, a description or example of the ideal performance sought by the judges, whether the student was particularly nervous and did not perform up to potential, or whether another instructional factor harmed the student's performance (e.g., the school lacks a particular software program or type of equipment). At least one skill area, precision machining technology, provides a detailed report that explains each item or element at each workstation, including the perfect answer or product the judges were looking for and an overview of how students as a group performed in each station's activities (reporting subscore averages).
Teachers who observe the job skills contests may begin thinking about ways in which to change their teaching if they find that industry demands skills they were not previously teaching or emphasizing sufficiently. Teachers can benefit from this knowledge whether or not they attend the national conference, either by discussing test content with participating colleagues or with contestants who report back from the conference, or by obtaining tests used in previous years. Teachers are likely to revise their definition of acceptable performance (generally by raising their standards) after experiencing the VICA competition. They are somewhat less likely to change teaching methods, because instruction is not discussed or modeled at the conference (though they may learn new techniques informally by discussing them with other teachers).
Teaching to the test was a common effect of being involved with VICA, but since the tests are thought to reflect the knowledge and skills that industry wants, this was seen as beneficial.[16]
Teachers who join VICA may attend seminars with industry representatives and statewide meetings at which they gain access to a range of benefits:
Some employers send representatives to the VICA competitions. These representatives attend partly to recruit employees and may talk to students they have seen performing especially well. Employers do not receive student scores from VICA, but they have access to the list of winners.
The costs of putting together a national competititon such as VICA's are enormous. They cover three main areas: equipment, materials, and facilities; labor for contest design, setup, judging, and breakdown; and personal transportation to the conference site. Most of these costs would not be incurred if a school or district were to implement a similar assessment locally. The equipment required would presumably be available at the school; however, school-owned equipment would probably not be as up-to-date and possibly would be less sophisticated or useful than what is available at the national conference (all of which is donated or loaned to VICA for the event). Materials beyond what a school's budget covers would likely need to be located for a competition (or in-class testing), either by purchase or by soliciting donations from industry. The substantial facilities costs, which VICA pays for the national competition (conference center rental, provision of plumbing connections and special ventilation ducts, electricity use, catering for certain events, etc.), could be avoided at local competitions. The costs of personal transportation (which in VICA's case are paid for by students, their families, school fund-raisers, supporting firms, and school/district funds, especially for teachers' travel) would also be mainly avoided.
The most substantial obstacle in replicating these assessments in schools is replacing the labor involved, especially for test design and judging. Although this labor is all contributed by professionals in the fields being tested and teachers, so there is no dollar amount that needs to be covered, it is unlikely that a local contest could begin to replicate this level of expertise. A key aspect of VICA job skills contests is that the judging is done by expert practitioners, giving both students and teachers a more realistic and current view of the performance expected in the industry than they would get having teachers as judges. Moreover, since the level of competition is bound to be higher with the larger, national pool of contestants, students may not work as hard to prepare for a local event. Knowing who their competition is (which would be the case if competition is restricted to their class, and may be the case if the competition is schoolwide) or sensing that the competition is not very fierce may keep students from putting in the same effort they would expend for a national competition.
However, despite these obstacles, the content of the tests can be useful to individual teachers and groups of teachers meeting to revise their courses or programs. The judging sheets are available in the Skills USA Championships technical standards (revised once every three years), and tests given at the nationals are released for use at state and district level competitions in subsequent years. The input from industry should also provide an important lesson for all vocational educators: even if it is not possible to command the high level of commitment from industry representatives that VICA does in designing and implementing these assessments, schools should seek and use industry's viewpoints and knowledge wherever possible.
[11]These two categories are ranked separately, even though the contests, with only rare exceptions, involve the same tasks.
[12]During an interview at the VICA Skills USA Championships, one technical committee member reported that to guard against possible cheating, a half-hour before a test began he changed a factual element (the type of metal, in automated manufacturing technology) that would change many of the calculations students needed to do. (This interview and all others at the competition were conducted anonymously.)
[13]The official rules state that only first-place winners from the state competitions go on to compete in the nationals, except that if a first-place winner is unable to attend, the competitor with the next highest score takes his or her place. Discussions with teachers/advisors indicated that a similar sifting process is supposed to occur in going from local/district competitions to the state level. However, in practice a small proportion of competitors have not had adequate preparation for the national contests, either because of weak competition in their state or because they did not win in their state but were asked to represent it due to unusual circumstances.
[14]One could argue that if schools want to prepare their students for work, they should have up-to-date computers and use the software most commonly used in the field, but obviously a given student's having worked on specific software is a matter of access and not a reflection of his or her abilities. Information about what software will be used is published in April, but some students may not have access to the right type of equipment to obtain adequate practice.
[15]Students are more likely to have time to observe others in the leadership development events, where they can observe each competitor who follows their own performance, and in job skills tests that take relatively little time to complete and have more than one batch of competitors. In job skills tests that take most of the day, students have little or no time for observation.
[16]It is noteworthy that some teachers, especially for some of the leadership development contests, apparently did not read the specific contest regulations carefully, or at all, and so were not in a position to teach their students to prepare well for the tests.