Appendix C

---

LABORERS-AGC ENVIRONMENTAL TRAINING
AND CERTIFICATION PROGRAMS

---

• DESCRIPTION AND PURPOSE
• RELATIONSHIP TO OTHER PROGRAMS
• IMPLEMENTATION AND ADMINISTRATION
• TECHNICAL QUALITY
• CONSEQUENCES AND USE OF ASSESSMENT RESULTS
• APPLICABILITY TO VOCATIONAL EDUCATION

In 1969, the Laborers International Union of North America and the Associated General Contractors of America (AGC) established a cooperative trust fund for the common purpose of improving the skills of construction laborers. The union sought to increase the demand for its workers, the contractors wanted more productive craft workers, and both parties had a vested interest in creating safer workplaces. For the last twenty-six years, the Laborers-AGC Education and Training Fund has been meeting these goals by developing and supporting occupationally focused courses for sixty-six affiliated local training schools in the United States and Canada. These schools are responsible for training the 350,000 union members (half of the membership) who work in construction or environmental cleanup. More than half of the schools (forty) offer environmental courses in addition to construction courses. Contractors pay money into local trust funds, which pay for running the affiliated schools and help defray Laborers-AGC's costs for curriculum and assessment development and technical support.

The first fifteen years of the fund's efforts concentrated on general construction safety programs and courses on specialized areas of the industry. Though they were developed with union funds and for union members, some of Laborers-AGC's films and course materials were used by U.S. and Canadian government agencies for worksite safety and awareness programs. In the mid-1980s, the fund shifted some of its efforts away from construction. Union officials noticed a significant lack of organized workforce development in the burgeoning environmental remediation industry. Labor market projections at that time exposed a potentially severe shortage of skilled environmental remediation workers. In 1987, Laborers-AGC received a grant from the National Institute for Environmental Health Sciences (NIEHS) to develop a program for hazardous waste cleanup workers. Favorable program evaluations led to further grant awards from the Environmental Protection Agency (EPA), U.S. Department of Energy (DOE), U.S. Department of Defense (DOD), and the National Institute for Occupational Safety and Health (NIOSH).

NIEHS and NIOSH distribute grant funds and monitor administrative requirements, but they rely on DOE and DOD for technical standards and evaluation. These agencies are each responsible for particular environmental areas and must regulate all training programs that certify workers for these fields. Because environmental workers handle substances that pose potentially serious risks to public health and safety, they must all be certified to work, and all their training programs must be formally approved to operate. The fund's program specialists, who develop course curricula and train course trainers, must do so in compliance with the mandates of the federal agencies that oversee each work specialization.

In addition to meeting the requirements of the federal agencies, programs often must meet state agency requirements. The differences among the various state and federal standards make it very difficult for Laborers-AGC to achieve programmatic consistency among its environmental courses. Each course (see Table C.1) is

independent and leads to a specialized certificate, but the fund maintains a single approach for all courses. One standardized element in these courses has been the assessment system used in the environmental remediation courses.

DESCRIPTION AND PURPOSE

The Laborers-AGC Education and Training Fund's environmental training assessment is a flexible system that uses performance-based tests and criterion-referenced multiple-choice tests to measure the competencies and knowledge of environmental trainees. The assessments are designed to certify each individual's competence, as well as to monitor course quality for the purposes of improvement and to report program completion information to the appropriate federal and state agencies. The programs receive federal funding, so the reporting of results is done both to comply with governmental certification requirements and to maintain quality standards and accountability for the ongoing grants or contracts.

The fund developed each assessment tool by employing an assessment expert to work with each course's program specialists and industry experts. In some instances, staff from regulatory agencies were consulted on specific issues. The cost for developing the written and performance assessments was $10,000 to $12,000 for each course, all of it covered by grant funds. Laborers-AGC staff attribute the relatively low development costs to high in-kind contributions from training school instructors and assessment specialists. Also, the fund is just now working to rigorously evaluate the assessments for validity and reliability .

Courses range from thirty-two to eighty hours in length. Students spend roughly half their time in the classroom and half in hands-on field activities; usually they must pass all of the performance tests in order to continue in the course and be eligible to take the multiple-choice exam given on the final day. When they successfully complete both, trainees gain Laborers-AGC-sponsored certification and can work for up to one year in the particular job. To remain eligible for work, they must complete annual refresher courses that update them on new regulations, procedures, and equipment.

Courses have from one to six performance tests, depending on course curriculum and length; each test may assess up to thirty-five tasks. The performance tests last from five to twenty minutes; the simplest requires a trainee to explain his or her actions while testing certain equipment. On the other end of the spectrum, the trainee may perform a complex series of actions in a simulated work procedure. In some courses, performance tests are distinct events that occur separately from the regular training, whereas in others, such tests are used as a training tool and then later as a measurement tool. In the latter case, trainees pair off during the training event, one evaluating the other's performance using a check-off sheet. The instructor monitors the evaluations, with little interference, and uses the same check-off sheet to test the trainees later. This shared evaluation helps trainees build a strong sense of responsibility not only for their own knowledge and performance but also for that of their coworkers, on whom they will rely so heavily at the worksite.

In the hazardous waste worker course, one procedure that is both a training activity and a tested event is decontamination after simulated field work in a Level-A protective suit. Often called a "moon suit," the Level-A fully encapsulates workers and their protective equipment (boots, hard hat, respirator, and air tanks). A trainee enters the three-stage "decon" area wearing the suit and proceeds through a battery of prescribed steps for washing and disrobing. He or she must properly spray and scrub the suit with disinfectant before moving to the disrobing stage, at which point the trainee removes each layer of protective clothing and equipment. Trainees take approximately fifteen minutes to perform all nineteen steps involved in the decon, each of which must be performed properly and in sequence to pass. There is no limit to the number of retests if trainees fail this procedure, since it must be passed to pass the course.

The performance test criteria vary from program to program, according to the degree of oversight by the regulatory agency. For example, the radiological worker course must meet carefully specified DOE regulations. The performance exam for this course thus utilizes importance-weighted point deductions for incorrect performance on tasks. For example, if a trainee fails to remove protective gloves in the proper sequence, two points are deducted, but if he or she improperly responds to an "unusual radiological event," five points are deducted. The underlying factor that determines each task's point weighting is the potential for health and safety risks to the individual, coworkers, or the public if the trainee makes a mistake. Thus, for instance, three of the performance test's twenty-three tasks each carry a possible deduction of twenty-one points--enough to fail the test--because these are crucial tasks that workers must never perform improperly. If an individual's point total drops below eighty on the assessment, he or she cannot continue the course. For every task, though, instructors have a box to check if the student recognizes his or her mistake just after making it, notifies the instructor, and rectifies it immediately. The point deductions decrease when trainees correct themselves this way, and, in the case of the three crucial tasks, the decrease is dramatic--from twenty-one points to seven or even three points (see Table C.2). The fund developed these weightings with DOE input and approval.

In contrast, the hazardous waste worker course has no such rigorous level of performance evaluation. The Occupational Safety and Health Administration (OSHA) requires workers in hazardous waste removal to be certified by an approved training provider, but it does not produce regulations that specify what the training must include. Laborers-AGC is left to define the requirements for certification, including designing the performance exam in this case. Each item on the performance exam is simply marked as correct or incorrect. Though it would be possible to generate importance weightings for

each task on this course's exams, Laborers-AGC would have to shoulder the costs of researching them and then justify the weight-ings to a federal agency that does not even require a performance test.

The written exams are given at the end of each course and consist of either fifty or 100 questions drawn randomly from a large test bank. Laborers-AGC creates the questions for each test and submits them to the appropriate federal agency for approval. The radiation worker course is an exception, though, in that Laborers-AGC staff must randomly select questions from those developed by the DOE. Once tests are formulated, they are disseminated to the training facilities, where local instructors administer them according to program guidelines.

Results of the written exams are tallied for each individual and later aggregated for whole classes, training schools, and the entire training system. Local training schools need student and class results in order to process worker certificates and to comply with state or local regulations for training providers. Laborers-AGC collects all data to monitor both of these functions and to keep track of program performance trends.

The assessment tasks are tied closely to the instructional objectives. These objectives were developed by the fund's content/industry experts and agency staff to mirror the skills needed in the occupation. The hands-on activities contextualize the classroom information in events that will be found at most, if not all, environmental remediation worksites. These field activities use mock hazard sites and actual tools and equipment to ensure that trainees who pass a performance event can work safely and effectively on real worksites.

The knowledge and skills measured by the assessments are highly specific to the occupation and the specialization area. At present, Laborers-AGC is considering ways to combine courses regulated by different agencies so as to create more comprehensive environmental worker courses. Though such courses might have great potential for workers and employers, the fund finds it very difficult to satisfy all the state and federal regulations simultaneously for each individual work area. However, it combines two training courses regulated by OSHA and the DOE (hazardous waste worker and radiological worker, respectively) into a single 120-hour course for workers at nuclear power plants. In January of 1996, program specialists were preparing for a trial run of the course at the Hanford nuclear facility in Washington state. Any significant alterations to these assessments or the curriculum will occur only after a review of course results and input from the agencies. Laborers-AGC administrators are considering how to combine the EPA-regulated asbestos and lead abatement courses into such a "cluster" course, but considerable work with EPA staff will probably be necessary to do so.

Using the fund's curricula is optional for local training schools, but the fact that Laborers-AGC programs have been independently approved by the regulatory agencies definitely makes their use advantageous. Documenting that they use the fund's federally approved program helps schools satisfy most, if not all, of their state's requirements for providers of training in these specializations.

RELATIONSHIP TO OTHER PROGRAMS

Many union laborers consider the environmental training courses useful for career advancement. Many construction laborers seek environmental certifications after working for several years in the construction field (with varying experiences). Union members generally agree that environmental courses are more technical and have more formal testing procedures than most construction courses, thus requiring greater cognitive abilities. The nature of these courses led to development of a preparatory course for union members who want to bolster their basic reading, math, and science skills before enrolling in an environmental course. The preparatory course lasts forty hours, uses some texts and materials from the certification courses, and is usually offered just before many environmental classes start so that the trainees can quickly apply their sharpened skills.

Although the fund's environmental programs are now equal in number and importance to its construction programs, there is little contact between them. They operate under separate departments and, in general, have separate sources of financial support. The construction programs use mainly local training fund contributions, whereas the environmental programs are supported by federal grants. The environmental courses have money to support activities such as hiring consultants to develop assessments and evaluate programs. The construction programs are less able to do this, but recently the fund's administrators have undertaken an initiative to research and create new performance tests for them. However, until this project is completed, most of the construction programs will continue to use informal instructor observations as the sole means of skill assessment. Environmental and construction programs differ in their level of technicality and certification requirements, and there is less overall consistency among the construction courses. Within the environmental department, though, courses are closely related. Program specialists often cross-train so that they can collaborate on curriculum development and train-the-trainer events. This collaboration helps to increase consistency of training in specialized areas that may be technically dissimilar.

Many other environmental programs also prepare workers for this field, but coordination or cooperation among them is rare and limited. Unions such as the carpenters, teamsters, and operating engineers offer certification programs in the same specializations, as do many private organizations and postsecondary institutions, such as the University of California at Los Angeles. Competition for students is strong. Private training schools and postsecondary institutions compete for students--the former to make profits, the latter to fill enrollment targets. And the unions compete among themselves, each wanting to gain more of the market by placing more of its workers in the courses and then in jobs This latter tension is difficult to resolve because under the NIEHS grant, Laborers-AGC is the primary grant recipient and the teamsters union is a subgrantee. Each is developing independent course curricula, but the fund has additional duties. Laborers-AGC is responsible for all administrative, budgetary, and reporting concerns. Interaction between the two is mainly confined to high-level administrative matters, and staff members do not confer frequently on curricular matters.

Some of the Laborers-AGC courses differ from those offered by other providers in that they require more--sometimes two times more--hours than the regulations mandate. Industry consultants recommended that the fund add time for extensive field exercises and assessment in addition to the time spent on classroom instruction. Not all of these hands-on activities are required by federal agencies, but many have real safety and productivity implications. While the Laborers-AGC hazardous waste worker course is recognized in the industry and agencies for its quality and comprehensiveness, the eighty-hour course time may be a disadvantage. The other unions and the private institutions that offer this worker certification do so in a forty-hour format, which is appealing to those paying for the training: it costs less and responds more quickly to employer requests for qualified workers. Contractors with large cleanup projects do not compare course quality or assessment procedures when trying to meet workforce needs and project deadlines. They simply need certified workers and may call upon another union if it can supply them faster and at lower cost. Because of this pressure, Laborers-AGC is considering designing another version of its field/classroom training in a forty-hour format. Currently, it does not know how this change would affect the assessments.

In addition to the environmental course assessments, trainees in the hazardous waste worker course must successfully pass a physical exam in which a registered nurse tests their pulmonary capacity, heart rate, and blood pressure. The physical exam is given at the outset of the course to provide assurance that each trainee has the physical capacity to perform strenuous training activities (and, later, work) in enclosed suits while wearing respiratory protection. Along with the signed approval of his or her physician, this assessment's results are recorded as part of each person's eligibility for training and subsequent certification. This requirement reduces the legal liability of the fund and the training schools for any incidents that may occur, and screens individuals before they begin the course.

IMPLEMENTATION AND ADMINISTRATION

The traditional model of assessments, multiple-choice final exams, has been used in one or more courses continuously since 1987. As the courses have been developed and come on-line, the assessments have been adapted slightly in order to reflect the standards and certification requirements of each course. Responsibilities for the environmental assessments are divided among staff at Laborers-AGC and staff at the forty training schools that offer at least one environmental course. Program specialists at Laborers-AGC develop and update the tests as well as monitor the quality and consistency of their use at local sites. The fund's director of environmental programs is responsible for overseeing all assessment and other curricular activities. Training-school staff administer the assessments, score them, and report the results to the fund and to the appropriate state and federal regulatory agencies.

The environmental assessments are updated by the fund once a year, or more frequently if significant changes occur in the industry or its regulations. When considering changes to the assessment, the fund relies on the expertise of its specialists and other industry or regulatory experts, as well as input from course instructors. At its annual instructor development program (IDP), the fund holds educational seminars on professional, technical, and life skills topics for the more than 200 instructors. Also at the IDP, the fund holds curriculum update sessions for each environmental course. In these, instructors can discuss issues directly with specialists, which is a way to maintain course integrity at the local level.

TECHNICAL QUALITY

Laborers-AGC has not extensively evaluated the assessment tools used in its environmental programs. In the late 1980s and early 1990s, much of the fund's efforts concentrated on developing and disseminating courses to meet the training demands of employers and the union. One course after another was developed and brought on-line throughout North America. Staffing and time constraints prevented extensive reliability and validity checks during this period, but the fund has more recently started efforts to evaluate and strengthen the technical quality of its environmental assessments.

Laborers-AGC staff, together with technical and assessment experts, began first by reviewing the oldest assessments--those from the hazardous waste worker course. Though fund staff originally developed the tests for this course with the guidance of similar experts, no experts remained involved throughout the development process, which may have contributed to test weaknesses. After lengthy evaluations of the written test, reviewers found items that did not comply with best-practice guidelines for multiple-choice criterion-referenced exams. The fund set out, with assessment specialists, to remedy the problem items by creating a bank of draft test questions that met the guidelines. These questions were then screened by subject specialists for content validity and by assessment specialists for construct validity. The resulting questions were used in pilot-course trials at several local training schools. Work is currently under way to synthesize the collected feedback from course instructors, students, and program specialists so that final changes can be made to these test items. Once reviewed and corrected, these items will be incorporated into the current test bank, and the same process will be applied to written exams for the other environmental courses.

All the performance exams will eventually undergo comprehensive evaluations, but the fund has not yet determined the process for this. Only the hazardous waste worker course's performance tests have undergone a preliminary evaluation. Content and assessment specialists found that test items are strongly correlated with the work performed on actual worksites, though it is clear to the fund that continual changes in technology, materials, equipment, and practices make content validity an ongoing concern. The items tested in the performance exam were found to closely reflect course content (as reflected in curricular materials), but in some instances they did not closely follow what was actually being taught. For example, certain items in the performance test, as in the written exam, are meant to measure a student's ability to integrate situational facts and circumstances so as to arrive at a proper solution or action. In some course-monitoring visits, reviewers found that instructors were not properly teaching the skills needed to do this. The situational facts were covered, but instructors often did not lead students through the synthesis steps of linking background information and circumstances with possible actions and their likely impacts.

Laborers-AGC considers this flaw both programmatic and instructional and is working to strengthen both the assessment skills of its program specialists (who develop the curricula and train instructors) and the instructional skills of its trainers. Fund staff work with each other and assessment consultants to understand how to develop curricular activities for these skills, and they work with small groups of instructors in yearly instructor refresher sessions to ensure that the skills are taught properly. The work that fund specialists and other staff plus consultants do on curricular and testing updates can be further refined and coordinated at the IDP, where the entire cadre of environmental instructors gathers yearly.

The American Council on Education has also evaluated these environmental courses through its program on noncollegiate-sponsored instruction (PONSI), though to a lesser degree. Instruction and subject experts representing PONSI compared each environmental course's content, learning activities, and assessments with those of current college offerings. Each course was given a recommended number of semester-hour college credits. PONSI reevaluates each course every five years, or sooner if course components are changed. This continuing evaluation is another source of maintaining high quality in the environmental assessment system.

Employers have reacted very positively to the quality of certified employees, which is especially notable because employers are mindful of the potential health and safety ramifications of improperly trained workers. Because the fund is a shared venture between labor and management, employers have immediate input channels if their needs are not being met. The construction and remediation contractors are not the only employers who rely on the assessments to accurately measure skill, though. The DOE, for example, has contracted with the fund to train workers at its headquarters and at several nuclear facilities. DOE experts take great interest in this training because the DOE requires that managers and some engineers at nuclear facilities be certified along with facility technicians. Reaction to the skills assessment from all levels of the DOE and from participating employees has been positive, just as it has from industry contractors and union members. When informal pre- and posttraining comparisons were conducted, workers showed improved knowledge, awareness, and overall performance.

Since the first generation of tests, Laborers-AGC has monitored the exam for any form of gender or racial bias and has made changes when necessary. For the most part, questionable test items are discovered either in monitoring visits or at curriculum update sessions at the IDP each year. Though union membership is roughly half female or minority, assessment results are not aggregated by gender or race/ethnicity to allow such comparisons.

CONSEQUENCESAND USE OF ASSESSMENT RESULTS

An obvious consequence of the assessments is a certificate specifying skill achievement and the attendant acceptance into a specialized field. Moreover, as a result of certification, some trainees have greater confidence in their own skills and knowledge, and they gain greater awareness of the potential effects of their actions on the job. The performance assessments in particular give them the ability to monitor their own work performance and the safety conditions affecting them and their coworkers. At their yearly certification refreshers, many trainees have commented that they mentally "test" their performance while working and, as a result, feel safer and more sure of their decisions in the field.

On the negative side, some potential students who doubt their classroom skills (technical reading, listening for comprehension, etc.) have considerable reservations about enrolling in these courses. Word-of-mouth accounts of the written tests' difficulty particularly cause many to fear that even if they successfully participate in all classroom and hands-on activities, they may fail the final exam. Although the failure rate (on the final exam) for the environmental courses is only about 10 percent, more than half of the students enter courses with a substantial fear of failure, which contributes to a fairly high dropout rate before the exam.

An additional result of the assessments is that some instructors see them as yet another of the fund's curricular mandates. Because local schools are independent, some are reluctant to comply with strict rules or to use required curricular components. Laborers-AGC staff suspect that some assessment rules are not followed from time to time (such as orally translating test questions into other languages), but such deviations are likely isolated and rare. The fund's program specialists visit each school every twelve to eighteen months to monitor particular courses for compliance. If a school blatantly disregards program rules, Laborers-AGC can take sanctions, including withdrawing its sponsorship, in which case the school would have to develop and accredit its own course. This would entail researching technical and pedagogical issues, developing the curriculum and materials, and purchasing new equipment and supplies, in addition to gaining state and federal agencies' approval for the program. This process would be extremely time-consuming and costly, so schools have a strong incentive to comply.

The courses cover a great deal of material, and instructors must essentially teach to the test. This is seen by the fund as both beneficial and necessary, because test-focused instruction gives students an acute sense that all items in the comprehensive course are applicable and important. Laborers-AGC staff see test-focused instruction as contributing to the strong correlation between assessments and work performance and to consistency in instructional and assessment practices across schools.

APPLICABILITY TO VOCATIONAL EDUCATION

The Laborers-AGC assessment model is one that may be applied very easily in vocational education settings. That is, a system of performance-based and written tests is not unusual in vocational settings. But two characteristics of the fund's model would be difficult for many vocational programs to match: the high level of industry support and the high level of funding. A key element of the fund's assessment that would not exist in many vocational settings is the strong systemwide partnership between employers and workers and the input that the employers and workers provide (indeed, gaining substantial input from either employers or workers is usually an obstacle). With such regular industry input, Laborers-AGC assessments can test for federally mandated skills as well as those required by employers. The broad industry base that provides this input allows certified workers to gain skill portability and enables training centers to meet many of the common demands local employers voice. Of course, vocational educators may be primarily concerned with the demands of employers in their particular area or state, but as national skill standards are developed, the prospect of a broader, industry-validated assessment may become desirable, and even necessary, in many programs.

Many educators could benefit from an assessment system that can adapt to the different standards and regulations governing occupations. In vocational programs not having a consistent assessment approach among courses, this system could serve as a model for linking common elements and emphasizing them in the courses while still allowing for variation between subjects. Laborers-AGC maintains the work-simulated performance tests and multiple-choice written exam in all courses, even though the nature of the course content and the applicable state and federal regulations may vary considerably.

It is also important to recognize that costs may be a barrier for vocational educators who seek an assessment with the depth and breadth of the hands-on activities in the Laborers-AGC model. Space and equipment requirements for the fund's model are quite high: it is very costly to obtain and prepare areas for practice and performance that closely simulate actual environmental remediation work. The Laborers-AGC assessments rely on intricate and varied field activities to measure how a student will perform on the job. It would not be plausible to conduct the fund's assessments in a small yard or shop bay that must be shared with other classes. Each simulation area is generally dedicated to a narrow range of tasks. For example, the hazardous waste course requires trainees to perform activities in an outdoor field simulating a hazardous substance dump in which they must maneuver to locate and uncover barrels buried in dirt or sunken in small pools of water. Even in the asbestos abatement course, where simulation areas are indoors, the curriculum calls for a dedicated room or properly enclosed structure that allows the simulated asbestos particles to be removed and hauled away, just as would be the case in a true remediation area.

As equally costly as creating the simulated worksites for practice and performance assessments is the use of actual equipment. The courses require enough equipment for all students to use or wear simultaneously. In general, this equipment--respirators, air tanks, and specialized air filtration vacuums--is very costly. Of course, not all vocational courses prepare students for occupations that use expensive tools and equipment, but each prepares them for jobs whose equipment and settings are unique. Without using that equipment in a wider variety of situational applications, as Laborers-AGC does, vocational teachers may not create or maintain such close ties between course content, assessments, and projected job performance.

Once the performance area is established and the equipment purchased, though, the cost of administering the assessments is fairly low. The instructors' time must be paid for (plus a small amount for materials), and more than one instructor must be present for safety and pedagogical reasons if more than five trainees are in certain kinds of protective suits or using certain equipment at one time. The instructors who administer the tests also score them, using criteria or answer sheets provided by Laborers-AGC, so the cost of their labor is the main expense for scoring.

There has been considerable outside interest in Laborers-AGC's environmental programs, though mainly from educators outside the United States. The fund is currently working on implementing some of its curricula with industry training organizations in countries such as Mexico and Russia. Though many environmental problems are common among countries, the new programs and assessments will have to adapt to different government or industry regulations where they exist. Because the assessments developed and used in the United States and Canada were built to accommodate such differences, the fund feels this dissemination effort will progress smoothly. Though generally minor, the adaptations necessary to accommodate differences between U.S. and Canadian regulations will prove valuable as the fund develops these foreign programs.