Is the thematic curriculum doing what it is intended to do? What positive impact does an "all aspects" base have? How might career clusters be refined? These are only a few of the many questions that may be posed about curricula. Unfortunately, educators and others may spend so much time creating and implementing a curriculum that little time is left to assess it. Realistically, many curricula are never adequately assessed. Some of the more common reasons given for not assessing a curriculum include "We just don't have time to do it," "The thematic curriculum is too complex to assess properly," and "We have no idea how to conduct a curriculum assessment." Although these are relevant concerns, they fail to consider why it is important to assess the curriculum.
There are several reasons for assessing the thematic curriculum. They include (1) considering possible areas of curriculum improvement; (2) considering whether the curriculum's benefits are worth the investment; and (3) deciding whether to keep the curriculum as it is, to modify it, or to eliminate it. The first two reasons relate most closely to curriculum improvement concerns. Given the thematic curriculum's broad scope, curriculum developers; teachers; students; parents; business, industry, community representatives; and others may have differing views of what the curriculum is and what it should be. Thus, when considering curriculum improvements and benefits, it is important to be sensitive to this wide range of views. The third reason for assessment involves administrators' and other policymakers' participation in deciding whether the curriculum will remain as is, be revised, or be eliminated. Policy decisions may or may not give consideration to curriculum improvement. Since these three reasons reflect a progression from improvement to possible loss, it is easy to see how assessment fits into the picture. If curriculum assessment is begun early in the development process and shortcomings are identified, changes can be made more easily and the curriculum may not need major revision. However, if assessment is ignored, the curriculum may ultimately need to undergo major change or, worse yet, be eliminated.
In this section, a brief introduction to thematic curriculum assessment is provided, followed by a description of the assessment process. Also included are ways that assessment information can be used for curriculum improvement.
What is assessment and how does it relate to the thematic curriculum? Basically, assessment may be viewed as determining the curriculum's value or worth. Central to assessment is the process of making informed decisions; that is, using assessment information to determine whether the curriculum is of acceptable quality and, if not, what must be done to make it acceptable. One of the more popular ways to view curriculum assessment or evaluation is to consider it as having two stages: formative and summative (Ornstein & Hunkins, 1988).
Formative assessment focuses on the improvement of a curriculum as it is being developed. Emphasis is placed on determining how the curriculum and its component parts may be improved during the development process. For example, results of career cluster curriculum pilot testing at a high school may reveal that objectives are not sequenced properly or that some teachers are having difficulty understanding their respective roles in the team-teaching process. Formative assessment can help ensure these problems are identified and corrected early in the curriculum's development rather than after the curriculum has been fully implemented. Several examples of questions for which answers may be obtained during the formative assessment stage are listed in Figure 8-1.
Figure 8-1. Questions Related to Formative Assessment
- Have the most meaningful contextual options been selected?
- Have the most meaningful organizational options been selected?
- Have the most meaningful delivery options been selected?
- Have the most meaningful content options been selected?
- Have the curriculum and its component parts been offered to students in meaningful and effective ways?
- To what extent is the curriculum accepted and used by teachers?
- To what extent do students accept the curriculum and learn from it?
Summative assessment has the fully operational curriculum as its main focus. This assessment is designed to obtain a holistic view of the implemented thematic curriculum. Emphasis is placed on determining the curriculum's cumulative impact. So, for example, after a career cluster focusing on all aspects of manufacturing has been implemented in a high school, the cluster should be examined to determine how it fares as a whole. Representative summative assessment questions that might be asked are highlighted in Figure 8-2.
Figure 8-2. Questions Related to Summative Assessment
- How well do curriculum components work together to create a high-quality curriculum?
- What impact does the curriculum have on teachers, students, graduates, parents, the community, the workplace, and others?
- To what extent is the curriculum better than what it has replaced?
- What are the curriculum's unintended benefits?
- How can the curriculum be improved to make it even better than it is?
Formative and summative assessment are not meant to be one-shot tasks. They are ongoing activities initiated early in the curriculum development process and conducted on a continuing basis throughout the curriculum's life. Assessment applied at regular intervals is one of the keys to creating and maintaining high-quality curricula.
The Long Road To Achieving Results*
When an environmental sciences academy was established at W. G. Harding High School nine years ago, both the administrators and the faculty members had expectations that it would dramatically decrease the student dropout rate and provide academy graduates with greater education and employment opportunities. These results were not expected to show up during the curriculum's first years of operation; they were clearly long-term expectations. The academy eventually did achieve these results, but only after many years of hard work that included a great deal of curriculum revision. However, some results could be observed after its first year of operation. By the end of the first year, almost every student enrolled in the academy was enthusiastic about the learning experiences and anxious to enroll for the second year. During the first year, faculty members who ran the academy had become accustomed to working as a team and were very excited about curriculum improvements they were planning for the future.*Based on the experiences of several high schools that have chosen to remain anonymous.
The assessment process typically consists of several sequential steps (Finch & Crunkilton, 1993; Levesque, Bradby, & Rossi, 1996; Ornstein & Hunkins, 1988):
Each of these steps is described below:
The assessment process is typically driven by important concerns people have about the curriculum's "impact." Kirkpatrick (1994) describes impact as having four levels: (1) reaction, (2) learning, (3) behavior, and (4) results.
Assessment at Level 1 (reaction) focuses on how persons associated with the curriculum (e.g., students, teachers, employers, community representatives) react to it. Kirkpatrick refers to this as a measure of customer satisfaction. If "customers" are not happy with the curriculum, it may have no future regardless of its quality.
Level 2 (learning) emphasizes "the extent to which participants change attitudes, improve knowledge, and/or increase skill" (p. 22) by participating in the curriculum. This level of assessment focuses primarily on students but could easily include others as well.
Level 3 (behavior) addresses the extent to which change in behavior has occurred among participants. Assessment at this level is directed toward applied behavior change. For some students, this may mean change is assessed in work-based learning settings. For other students, assessment of change may take place in school-based learning settings. Again, this assessment level may apply to others.
At Level 4 (results), a determination is made about the curriculum's overall outcomes. This is at once the most beneficial and the most difficult area to assess. Examples of results that might be assessed include reduced student dropout rates, increased higher education and/or employment options for students, graduates' increased satisfaction with their schooling, and improved communication among teachers. Figure 8-3 includes levels of impact examples for three different thematic curricula.
Figure 8-3. Levels of Curriculum Impact: Abbreviated Examples
Curriculum Focus Assessment Level 1: Reaction Assessment Level 2: Learning Assessment Level 3: Behavior Assessment Level 4: Results Engineering & Technology Cluster Students react favorably to the emphasis on all aspects of a field Students can distinguish among the various careers in the cluster Studentsenroll and succeed in higher-level math courses Graduates more education career available Agricultural Sciences Career Major Teachers feel positive about working together to teach the major Students recognize that there are many opportunities to work in the field Enrollment in the major results in increased science test scores Graduates apply they have learned to workplace Arts and Communication Academy Parents appreciate the broad exposure their children are getting to the field Students can describe aspects of the field's competitive nature Students develp a much greater interest in studying English Graduates able to more career choices
These examples provide a better feeling for the impact a curriculum may have. However, when determining the assessment focus, it is best to state curriculum outcomes as specifically as possible. Otherwise, those who conduct the assessment may have little idea if the curriculum has been successful or unsuccessful. Three examples of the same outcome for students enrolled in a thematic curriculum are presented for review:
Even though these three outcome examples all focus on student achievement, they range from example one which is quite vague to a more specific example two and finally to an even more specific example three. Since the third example provides more specific information about what is expected, it is both easier to assess and to defend. Therefore, it is recommended that curriculum outcomes be prepared so they reflect example three's characteristics.
The assessment information gathered is a function of the level being examined (i.e., reaction, learning, behavior, results) and the expectations people have for the curriculum (i.e., outcomes). To be most effective, methods used to gather information should be compatible with assessment levels and peoples' expectations. Curriculum assessment information may, therefore, be gathered in a variety of ways. Assessment information falls into two distinct yet interrelated categories: (1) quantitative and (2) qualitative. Quantitative information consists of "hard" information such as numbers, scores, means, and standard deviations; whereas, qualitative information includes "soft" information such as interview text and responses to open-ended questions (Brainard, 1996, p. 11). Provided in Figure 8-4 are several examples of ways assessment information may be gathered.
Figure 8-4. Examples of Ways Assessment Information May Be GatheredQuantitative Data Sources
- Surveys (questionnaires, opinion polls)
- Rating scales
- Checklists
- Delphi technique
Qualitative Information Sources
- Structured interviews
- Focus groups
- Nominal group process (Brainard, 1996, p. 27)
If reaction information is needed, it might be obtained via carefully developed questionnaires, personal interviews, or a combination of the two. Information about learning, behavior, and results may be somewhat more difficult to gather. This might require obtaining information about student and/or faculty member change over time, a task that includes taking into account factors that may cause the assessment results to be invalid. Or it might demand that detailed observations are made to gather information about the settings in which learning is taking place.
If assessment information is linked to expectations of what the curriculum should be, the information examination task can be relatively easy. For example, say that after three years of operation, a travel and tourism academy in a high school is expected to effect a 20% lower student dropout rate. If student dropout information has been gathered at certain points in time and potentially confounding factors have been accounted for, it should be relatively easy to see if the curriculum has had impact on the student dropout rate. If it is anticipated that business and management academy graduates at that same high school will be better prepared for both higher education and employment, information about these graduates, and perhaps a comparison group, must be gathered and arrayed in a ways that it can be seen whether the expectation has, in fact, been realized. In sum, assessment information should be gathered and organized so it can be easily compared with expectations.
Discrepancies that exist between what is and what should be (see Figure 8-3) serve as starting points for curriculum improvement. If discrepancies emerge during the formative assessment of a thematic curriculum, they may point to changes that should be made before the curriculum is fully implemented. Shortcomings such as a lack of teacher and student satisfaction with the curriculum, teacher resistance to change, and lack of collaboration from employers may each need to be dealt with in a different way. Deciding which changes to make and making them can result in meaningful curriculum improvements. Assessing the curriculum during its formative development and making necessary adjustments is much easier than completing them after the curriculum is fully operational.
Results of summative assessment can confirm that the curriculum development process has been successful or point out what changes should be made before it can be called a success. Since summative assessment focuses on the entire curriculum, deficiencies identified at this point in time may be difficult to translate into meaningful changes. For instance, if assessment reveals that some students are not learning certain content as rapidly as anticipated, this deficiency may not point directly to what should be changed. Detective work may be needed such as determining which students are not learning the content and why they are not learning it. This, in turn, can assist in discovering what portions of the curriculum should be changed or, perhaps, determining if it is a teaching problem rather than or in addition to a curriculum problem.
One of the more interesting aspects of summative assessment focuses on discovering unintended curriculum outcomes. Who knows what will be uncovered in this area? Such a search may not identify anything or it may reveal the curriculum has a variety of unintended benefits that reflect a major return on the curriculum development investment. For example, an examination of career cluster curricula in a high school might reveal that, even though not identified as an intended outcome, students' career decisionmaking skills have improved.
Although assessment is a very important element of thematic curriculum design and implementation, it is often ignored or delayed until very late in the development process. Curriculum assessment can be conducted in two stages--formative and summative--with formative assessment focusing on improving the curriculum as it is being developed and summative assessment focusing on the total curriculum after it has been developed. Both formative and summative assessments should be planned early in the curriculum development process and continue on a regular basis throughout the curriculum's life.
The assessment process typically includes determining the assessment focus, obtaining assessment information, examining the assessment information, and using the information for curriculum improvement. Comprehensive assessment can focus on one or more impact levels: reaction, learning, behavior, and results. When assessment information is gathered and examined in a comprehensive and systematic fashion, the improvement process is made much easier. Thus, for improvement to be maximally effective, changes should be made during both the formative and summative assessment stages.
Brainard, E. A. (1996). A hands-on guide to school program evaluation. Bloomington, IN: Phi Delta Kappa Foundation.
Finch, C. R., & Crunkilton, J. R. (1993). Curriculum development in vocational and technical education: Planning, content, and implementation (4th ed.). Needham Heights, MA: Allyn & Bacon.
Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler.
Levesque, K., Bradby, D., & Rossi, K. (1996, May). Using data for program improvement: How do we encourage schools to do it? Centerfocus (issue #12).
Ornstein, A. C., & Hunkins, F. P. (1988). Curriculum: Foundations, principles, and issues. Englewood Cliffs, NJ: Prentice-Hall.