At Your Fingertips: Using Data for Program Improvement
Work in Progress at MPR Associates
by Karen Levesque, Denise Bradby, and Kristi RossiMPR Associates, under the auspices of NCRVE, is developing training materials on using data for district, school, and program improvement. The materials are designed to be used by teams of administrators, teachers, and other education stakeholders at the local level. Specifically, the materials focus on using the wide variety of data that are locally available, such as average daily attendance (ADA) rates, transcript data, disciplinary action records, and standardized achievement test scores, as well as developing crucial new data sources.
Although local data can serve many purposes, they are typically used to satisfy administrative requirements, rather than to support program improvement efforts. The At Your Fingertips training materials offer an alternative vision that emphasizes how data can be used to monitor progress toward achieving local education goals.
Who Should Be Interested in the At Your Fingertips Training Materials?
The At Your Fingertips Workbook and Facilitator's Guide will be ideal for (1) state-level administrators who want to encourage local use of data for program improvement purposes, including use of Perkins performance measures and standards, and (2) local educators who want to develop a performance indicator system that is tailored to their information needs and supports continuous improvement efforts. While the examples that are used focus on high schools, elementary and middle schools and postsecondary institutions may also find the training materials useful for improvement efforts.What Is a Performance Indicator System?
Performance indicators are statistics that "indicate" something about the performance or health of a district, school, or program. Indicators describe crucial educational outcomes, practices, and inputs, and typically appear as averages, percents, rates, or simple counts. A performance indicator system not only establishes loose relationships among the outcome, practice, and input statistics, but also enables educators to monitor these statistics on an ongoing basis. Such a system helps to identify strengths and weaknesses and generates discussion about causes and appropriate improvement strategies. Ultimately, a performance indicator system produces evidence about whether strategies are working or not.The At Your Fingertips training materials describe a six-step program improvement process that supports continuous improvement efforts. These six steps are described below.
Step 1: Establish Goals
The At Your Fingertips process is goal driven. Rather than allowing available data to drive improvement efforts, program improvement teams are encouraged first to identify what it is they want to achieve at their site, and then what information they need to determine whether they are achieving their goals. The training materials encourage sites to involve a wide variety of education stakeholders in the process of identifying goals. These stakeholders may include academic and vocational teachers, counselors, school- and district-level administrators, school board members, state education agency staff, parents and students, local employers, and local postsecondary institutions, among others. Generally, all those who have a stake in educational outcomes or who will be responsible for helping to achieve the goals should participate in establishing them.Step 2: Identify Related Outcomes, Practices, and Inputs
At Your Fingertips offers guidelines for developing performance indicator systems that incorporate three basic elements of the schooling process: (1) student outcomes, (2) school practices, and (3) school inputs. Student outcomes describe the ultimate end product of the education system-or what we want students to know, achieve, or be able to do. School practices describe the strategies adopted to achieve those student outcomes, including curriculum, instructional strategies, assessment methods, and supporting structures. Finally, school inputs describe the background affecting both practices and outcomes, and include conditions that are difficult to influence directly, such as student demographics, local economic conditions, facilities, and school funds. Building on Step 1, educators are encouraged to identify the key outcomes, practices, and inputs (OPIs) that are related to their goals. Collecting data on all three elements of the schooling process provides a more powerful analytic tool than collecting data on only one or two elements.Step 3: Identify Data Sources
The next step in the program improvement process involves identifying potential data sources. Educators begin by identifying data sources that are already maintained locally or are available from a district or state agency. Only when existing data sources do not provide sufficient information on OPIs are participants encouraged to consider developing new data sources. Examples of commonly available data sources include enrollment records, transcripts, guidance records, standardized achievement test scores, occupational competency checklists, student follow-up surveys, funds and expenditures reports, and community surveys or needs assessments, among many others. Examples of special data collection efforts that may be developed to supplement existing data sources include surveys and questionnaires, interviews and focus groups, teacher logs and diaries, classroom observations, and alternative assessments.Step 4: Develop Indicators for Outcomes, Practices, and Inputs
Once data sources have been identified, program improvement teams can develop performance indicators for their key OPIs. As previously mentioned, indicators typically take the form of averages, percents, rates, or simple counts. Although indicators are essentially quantitative, they can also be used to describe qualitative aspects of schooling. Examples of performance indicators include high school graduation rate and rate of placement into further education and employment (reflecting student outcomes); percent of teachers integrating academic and vocational education and percent of students participating in work-based learning activities (reflecting school practices); and percent of students from low-income families and per-pupil expenditures (reflecting school inputs). It is generally a sound practice to select multiple indicators for each OPI, since teaching and learning are complex processes and a single indicator will rarely adequately describe a particular concern.Step 5: Examine and Interpret the Data
After developing their performance indicators, program improvement team members gather the relevant data. Then they examine these data to determine whether they are achieving their goals, to identify their strengths and weaknesses, and to discuss the factors contributing to their performance. The At Your Fingertips training materials help team members to understand both what the data say as well as what they mean, and to identify any gaps in information. The materials also provide explanations of some basic statistical concepts to assist team members in analyzing the data.Step 6: Develop Improvement Strategies
After reviewing the data and discussing what they mean, program improvement teams develop specific strategies aimed at improving performance on the indicators and reaching their goals. Teams also set performance targets for each indicator, enabling them to measure their progress in concrete terms. Over time, the performance indicator system produces data that allow teams to assess whether their performance is improving. If it is not improving, then team members discuss why this is so. By collecting data on key outcomes, practices, and inputs, teams have a wealth of information upon which to base their analysis. Perhaps the school's demographics have changed unexpectedly, or certain strategies have not been fully implemented, or perhaps the chosen improvement strategies were ineffective and need to be reevaluated. Ultimately, a performance indicator system supports ongoing improvement efforts and produces evidence about end results.
The At Your Fingertips training materials will be available in early 1997 from NCRVE's Material Distribution Service. If you would like to be notified when the materials become available for purchase, please contact Kristi Rossi, Research Associate, MPR Associates, at (510) 849-4942.