IMPORTANCE OF ON-THE-JOB EXPERIENCES IN DEVELOPING LEADERSHIP CAPABILITIES

MDS-814

Judith J. Lambrecht
Charles R. Hopkins
Jerome Moss, Jr.
University of Minnesota

Curtis R. Finch
Virginia Polytechnic Institute and State University

Research Staff
Eric C. Crane
University of Minnesota

Lex Bruce
Virginia Polytechnic Institute and State University

National Center for Research in Vocational Education
Graduate School of Education
University of California at Berkeley
2030 Addison Street, Suite 500
Berkeley, CA 94720-1674

Supported by
The Office of Vocational and Adult Education
U.S. Department of Education

October 1997

FUNDING INFORMATION

• EXECUTIVE SUMMARY
• INTRODUCTION
  • Need
  • Purpose
• PROCEDURES
  • Sample
  • Developing the Instrumentation
  • Training Interviewers
  • Conducting the Telephone Interviews
  • Coding and Analyzing Data
• RESULTS
  • Characteristics of Experiences Considered
    Most Helpful in Leader Development
  • Leader Qualities Perceived To Be Improved by On-the-Job Experiences
  • Developmental Aspects Associated with
    Types of On-the-Job Experiences
  • Developmental Aspects of On-the-Job Experiences
    Associated with Improved Qualities
  • Improved Qualities Associated with Types of Experience
  • Types of Experiences CVAs
    Recommend for Preparing Future Leaders
  • Perceived Developmental Aspects of Experiences
    Recommended for Future Leaders
  • Summary of Results
• DISCUSSION
• IMPLICATIONS FOR PRACTICE
  • Nature of Desirable On-the-Job Experiences
  • Timing of On-the-Job Experiences
  • Leadership Qualities That Will Be Developed
  • A Word About Non-Job Education-Related Experiences
• REFERENCES
• APPENDIX A
  INTERVIEW WRITE-UP
  
• APPENDIX B:
  INTERVIEW CODING SYSTEM

EXECUTIVE SUMMARY

The purpose of this study was to explore the importance of on-the-job experiences as a means of complementing and supplementing leadership development provided in formal education programs. Not only do on-the-job experiences have potential to assist persons who are actively involved in leadership programs; they may also be of value in reaching and impacting vocational education professionals who have not had an opportunity to attend these programs. This study used the following seven questions to guides its procedures:

1. What are the characteristics of on-the-job experiences that successful vocational education administrator-leaders consider most helpful to their development as leaders? Are there differences that relate to gender?
2. What leader qualities are perceived to be improved by the on-the-job experience?
3. What developmental aspects are associated with various types of on-the-job experiences? (What is it about the experiences that make them effective?)
4. What developmental aspects of on-the-job experiences are associated with improved leader qualities?
5. What improved leader qualities are associated with various types of experiences?
6. What types of experiences do vocational education administrator-leaders recommend for preparing future leaders?
7. What are the perceived developmental aspects of the experiences recommended for future leaders? (What is perceived to make the experiences effective?)

This study builds directly on research findings from the corporate world that indicate how on-the-job experiences relate to leadership development. This research also responds to the need to determine the ways in which these findings apply to education in general and vocational education professionals in particular. And, finally, this study links closely to and builds directly on over six years of leadership research and development conducted by the National Center for Research in Vocational Education (NCRVE). It was the next logical step in NCRVE's long-term research and development program.

From among the 220 chief vocational administrators who participated in the collection of normative and standard data for the Leader Attributes Inventory (LAI) and the Leader Effectiveness Inventory (LEI), the 78 with the highest scores (top one-third) on the LEI were identified and asked to participate in a study to examine the importance of on-the-job experiences in the development of leadership capabilities. Sixty-nine vocational education administrator-leaders (26 women and 46 men) from 12 states participated in the telephone interview process. The Behavioral Event Interview (BEI) technique was chosen as the data collection method because of its ability to focus on meaningful dynamic behaviors demonstrated by leaders that they judge have had an impact on their development as leaders.

The Interview Protocol asked each vocational administrator-leader to describe two on-the-job incidents that had the most impact on their professional development as leaders. Specific probes were then used: (1) Can you give a brief overview of the incident (that had the most impact on the development of your leadership qualities)? (2) Can you briefly describe what made this on-the-job incident developmental? (3) When in your career did the incident take place? (4) Who or what initiated the incident? (5) How did the incident unfold? (6) In what ways did your leadership qualities improve or develop as a result of the incident? (7) Are there other things I should know about the context of this incident? (8) If I wanted to provide a similar on-the-job developmental experience for someone else, what else would I need to know?

The intent of these probes was to gain as much information as possible about the incident and to obtain specific examples to capture what happened and what the administrator was thinking and feeling during the experience. The ultimate goal was to gain an understanding of how this experience was developmental and how its developmental effects might be replicated on the job for others. A total of 140 incidents were described by the vocational education administrator-leaders.

Five types of experiences were identified by successful vocational education administrator-leaders as most helpful to their development as leaders: (1) new positions that offer new and/or increased responsibilities; (2) special start-up work assignments; (3) handling personnel problems like conflicts and firings; (4) being mentored, counseled, supported; and (5) working with a supervisor. These experiences were perceived to provide the following kinds of opportunities for leadership development: (1) the challenge of new and/or complex tasks or problems; (2) the chance to learn new ideas, practices, insights; (3) the opportunity to apply and practice skills and knowledge; (4) encouragement and confidence building; and (5) exposure to positive role models. The kinds of leadership development opportunities identified were most frequently perceived to result in the development of the following leadership qualities: (1) communication (listening, oral, written) skills; (2) administrative/management knowledge and skills; (3) unspecified interpersonal skills; (4) team building skills; (5) sensitivity, respect; (6) confidence, self-acceptance; (7) a broader perspective about the organization; and (8) the appropriate use of leadership styles.

Men tended more than women to initiate their own challenging experiences, and were more frequently motivated by the risk of failure and/or by the interest and excitement generated by the experience. On the other hand, challenging experiences for women came more often from new and complex tasks, where they had the support of superiors and/or other positive role models, or from job stress and barriers they perceived to accomplishing the tasks. Further, more men than women used their on-the-job experiences to improve their team building, motivational, and use of leadership styles qualities, while more women than men felt they improved their insightful, networking, and organizational skills. With two exceptions, men and women agreed upon the qualities most frequently perceived to have been improved by on-the-job experiences. Women included networking and organizing rather than team building and using appropriate leadership styles among the seven most frequently perceived improved qualities. It is also noteworthy that a higher proportion of women than men reported gains in insightfulness, while a higher proportion of men than women indicated the quality of motivating others to have been developed.

The five most frequently recommended types of experiences for future leaders were (1) mentoring, counseling and advocate support; (2) formal training programs (e.g., leadership academy); (3) internships; (4) various special assignments (while on-the-job); and (5) simulations/case studies. Missing from this list were three of the most frequently mentioned types of on-the-job experiences that respondents had reported were effective in developing their own leadership abilities: (1) providing new and/or increased responsibilities, (2) special start-up assignments, and (3) handling personnel problems like conflicts and firings. Men and women differed somewhat in their recommendations for the types of experiences that future leaders should have. A higher proportion of women than men favored mentoring, counseling and advocate support as well as formal training programs and simulations/case studies. A greater proportion of men than women favored internships.

Based on the results of this study, on-the-job experiences can certainly be promoted as one effective, and perhaps indispensable, means for developing future leaders. Successful leaders participating in this study all had vivid positive memories of experiences which they said significantly effected their development as leaders. Further, the successful leaders advocated on-the-job activities that they believed could be used effectively in developing future leaders for vocational education. The findings of this study are consistent with studies in business and industry which also report that certain kinds of on-the-job experiences are effective for developing leaders. Thus, it is important that current vocational education administrator-leaders take advantage of the opportunities they have for using on-the-job experiences to develop and improve the leadership capabilities of persons on their staffs who are preparing to assume new and more advanced leadership roles.

Not all on-the-job experiences are equal in their potential effectiveness for leadership development. Two characteristics of effective experiences have been consistently revealed by this and other research. On-the-job learning is most likely to occur for both men and women when

1. Individuals are placed in a variety of challenging situations with problems to solve and choices to make under conditions of risk. These situations (1) motivate individuals to learn, (2) provide opportunities to gain new ideas and knowledge and to practice skills and apply knowledge, and (3) encourage new insights through reflection on prior actions.
2. Individuals gain their experiences in a supportive environment with supervisors who provide positive role models and constructive support and mentors who provide counsel.

Some examples of challenging situations include the provision of new or increased responsibilities; special start-up assignments such as initiating a new program or project; and handling personnel problems such as hiring and firing.

Given that men were more likely than women to be the initiators of their developmental experiences, vocational education administrator-leaders may need to be more aggressive in identifying and providing appropriate on-the-job developmental opportunities for women preparing for leadership roles.

The most important kinds of outcomes from on-the-job experiences for both men and women appear to be growth in personal and interpersonal leadership skills, knowledge, and values. These outcomes most typically include improvement in communication (listening, speaking, writing) skills, sensitivity to and respect for others, team building skills, appropriate use of leadership styles, self-confidence, networking, planning, organizing, and decisionmaking. Additionally, it is common for on-the-job experiences to further develop administrative/management knowledge and skills specific to the context, as well as to broaden one's perspective about the organization.

Vocational education administrator-leaders participating in this study were not asked to identify examples of formal education program-related experiences through which they developed their leadership qualities. Thus, relatively few (9%) reported formal training programs (e.g., leadership academies) and the use of simulations/case studies as significant leadership development experiences in their own development. Yet, about 30% of them recommended the use of formal preparation programs for future leaders. Formal preparation programs should not be considered as a substitute for appropriately challenging on-the-job experiences, but only as a very useful supplement to them.

It is safe to conclude that when using on-the-job assignments for leader development purposes, the key is to provide multiple opportunities to assume responsibility for challenging assignments and to reflect on the meaning of these events for accomplishing important common purposes within given communities of practice.

INTRODUCTION

Yukl (1994) noted in his comprehensive book on leadership in organizations that many of the skills learned by corporate managers are based on experience rather than formal education. He went on to say that "Managers are more likely to learn relevant leadership skills and values if they are exposed to a variety of developmental experiences on the job, with appropriate coaching and mentoring by superiors and peers" (p. 456). Others (Bray, Campbell, & Grant, 1974; Hall, 1976; MacKinnon, 1975) agree that much if not most managerial-leadership development takes place on-the-job rather than in seminars or classrooms, and that early leadership experiences are among the key influences on career development (Margerison & Kakabadse, 1984).

Some, such as V. A. Howard (1995), have developed arguments for why leadership cannot be taught. Howard has argued that many of the propositional (knowledge about leadership) and procedural (knowledge about leading) components of leadership are teachable--at least available for discussion and reflection. However, he sees leadership as more like creativity, virtue, and musicianship than solely a cognitive accomplishment. That is, leadership is as much learned as it is taught. The learning is dependent "upon the interpretive abilities and practical opportunities of the learner given everything that has been taught" (p. 119). He would suggest that the instances when leadership can be identified as having occurred, and thus been "successful leadership" cannot be attributed to the behavior of a single individual. These successes are contingent on too many contextual factors. Thus, the cycle seems to return to emphasis on experience. In Howard's words,

Now leadership is very similar to musicianship in that regard [its teachability]. While it cannot be taught directly, it can be learned depending upon the interpretive ability and practical opportunities of the learner given everything that has been taught. (p. 119)

Need

Until recently, the specific characteristics and impact of effective on-the-job experiences have been largely unexplored (Kotter, 1988; McCauley, 1986). Beginning in the 1980s, researchers at the Center for Creative Leadership (CCL) conducted a series of studies directed toward on-the-job learning. One of these studies (McCall, Lombardo, & Morrison, 1988) focused on interviews with nearly 200 senior business executives to determine experiences that had the greatest impact on their careers and what had been learned from them. Findings of this study were used to create an instrument capable of assessing developmental opportunities in diverse management jobs. The instrument's manual and trainer's guide (Ohlott, McCauley, & Ruderman, 1993) states that the fundamental conclusion of the McCall et al. (1988) study and the basic premise of the instrument are that

on-the-job learning is most likely to occur when managers are faced with challenging situations. Challenging situations place the manager in dynamic settings full of problems to solve and choices to make under conditions of risk and uncertainty. These situations are developmental for two reasons. First, they provide a motivation for learning . . . Second, challenging situations provide the opportunity to take action and to learn from the effectiveness of the action. (p. 1)

The McCall et al. (1988) study results also revealed that what was learned from challenging on-the-job experiences was not technical in nature, but consisted of leadership attributes such as handling relationships, temperament, basic values, and personal awareness. These results were confirmed by Valerio (1990) in her study of New York Telephone Company managers.

The results of several CCL studies support the notion that persons' on-the-job experiences can have a positive impact on managers' leadership development. McCall (1988) prepared a report that summarized a series of studies focusing on successful, high-potential executives. Sixteen developmental experiences were described, as well as the elements that made them developmental, and what executives stated they learned from the experiences. A report written by Lombardo and Eichinger (1989) documented five broad categories of experience that research has shown executives indicate are potentially developmental: challenging jobs, other people (mostly bosses), hardships, coursework, and off-the-job experiences. The authors then detailed 88 developmental experiences that executives can use to help them have a greater variety of leadership challenges and to assist them in learning from these challenges. In a companion report, Eichinger & Lombardo (1990) recommended 22 ways that staff managers could develop leadership skills and, thus, close the leadership gap between staff jobs and careers.

Thus, research conducted in business settings has shown that the timing and type of on-the-job experiences (e.g., use of mentors, cross-organizational experience opportunities) are relevant variables in the study of on-the-job leadership development. Further, the extent and nature of the challenge an experience presents to an individual are of significance in its developmental effectiveness. Researchers have also identified a number of ways that on-the-job experiences can be used to improve and enhance executives' and staff managers' leadership capabilities. Unfortunately, little is known about the ways on-the-job experiences contribute to public educators' leadership capabilities. Research related to leadership development in public education settings has traditionally tended to focus on the study of formal leadership programs and their impact on participants with little consideration given to how leadership may be developed on-the-job (Griffiths, Stout, & Forsyth, 1988).

Purpose

The purpose of this study was to explore the importance of on-the-job experiences as a means of complementing and supplementing leadership development provided in formal education programs. Not only do on-the-job experiences have potential to assist persons who are actively involved in leadership programs; they may also be of value in reaching and impacting vocational education professionals who have not had an opportunity to attend these programs. This study used the following seven questions to guide its procedures:

1. What are the characteristics of on-the-job experiences that successful vocational education administrator-leaders consider most helpful to their development as leaders? Are there differences that relate to gender?
2. What leader qualities are perceived to be improved by the on-the-job experience?
3. What developmental aspects are associated with various types of on-the-job experiences? (What is it about the experiences that make them effective?)
4. What developmental aspects of on-the-job experiences are associated with improved leader qualities?
5. What improved leader qualities are associated with various types of experiences?
6. What types of experiences do vocational education administrator-leaders recommend for preparing future leaders?
7. What are the perceived developmental aspects of the experiences recommended for future leaders? (What is perceived to make the experiences effective?)

This study builds directly on research findings from the corporate world that indicate how on-the-job experiences relate to leadership development. It also responds to the need to determine the ways in which these findings apply to education in general and vocational education professionals in particular. And, finally, this study links closely to and builds directly on over six years of leadership research and development conducted by the National Center for Research in Vocational Education (NCRVE).

NCRVE's research activities related to leadership focused initially on establishing a definition and an explanatory model for leadership (Moss & Liang, 1990) and verifying the model (Finch, Gregson, & Faulkner, 1991; Moss, Finch, & Johansen, 1991; Moss & Jensrud, 1995; Moss & Liang, 1990). Additional research and development efforts have included the stimulation, conduct, and evaluation of leadership programs for graduate students and inservice practitioners (Moss, Jensrud, & Johansen, 1992); creation and field testing of leadership case studies and an organizational simulation designed for use in leadership development programs for professionals (Finch, 1992; Finch, Reneau, Faulkner, Gregson, Hernández-Gantes, & Linkous, 1992); development of a Leader Attributes Inventory (LAI) (Moss, Lambrecht, Jensrud, & Finch, 1994b) and a Leader Effectiveness Index (LEI) (Moss, Lambrecht, Jensrud, & Finch, 1994a); and preparation of a comprehensive, transportable leadership development program targeted at underrepresented groups in the vocational education profession (Moss, Schwartz, & Jensrud, 1994). This study is the next logical step in NCRVE's long-term research and development program.

PROCEDURES

This study included in its research procedures (1) selecting the sample of chief vocational administrators to be interviewed, (2) developing the instrumentation, (3) training the interviewers, (4) conducting the interviews, and (5) coding and analyzing the data gathered. Each of these areas is described below.

Sample

The sample of chief vocational administrators who participated in this study was drawn from the larger sample of vocational educators who were part of the 1993 study to collect normative data and standards for the LAI (Moss et al., 1994b) and the LEI (Moss et al., 1994a). The selection of the original sample will be described, followed by a description of how the smaller sample of chief vocational administrators was identified.

National Sample of Vocational Educators

In order to collect national normative data on the LAI and the LEI, vocational educators with three different roles (each with an expectation that effective leadership should be provided) were used to form three purposive samples:

1. Chief Vocational Administrators (CVA). These were the chief line administrators in (1) specialized public secondary vocational institutions (e.g., principals, directors); and (2) both specialized and comprehensive public postsecondary institutions (e.g., presidents, directors, deans).
2. Vocational Department Heads (VDH). These were administrators/managers of clusters of vocational programs (e.g., department heads and coordinators in specialized public secondary vocational institutions, and both specialized and comprehensive public postsecondary institutions).
3. Vocational Teacher Leaders (VTL). These were professionals in non-administrative/management positions (e.g., teachers and counselors) who were viewed by their chief administrator and/or department head as particularly influential among their peers. Examples are teachers who held elective positions in faculty associations, professional organizations, or unions.

With the advice of consultants, a group of 12 states was selected from which the three samples were drawn. These states were deemed to have well-developed secondary and/or postsecondary vocational systems with relatively high proportions of minority teachers and administrators. Table 1 presents the total population of CVAs in each of the 12 states by type of institution. It is from this population that the sample of CVAs was selected for participation in this study.

All of the 329 CVAs in the 12 states were contacted by letter and then by telephone. The study and their role was explained, and their agreement to participate was solicited; 311 CVAs agreed to participate. Whether or not the CVAs agreed to participate, they were also asked to nominate three VDHs including (where possible) at least one member of a minority group and one female; and up to three VTLs, giving consideration to minorities and females whenever possible.

VDHs were then called and their participation in the study solicited. Minority VDHs were contacted whenever they had been nominated by CVAs. When no minority member had been nominated, or if the nominee declined to take part in the study, an attempt was made to randomly select an equal number of men and women to contact for the VDH sample. Two hundred eighty nine VDHs consented to take part in the study. During the interviews, VDHs were asked to provide the names of up to three VTLs, giving consideration to including minorities and females whenever possible.

Finally, VTLs were called and invited to be a part of the study. Minority members who had been nominated by either the CVA or VDH at a given institution were called first. If no minority member had been nominated, or if she or he refused to participate, individuals who had been named by both the CVA and the VDH were called. In lieu of joint nominees, an attempt was then made to randomly telephone an equal number of men and women. A total of 305 VTLs agreed to participate in the study.

Collecting the Normative Data

Each of the 905 persons who consented to take part in the norming and standards study was sent a packet of materials containing the following pieces:

• a cover letter explaining what they as ratees were being asked to do
• a form to collect demographic information about the ratee (participant)
• a form for the ratee to name the five persons who were to complete the LAI and the LEI as observer-raters
• five copies of the LAI rating-by-observer forms
• five copies of the LEI
• five copies of a cover letter
• envelopes (return addressed and postage paid) for completed forms to be sent back directly to the researchers

Directions to the participants stipulated that the LAI and LEI were to be given to five persons who "(a) Report to you either directly or indirectly (or in the event that you do not have five subordinates, they may be peers); (b) know you well at work; and (c) who, as far as possible, include females and persons from minority groups."

One follow-up was conducted with individuals who had agreed over the telephone to participate in the study, but who either had not returned completed forms containing demographic information, or who had fewer than three observer-raters return completed LAI and LEI instruments. (A minimum of three raters was considered essential for reliable ratings.) Most of the follow-ups were conducted by telephone; the remainder were sent letter reminders.

All of the completed LAIs were electronically screened for eligibility and then scored. To be eligible, the respondent had to report that she or he (1) knew the participant/ratee "very well" or "fairly well" (not "casually" or "not at all"); and (2) was a subordinate or peer (not a superior) of the ratee. The responses of ineligible raters were eliminated and if this reduced the number of eligible raters below three, the ratee was dropped from the sample. Table 2 summarizes the numbers of participants by each of the three samples at key stages of the data collection process.

The required number of completed LAI forms (at least three) were received from 77% of the individuals who had agreed over the telephone to participate in the study. After screening the completed LAI forms for eligibility, 61% of those who had agreed to participate remained in the three samples.

Data Collection Stage	Chief Vocational Administrators	Vocational Department Heads	Vocational Teacher Leaders	Total
1. Total Number in the 12 Selected States	329	-	-	-
2. Agreed To Participate in the Study	311	289	305	905
3. Returned a Sufficient Number of Responses	260	221	212	693
4. Returned a Sufficient Number of Eligible Responses	220	168	163	551
5. Eligible Responses as a Percent of Those Agreeing To Participate	71%	58%	53%	61%

Selection of On-the-Job Interviewee Sample

From among the 220 CVAs who participated in the collection of normative and standard data for the LAI and the LEI, the 78 with the highest scores on the LEI were identified; that is, those who scored in the top one-third of their norm group. In a letter, the 78 CVAs were told their actual scores on the LEI, that they were in the top one-third of their norm group, and asked if they would be willing to be part of the current study to examine on-the-job experiences as a means for developing leadership capabilities. This letter was followed up by a phone call to determine if the CVA would be willing to participate in a telephone interview which was estimated to take approximately 30-45 minutes.

When CVAs agreed to participate in the study, they were told that they would be asked to describe two incidents that had the greatest impact on the development of their leadership capabilities. Since each person needed to have time to think about these two incidents, the interview was not carried out immediately. Rather, a time for the actual interview was scheduled for a later day, generally within the next two weeks. The initial phone call was followed up by a fax to confirm the interview date and time and restating the request for two critical incidents. A total of 69 CVAs agreed to participate in the telephone interviews and to relate at least two on-the-job incidents that affected their development as leaders. Table 3 presents the number of subjects by gender and type of employer.

Developing the Instrumentation

The interview technique was selected as the primary approach to collect field data because of its ability to assist in interpreting the significance of particular variables (Borg & Gall, 1989; Richardson, Dohrenwend, & Klein, 1965). It was chosen over other techniques and "objective" instruments because of its flexibility and adaptability (Kerlinger, 1986). The interview this study used can, in part, be labeled as moderately structured (Stewart & Cash, 1985). This is because the interview contained open-ended and probing questions that followed each major question. The probes were a mixture of open-ended and closed-ended questions. This funneling technique allowed the freedom to probe into and to adapt to different interviewee answers like the nonstructured interview does, but it also provided a schedule for the interview. As with the structured interview, this technique can be replicated fairly easily; it produces data that can be analyzed and compared, and it does not require the use of a highly trained interviewer. In effect, the moderately structured interview was selected based on a decision to utilize the strength and not the weaknesses of both the structured and nonstructured interviews (Stewart & Cash, 1985).

Though Kerlinger (1986) and Borg and Gall (1989) describe the face-to-face interview as "perhaps the most powerful and useful tool of social scientific research" (Kerlinger, 1986, p. 379), it is also costly, time consuming, and is subject to compromises made to improve the social context of the interview (Frey, 1983). Consequently, this study utilized the telephone to minimize the cost and time.

A key element of the interview schedule was a Behavioral Event Interview (BEI). The BEI was developed by David McClelland (1978) and colleagues at McBer and Company. It is based on the Critical Incident Technique that was created by Flanagan (1954). Flanagan had job incumbents write behavioral descriptions of critical incidents they experienced in their work. However, a problem with this technique was that the written incidents tended to not be detailed enough to determine what the job incumbent was thinking, feeling, and specifically doing. As a result, the BEI was developed so that a particular critical incident could be explored until behaviors, thoughts, and feelings were adequately reported (McClelland, 1978). Though BEI respondents may initially only discuss behaviors they believe are critical, additional probing can reveal other relevant behavior that occurred in the event. Klemp (1979) maintained that through the use of extensive probing, the interviewer can elicit descriptions of behaviors that were actually performed in the event, rather than more selective recollections of behaviors.

The BEI served to rectify some of the weaknesses that have been identified in studies of leadership (Boyatzis, 1982; Campbell, Dunnette, Lawler, & Weick, 1970; Murphy, 1988). The BEI has had a very successful history of use in a variety of settings, including business, industry, education, and the military (see Goleman, 1981; Huff, Lake, & Schaalman, 1982; Schmidt, Finch, & Faulkner, 1992; Spencer, 1979). Campbell et al. (1970) describe the Critical Incident Technique (upon which the BEI is based) as one of the most effective methods for assessing managerial behavior. In summary, the BEI was chosen because of its ability to focus on meaningful dynamic behaviors demonstrated by leaders that they judge have had an impact on their development as leaders.

Each CVA was asked to describe two on-the-job incidents that had the most impact on their professional development as leaders. The specific probes to be used by the interviewers were developed through group discussion by the research team as a whole. The intent of these probes was to gain as much description as possible about the incident and to obtain specific examples to capture what happened and what the administrator was thinking and feeling during the experience. The ultimate goal was to gain an understanding of how this experience was developmental and how its developmental effects might be replicated for others.

The eight-part Interview Protocol used the BEI technique. The following were the initial probes:

1. Can you give a brief overview of the incident (that had the most impact on the development of your leadership qualities)?
2. Can you briefly describe what made this on-the-job incident developmental?
3. When in your career did the incident take place?
4. Who or what initiated the incident?
5. How did the incident unfold?
6. In what ways did your leadership qualities improve or develop as a result of the incident?
7. Are there other things I should know about the context of this incident?
8. If I wanted to provide a similar on-the-job developmental experience for someone else, what else would I need to know?

Training Interviewers

The BEI requires that interviewers have a common understanding of the questions to be asked, the manner in which rapport can be established with an interviewee, and effective use of follow-up questions for probing answers. The research team sought the assistance of a specialist in the BEI process who shared instructional materials about how such interviews should be conducted. The entire research team met at a single site to role-play interviews with administrator-volunteers. These interviews were observed and taped for self-critique and critique by the entire group. When the team judged that a common level of interviewing expertise was evident among all researchers, the individual interviews with CVAs were scheduled.

Conducting the Telephone Interviews

Interviews were completed with 69 CVAs in 12 states. These telephone interviews were recorded on tape (with the CVA's permission) for later transcription and coding. A total of 140 incidents were described by the CVAs. Table 4 presents the number of on-the-job incidents reported by gender and type of employer.

Interviewers found that the original time estimate of 30-45 minutes per interview was generally accurate, though some interviews extended for longer periods of time. The interviews were transcribed in their entirety as they were completed.

Coding and Analyzing Data

This section describes the procedures used to code and analyze information collected from interviews. It should be noted that analysis was not a separate phase of the study to be implemented following completion of interviews. As has been done in previous studies (Finch et al., 1991; Lofland, 1971), the researchers chose to interview and analyze concurrently. Miles and Huberman (1984) also advocate analysis during data collection, stating that it allows interviewers to improve their craft. They maintain that such activity enables the researchers to "cycle back and forth between thinking about the existing data and generating strategies for collecting new--often better quality data" (p. 49). Through the use of this concurrent scheme, researchers were able to grow through experience and formulate a meaningful thematic structure for the final text from the collection of critical events.

As each interview was conducted, the interviewer completed a "write-up" for each incident. The "write-up" was prepared in the language of the interviewee, thus reading in the first person. It read like a story telling what actually happened. The purpose of the "write-up" was to organize and present the interview transcript and note-taking information in a more easily understandable sequence and format. Even though each interview was recorded and transcripts of the interview tapes were prepared, the "write-up" provided researchers with meaningful information that had been carefully organized to better facilitate coding and analysis (Mentkowski, O'Brien, McEachern, & Fowler, 1982). Information contained in the "write-up" was organized into sections which paralleled the Interview Protocol described in the earlier section on "Developing the Instrumentation."

A sample of an interview "write-up" is provided in Appendix A. The interview "write-ups" were checked against the original tapes of the interviews by another member of the research team in order to assure their accuracy and completeness.

After a few interviews had been completed, the research team met to discuss the coding scheme. Based on the initial Interview Protocol, actual experience from the interviews, the interview "write-ups," and using the conceptual structure of the LAI (Moss et al., 1994b), a coding sheet was prepared that integrated the potential attributes developed by the on-the-job incidents with the Interview Protocol. This coding scheme was to become the formal guide for interview analysis. As Boyatzis (1982) has noted, "the coding system attempts to explain how an interview should be assessed to determine the presence or absence, or degree of presence, of a particular characteristic" (p. 51). A coding system must, therefore, be clearly delineated so that various researchers agree as to how information will be coded.

While the initial Interview Protocol and the conceptual structure of the LAI were the starting points for the code list, other items were added so that all aspects of the interviews could be placed into a logical category. The decision to add new categories as needed aligns with the perception of Lofland and Lofland (1984) and Miles and Huberman (1984), who advocate modifying coding schemes during data collection and analysis instead of forcing data into predetermined coding schemes.

In addition to basic demographic coding for administrator code, interview code, gender, and type of employer, the following broad categories were coded for each interview:

1. Type of Experience, such as varied jobs/positions, new positions, within a position, or non-job activities.
2. Developmental Aspects of the Experience, such as motivating aspects and facilitating aspects.
3. Timing of Experience, including the judgment of whether the timing was critical.
4. Initiator of the Experience, namely, the administrator or another.
5. Qualities Improved or Developed, based on the attributes included in the LAI.
6. Type of Planned Experience To Be Provided for Others, such as on-the-job experiences in varied jobs or positions, new positions, or within a position, or non-job activities that may or may not be education-related.
7. Important Aspects of Planned Experience, such as motivating aspects or facilitating aspects.
8. Macro or Overarching Insights that could not be coded elsewhere.

The complete Interview Coding System sheet is provided in Appendix B. This sheet contains the finer detail of the outline under each of the elements listed above. The researchers agreed that a response should be coded in each of the eight categories, and that more than one response may be appropriate for some of the categories. However, a particular code was used only once per interview "write-up." In other words, while the interviewee may have provided several descriptions within an incident that would support the use of a particular code, the code was applied to that interview only once.

The researcher who conducted an interview did the first coding of that interview from the prepared "write-up." After the first coding of an interview, each interview was then coded a second time from the original "write-up" by a different member of the research team. This second person had also coded his or her own interviews prior to completing the seconding of a different set of interviews. Thus, each interview was coded twice by persons familiar with the interviewing, "write-up," and coding process. Both persons who had coded a set of interviews then met to discuss each interview. Where codes differed, the rationale for the coding was discussed until consensus was reached.

The coded interviews were entered into The Ethnograph V4.0 (Seidel, Friese, & Leonard, 1995) software package in order to obtain summary data about the frequency of each type of code and listings of the passages for each code type. Crosstabulations were carried out between major code categories to examine relationships between types of incidents and the leadership qualities which were developed. These summaries and crosstabulations are presented in the "Results" section.

RESULTS

As discussed previously, 69 individuals (26 females and 43 males) described 140 incidents they believed helped them to develop their leadership qualities. Participants were asked to identify the characteristics of the incidents they considered most helpful to their development as leaders, the qualities they perceived improved as a result of the experience, and the types of experiences they would recommend for preparing future leaders. The information provided in these incident interviews is discussed in the sections that follow. Included within blocked boarders are illustrative quotations from interviewees that help explain and support the results.

Characteristics of Experiences Considered
Most Helpful in Leader Development

CVAs were asked to provide detail about the 140 incidents and resultant experiences they reported. This included such information as whether the experience was job or non-job related, whether it occurred in a new position or within an existing position, whether the timing of the experience was important to their career development, what the developmental aspects of the experience were, and who initiated the experience.

*The percentages do not total 100% because of rounding.

1. Type of Experience
   1. As can be seen in Table 5, 185 types of experiences were reported in the 140 incidents as most helpful to the respondents' development as leaders. Sixty-four percent occurred on-the-job as respondents worked within a position. The greatest number of these effective on-the-job experiences resulted from challenges inherent in special assignments to start-up new tasks/projects and from handling personnel problems. Receiving support and counseling from mentors and working with a supervisor were also considered important characteristics of effective on-the-job experiences.

2. Of the 185 types of experience, just 10% occurred as the result of respondents holding a variety of positions, mainly in business and industry.
3. Thirteen percent of the 185 types of experiences came from movement to a new position, principally as a result of new and increased responsibilities.
4. Only 9% of the 185 types of experiences took place in a non-job, education-related activity, like a formal training program, while 3% occurred in a variety of other noneducation-related situations.
5. There were few meaningful differences between men and women in the types of their experiences considered most helpful; women reported a somewhat greater proportion of experiences as a result of moving to a new position and education-related experiences.

2. Timing of the Incident
   1. Timing, that is, when the incident came during their careers, was considered significant in 30% of the incidents reported by respondents. See Table 6.
   2. There appeared to be no meaningful difference between male and female respondents on this factor.

3. Developmental Aspects of the Experience (What makes them effective?)
   1. As shown in Table 7, 385 developmental aspects of the 185 types of leadership development experiences were reported. Developmental aspects of the leadership development experiences were identified as being either motivational or facilitating. Motivational aspects were those factors that cause a person to act or perform in a certain way. Facilitating aspects were those factors in the experience that made it easier to perform the task or activity. Thirty-three percent of the developmental aspects were identified motivational, while 66% were identified as facilitating.

2. Of the 127 motivating aspects, the most frequently reported (63%) were the challenges associated with a new or complex task or problem and the risk of failure. Table 8 shows the differences in experiences reported by men and women. Men appeared motivated more by risk of failure than women, while women reported a higher proportion of challenging experiences with a new or complex task or problem. It is also noteworthy that a higher proportion of women than men reported job overload/stress and barriers as significant developmental aspects, while more men found the experiences to be interesting, enjoyable, and exciting.

3. Among the facilitating aspects, the most frequently reported developmental aspects (40%) were the opportunity to learn new ideas and practices and to apply and practice knowledge and skills. (See Table 7.) Exposure to positive role models and the encouragement and confidence gained from the experience were also frequently reported by both men and women (22%), although by a slightly higher proportion of women than men. (See Table 8.) These findings are consistent with the types of experience respondents considered most effective as reported earlier. A slightly higher proportion of women than men also reported that an experience that was negative or resulted in failure was also developmental.

4. Initiator of the Experience

   Participants were asked to identify the person who initiated the experience being discussed--that is, self, superior, peer, subordinate, and other.

   1. The most common initiators for the sample as a whole were self (37%) and superior (30%). (See Table 9.)
   2. Men, however, depended most upon themselves (45%) to initiate the experience, while women depended most upon superiors (44%).

Leader Qualities Perceived To Be Improved by On-the-Job Experiences

Participants were asked to identify the leadership qualities perceived as improved as a result of the experiences. Table 10 shows the responses to this question both as a total of all respondents and by gender.

1. The seven qualities perceived to have been improved by the greatest number of total respondents as a result of their on-the-job experiences were (1) communication (36%); (2) sensitivity, respect (29%); (3) administrative/management knowledge and skills (27%); (4) unspecified "interpersonal" skills and orientation to people (27%); (5) team building (24%); (6) appropriate use of leadership styles (24%); and (7) broader perspective about the organization (24%).

2. The greatest number of men respondents and the greatest number of women respondents agreed upon the above seven qualities with two exceptions. As the exceptions, women included networking and organizing among their seven most frequently perceived improved qualities while men included team building and using appropriate leadership styles.
3. Although not among the seven most frequently named qualities to have been improved, it should be noted that a higher proportion of women than men reported gains in insightfulness (reflecting on the relationship among events and grasping the meaning of issues quickly) while a higher proportion of men than women indicated motivating others as having been developed.

Developmental Aspects Associated with
Types of On-the-Job Experiences

There appeared to be an empirical relationship as well as a logical one between the types of experience most frequently considered helpful to respondents and the developmental aspects of the experience that made them helpful. Empirically, the crosstabulation shown in Table 11 revealed that the five most frequently named types of experience tended to be most highly related to the seven most frequently named developmental aspects. The associations were also logical because they seemed to reflect what one might reasonably expect. For example:

1. The experience of new and/or increased responsibilities was most frequently related to (1) the challenge of new and/or complex tasks or problems; (2) the chance to learn new ideas, practices, insights; and (3) the opportunity to apply and practice skills and knowledge.
2. The experience of a special on-the-job start-up assignment was most frequently associated with (1) the challenge of new and/or complex tasks or problems; (2) the chance to learn new ideas, practices, insights; (3) the opportunity to apply and practice skills and knowledge; and (4) the opportunity to plan and reflect.
3. The experience of handling personnel problems like conflicts and firings was most frequently associated with (1) the challenges of new and/or complex tasks or problems; and (2) the opportunity to apply and practice skills and knowledge.
4. The experience of being mentored, counseled, or supported was most frequently associated with a positive role model and a chance to learn new ideas, practices, insights.
5. The experience of working with a supervisor was most frequently associated with (1) negative experience/failure; (2) encouraging/confidence building; (3) exposure to a positive role model; and (4) exposure to a negative role model.

Developmental Aspect
Item No.	Item No.	36	37	38	39	40	41	42	43	44	45	46	47	48	49	50	51	52	53	Total
	Experience	Inter	Risk	New	Ove	Barr	Oth	Neg	Plan	Asmt	Lear	App	Con	Pos	Neg	Free	Cult	Oth	No Rel
8	Across educ. inst./org.	1	0	0	0	0	0	1	0	1	1	0	1	0	0	0	1	0	0	6
9	Across educ. functions	0	0	9	0	0	0	0	0	0	2	0	0	0	0	1	0	1	0	13
10	Work in business/industry	1	1	3	0	2	0	2	2	0	7	3	1	0	0	1	1	0	0	24
11	Work in government	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	1
12	Other	0	1	2	1	1	0	1	0	0	1	0	1	0	1	1	0	0	0	10
13	Change in context	1	1	3	1	0	0	0	2	1	3	2	2	2	0	2	0	0	0	20
14	New/increased respons.	2	2	14	2	4	0	3	4	1	9	8	4	2	0	3	1	0	0	59
15	Other	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
16	Internship	0	0	2	0	1	0	0	0	0	2	2	0	0	0	1	1	0	0	9
17	Spec. asgmnt: start-up	6	8	17	1	4	0	3	9	3	11	11	8	3	0	8	5	1	0	98
18	Spec. asgmnt: fix-its	1	2	4	1	0	0	0	1	1	2	3	0	1	0	0	0	0	1	17
19	Spec. asgmnt: other	3	0	3	0	0	0	0	1	2	5	4	2	1	1	0	3	0	0	25
20	Teaching/counseling	0	1	1	0	0	0	0	0	0	2	1	0	1	1	0	0	0	0	7
21	Personnel: hiring	0	0	1	1	0	0	0	1	0	1	2	1	1	0	1	0	0	0	9
22	Personnel: conflict/firing	0	7	8	5	5	0	6	4	3	5	8	5	0	3	2	0	0	0	61
23	Personnel: developing	0	1	2	1	2	0	0	0	0	1	3	1	0	1	0	0	0	0	12
24	Work on teams/committees	2	1	3	0	0	0	1	2	2	4	5	1	3	0	1	1	0	0	26
25	Mentors/couns./advocate	1	1	5	1	1	0	0	2	5	8	4	3	11	0	1	2	0	0	45
26	Work with supervisor	0	2	3	2	3	0	5	1	2	3	3	5	5	6	1	0	0	0	41
27	Other	4	1	4	0	1	0	0	1	1	1	2	0	1	1	0	1	2	1	21
28	Prof. association/union	1	0	0	0	1	0	0	0	0	2	0	1	0	0	0	0	0	0	5
29	Formal training	0	0	3	1	1	0	0	1	3	8	4	2	2	0	0	2	0	0	27
30	Simulation/case study	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
31	Shadow/observation	0	0	0	0	0	0	0	1	0	1	0	0	1	0	0	0	0	0	3
32	Networking	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
33	Other	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
34	Community service	0	1	1	0	1	0	0	0	0	2	3	1	0	0	0	1	0	0	10
35	Other	1	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	2
	Total	24	30	88	17	27	0	22	32	25	82	69	39	34	14	23	19	4	2	551

Developmental Aspects of On-the-Job Experiences
Associated with Improved Qualities

As can be seen in Table 12, there also appears to be an empirical and a logical relationship between the aspects of experiences most frequently considered developmental (those aspects that made the experience effective) and the leader qualities that were most frequently perceived as having been improved. For example:

1. The developmental aspect of new and/or increased responsibilities was most frequently related to the improved leader qualities of (1) communication (listening, oral, written); (2) sensitivity, respect; (3) team building; (4) confident, accepting of self; (5) appropriate use of leadership styles; (6) administrative/management knowledge and skills; (7) broader perspective about the organization; (8) unspecified "interpersonal" skills; (9) organizing; and (10) planning.
2. The developmental aspect of a chance to learn new ideas, practices, insights was most frequently associated with the improved leader qualities of (1) communication (listening, oral, written); (2) unspecified "interpersonal" skills; (3) appropriate use of leadership styles; (4) administration/management knowledge and skills; (5) broader perspective about the organization; (6) sensitivity, respect; (7) networking; (8) planning; (9) team building; and (10) confident, accepting of self.
3. The developmental aspect of opportunity to apply and practice skills and knowledge was most frequently related to the improved leader qualities of (1) communication (listening, oral, written); (2) unspecified "interpersonal" skills; (3) team building; (4) administrative/management knowledge and skills; and (5) sensitivity, respect.
4. The developmental aspect of encouraging, confidence building was most frequently related to the improved leader qualities of (1) confident, accepting of self; and (2) communication (listening, oral, written).
5. The developmental aspect of exposure to positive role models was most frequently related to the improved leader qualities of (1) administrative/management knowledge and skills; (2) sensitivity; (3) unspecified "interpersonal" skills; (4) broader perspective about the organization; (5) team building; and (6) communication (listening, oral, written).

Improved Qualities Associated with Types of Experience

The prior two sections have reported empirical and logical relationships between types of on-the-job experiences and the developmental aspects associated with those experiences, and between the developmental aspects of the on-the-job experiences and the leader qualities improved. It is not unexpected, therefore, to find empirical and logical relationships between the types of experience most frequently considered helpful to respondents and the qualities that were improved/developed by the experience. As shown in Table 13,

1. The on-the-job experience of new and/or increased responsibilities was most frequently related to the leader qualities of (1) communication (listening, oral, written); (2) administrative/management knowledge and skills; (3) broader perspective about the organization; and (4) unspecified interpersonal skills.
2. The experience of a special start-up assignment was most frequently associated with the improved leader qualities of (1) visionary; (2) communication (listening, oral, written); (3) team building; (4) administrative/management knowledge and skills; and (5) unspecified interpersonal skills.
3. The experience of handling personnel problems like conflicts and firing were most frequently related to the improved leader qualities of (1) communication (listening, oral, written) and to (2) sensitivity, respect.
4. The experience of being mentored, counseled, or supported was most frequently associated with the improved leader qualities of (1) confident/accepting of self; (2) communication (listening, oral, written); (3) sensitivity, respect; (4) unspecified interpersonal skills; (5) administrative/management knowledge and skills; and (6) broader perspective about the organization.
5. The experience of working with a supervisor was most frequently related to the improved leader qualities of (1) sensitivity, respect; and (2) appropriate use of leadership styles.