KEY DECISIONS IN THE PROGRAM EVALUATION
A number of key decisions must be made in any systematic program evaluation. We examine each of these decisions.
What is the Purpose of the Evaluation?
There are two broad purposes of educational evaluation. Formative evaluation is intended to improve a program. It is carried out while the program is in progress and can be ongoing throughout the life of the program. Summative evaluation results in a definitive judgment about the value of the program. It is carried out after a program has been in existence for a period of time. A summative evaluation is usually the basis for a decision about whether the program will continue, undergo major revisions, or terminated. Formative and summative program evaluation are not always mutually exclusive. For instance, data gathered for formative evaluations might be reanalyzed as part of a summative evaluation.
Who Performs the Evaluation?
Whether the supervisor, a team of faculty members, central office personnel, or private consultants should have major control over evaluation. However, it is critical that teachers be involved in evaluation of instructional programs and the overall instructional effectiveness of their schools and districts. All stakeholders (those affected by the decision from the evaluation) should be not only subjects of study but co-investigators of the study as well. As Greene (1986) notes, there is a “consensus on the need for stakeholders participation” (p. 1), and such participation is defined as the need decision making, rather than just advising or providing input” (p. 9).
What Questions Must Be Answered?
Evaluation questions depend on the nature of the program and what members of the evaluation’s audience wish to learn about the program. Let’s say that a new social studies curriculum is to be evaluated. An implementation question might be: To what extent has the new curriculum been implemented at the classroom level? An outcomes question might be: What changes in students’ knowledge, skill, and attitude have resulted from the new curriculum? Once the evaluation question have been formulated, they become the basis for the remainder of the evaluation.
What Data Is Gathered and How?
Data sources are persons, places, things, events or processes from which data needs to answer evaluation question can be gathered. Examples of data sources are students, teachers, principals, parents, teaching episodes, students, and school records, administration of rating scales and surveys, and interviewing. Our bias is towards using multitude sources of data and multiple data-collection methods for each evaluation question. Multiple sources and methods increase the likelihood that the evaluation results will be valid.
How Are the Data Analyzed?
Data analysis is largely determined by the evaluation questions and types of data. Decisions to be made include how to organize, summarize, display data, and how to reach conclusion based on the data. Central office, university, or private experts may be necessary to assist with complex quantitative or qualitative analysis. Stakeholders, however, can make valuable contribution to data analysis, especially by reviewing results and suggesting explanations, implications, and conclusions.
How Is the Evaluation Reported?
After collecting and analyzing the results of tests, observations, survey, interviews, and testimonies, how should the evaluation be reported? The answer is largely determined by the audience. Most school board members and superintendents will not read a 200-page technical report on the raw data, statistical treatments, and evaluation methodologies. They are interested in the results and conclusions. The technical report should be available to decision makers as a reference to the summarized paper. Any reader of the condensed paper who is confused or desires more information about certain parts of the paper can check the complete technical paper.
However, if the audience for the report consist of people with sophisticated evaluation skills, a complete technical report would be in order. In a study of violations pf evaluation standards Newman and Brown, 1987, p.9), among the most frequent violations were “those concerning the evaluator’s lack of knowledge of the audience.”
Regardless of the audience, certain types of information are included in most evaluations reports. (Again, these components will vary in length and technical sophistication depending o the audience.) Typical evaluation reports include discussion of the following:
1. The purpose of the evaluation
2. A description of the program being evaluated
3. Evaluation questions or objectives
4. Methodology, including data sources, data-gathering methods, and data analysis methods
5. Results and conclusions, including strength and weakness of the program
6. Recommendation for the future
TEACHER EVALUATION
Although teacher evaluation involves some of the same skills as program evaluation (e.g., data gathering, data analysis), it is essentially a separate process requiring different strategies and techniques. In this section, we compare summative and formative teacher evaluation, propose that these two types of evaluation be carried out separately, and discuss self-evaluation.
Comparing Summative and Formative Teacher Evaluation
Summative and formative are two broad categories of teacher evaluation. Summative teacher evaluation is an administrative function intended to meet the organizational need for teacher accountability. It involves decisions about the level of a teacher’s performance. Summative evaluation always seeks to determine if the teacher has met minimum expectations. If the teacher has not met his or her professional responsibilities, the summative process documents inadequate performance for the purpose of remediation and , if necessary, termination. Sometimes, summative evaluation also gathers data to determine if a teacher is eligible for reward provided by the district for outstanding performance.
Summative evaluation is based on policies that mandate its purpose, frequency, and procedures, an administrator completes checklist, rating scales, or narratives indicating the extent to which the teacher has met performance criteria. Evaluation forms usually are standard (same criteria for all teachers) and global (general enough to apply to teacher with different responsibilities). evaluation forms judge teachers on the quality of their instruction, including such areas as classroom climate, planning, the teaching act, and classroom management (Danielson and McGreal, 2000). however, evaluation criteria usually include noninstructional areas as well as compliance with school regulation, cooperation with colleagues, and completion of extracurricular assignments.
Evaluation instruments must be valid (accurate) and reliable (consistent). valid instruments are those that include all criteria considered essential for effective performance, exclude criteria considered extraneous to effective performance, and weight-relevant criteria in proportion to their importance (Haefele, 1993; Danielson and McGreal, 2000). reliable instruments include low-inference (requiring a high level of evaluator subjectivity) indicators. For example, “teacher clarity” is an example of a high-inference indicator, and “use examples when explaining” is an example of a low-inference indicator (Hawfele, 1993, p. 25). reliability also requires that administrators be trained properly in the evaluation instrument, so that they become aware of rating errors to avoid and develop a high level interevaluator reliability (agreement with experts and other administrators in their ratings of the same teachers).
In addition to concerns with and reliability of individual evaluators, experts suggest that evaluation be done several different times during the evaluation period rather than rely on a one-shot visit to a teacher’s classroom. Finally, legal and ethical considerations require that when a teacher’s performance has been judged to be inadequate, the teacher be noticed of deficiencies, given an opportunity to respond, provide a remediation plan and support for implementing the plan, and be reevaluated (Sutton, 1989)
Formative teacher evaluation is a supervisory function intended to assist and support teachers in professional growth and improvement of teaching. It is focused on the need of teachers rather than on the organization’s need for accountability. Unlike summative evaluation, which usually considers teacher behavior inside and outside the classroom, formative evaluation is focused only in teaching and learning. Summative evaluation is considered with a summary of performance over specific time period, whereas formative evaluation is ongoing and concerned with continuous improvement. Rather than rely on standard evaluation instrument based on systematic observation, which is limited to a single aspect of classroom process (e.g., questioning techniques, student participation, classroom movement).
Because it is not concerned with standardized, global criteria, formative evaluation can concentrate on particular context and needs of individual teachers. Because its purpose is helping teachers, not judging them, formative evaluation is not concerned with legal issues like due process. Rather, it is concerned with building trust and rapport, developing a collegial relationship between evaluator and teachers, and addressing teacher needs and concerns.
Comparison of summative and formative teacher evaluation
| Summative | Formative |
Function | Administrative | Supervisory |
Purpose | Accountability; Judgment on teacher performance, Employment decisions | Assistance, Professional development; Improvement of teaching |
Scope | Instruction, Compliance with regulations, Extracurricular responsibilities; Personal qualities | Instruction |
Focus | Evaluation form | Any classroom data )observation, artifacts, etc) relevant to the teacher’s instructional concerns and needs |
Duration | Set period (usually one academic year) | Ongoing (aimed at continuous improvement) |
Concerns | Standardization, validity, reliability, due process | Building trust, rapport, collegiality; Understanding context; Understanding and addressing teacher concernsand needs |
Evaluator | Usually an administrator, Final decision by administrator | Administrator, supervisor, self, peers, students, sometimes parents |
Why Summative and Formative Teacher Evaluation Should Be Separate
Most school districts have a single evaluation systems and maintain that their system meets both summative and formative needs. However, when schools attempt to carry out summative and formative evaluation simultaneously, they tend to place primary emphasis on summative goals, and formative evaluation is reduced to secondary status. “Too often schools districts espouse a strong growth-oriented position but the evaluation system constructed does not reflect that stance” (McGreal, 1989, p. 38)
Evaluation systems that purport to combine summative and formative evaluation while relying on rating scales alone are particularly suspects. “A school system that relies solely on periodic evaluation of teachers performance through rating scales may capture data suited for in-system summative purposes but will be handicapped in pursuing formative/developmental objectives” (Allison, 1981, p 15; empasis in original). one reason for this is that summative rating scales are designed to be standardized, global, legally defensible, efficiently completed and processed , and include many noninstructional criteria. This means not only that the ratings have little value for formative evaluation but also that the richest, most meaningful data for formative assessment is precluded (Allison, 1981; Stiggins and Bridgeford, 1984).
It is widely recognized that summative evaluation, although necessary to make employment decisions, does not lead to instructional improvement for most teachers (Stiggins and Bridgeford, 1984). in fact, summative evaluation can actually discourage improvement by promoting “negative feelings about evaluation which, in turn, lead to a lack of participation and a lower likelihood of teachers being willing to alter classroom behavior” (McGreal, 1983, p.303). successful formative evaluation depends on trust and open communication between he teacher and the evaluator. Yet summative evaluation is potentially punitive (Sutton, 1989). the possibility of a bad performance rating is always lurking in the background. It’s no wonder that the two types of evaluation don’t mix!
We are not arguing that summative evaluation should be eliminated in favour of formative evaluation. Both types of evaluation are necessary. Like Popham (1988), we maintain that because they have entirely different purposes, they must be kept separate. With McGreal (1983), we argue that the likelihood that either type of which can be accomplished only by the two systems being kept separate. If separate, can the two systems coexist? Yes, but only if the purpose of each is clearly defined, they perceived by teachers as distinct, and the integrity of each is protected (Allison 1981).
How to Separate Summative and Formative Evaluation
One way to separate the two types of evaluation is to use different evaluators. Teachers could receive summative evaluation from the principal and formative evaluation from an assistance principal for instruction, lead teacher, peer coach, or mentor. There are many different combinations of summative and formative evaluators. The important thing is to make clear to everyone who is responsible for each type of evaluation and to have each evaluator carry out their assessment separately from the others.
Another way to separate summative and formative evaluation relates to the time period when each is carried out. For example, all summative evaluations could be carried out in the fall of each schoolyear, leaving the remainder of the year for formative assessment. When this strategy is used, the same person or persons can perform both types of evaluation. This strategy does not work as well if formative evaluation is carried out in the fall of and summative evaluation takes place throughout the rest of the year, because the teacher involved in formative evaluation during the fall realizes that summative evaluation is looming on the horizon. That knowledge may affect the willingness of the teacher to engage in open and honest communication about his or her need for instructional improvement. It’s better to get the summative evaluation out of the way early in the year, give the teacher his or her “seal of approval.” and then allow the teacher and supervisor to engage in non-judgmental assessment for the remainder of the schoolyear. A long-term variation of the “separate time periods” strategy is to conduct summative evaluation throughout the first year of a multiyear cycle and then forcus on formative evaluation for the next two to three years, returning to a summative year at the beginning of the next three-to-four-year cycle. Should serious problems with a teacher’s performance develop during a formative assessment year, that teacher could be shifted back to a summative evaluation-remediation track until the problem is resolved.
A third way of separating summative and formative evaluation was suggested by Thomas McGreal (1983). Under McGreal’s model, a clear and visible set of minimum performance expectation is developed, including administrative, personal and instructional expectations. Teacher performance regarding these minimal expectations is continuously and informally monitored, but no special procedure or evaluation instruments are established. If a problem occurs with a teacher’s perfomance, the administrator reminds the teacher of minimum expectations. If the problem continues to occur, the administrator issues to the teacher a written notice of the teacher’s deficiency, with a copy in the teacher’s file. If serious violations continue even after the formal notice, the administrator recommends more serious administrative action. Beyond these contingencies, there is no standard summative evaluation process or annual write-up. This takes care of summative evaluation. Most of the time, a focus on teaching, systematic classroom observation, and collecting and analyzing additional classroom data. This additional data could include peer, parent, student, and self-evaluation, as well as student performance and classroom artifact (McGreal, 1983).
Which strategy for separating summative and formative evaluation is the best for a district or school depends on the level of administrative and supervisory expertise, the size of the staff, teacher preference, and available resources. The important thing is that they be kept separate. Doing so means that both summative and formative evaluation are carried out more effectively.
Self-Evaluation
Self evaluation can be an important part of the formative evaluation process for teachers functioning at moderate or high level of development, expertise, and commitment. Self-assessment can take a variety of forms, including any of the following:
l Visit to the classrooms of several expert teachers for the purpose of comparative expert teaching to one’s own teaching and identifying self-improvement goals based on such comparison.
l Video recording one’s teaching across several lessons and then analyzing teaching performance while reviewing the video
l Designing or selecting and analyzing results of surveys or questionnaires administered to students or parents
l Interviewing supervisors, peers, students, or parents about effective teaching and learning or about one’s own instructional performance
l Keeping a journal of teaching experience, problems, and successes, accompanied by critical reflection for the purpose of instructional improvement
l Conducting a comprehensive review of student achievement on traditional tests as well as student projects, presentations, portfolios, social behavior, and so on
l Developing a teaching portfolio for the purpose of self-reflection and analysis
Danielson (1996) recommends a variety of items for possible inclusion in a portfolio, including unit and lesson plans, knowledge of students and resources, video of teaching, example of student work, written reflections of lessons taught, and logs on professional services, growth, and research. Langer (Teaching for Performance, 1996) argues for a more focused approach, in which the teacher’s portfolio documents a process I which the teacher defines a problem, sets an improvement goal, designs a plan to reach the goal, implements the plan, collects data on professional growth, and reflects on results. Both Danielson’s broad approach and Langer’s focused approach go beyond the documentation of teaching accomplishments (the purpose of portfolio used in summative teacher evaluation) by providing teachers with opportunities for self-assessment as the basis for instructional improvement.
It’s important to note that self-evaluation of teaching need not be done in isolation. Videos, surveys or interview results, journals, student achievement data, and teacher portfolio can be analyzed and discussed collaboratively with a supervisor or peers, and in some cases with students or parents. The process is called self-evaluation because the teacher assumes full responsibility for decision making regarding planning and implementing the evaluation as well as the instructional improvement plan that results (Keller and Duffy, 2005)
WAYS OF ASSESSING NEED
The first question for the supervisor is: What do we hope to accomplish with a new curriculum guide? To answer this question, we must collect information about both the past and the present state instruction. The supervisor can use multiple ways of assessing need: (1) eyes and eyes, (2) systematic classroom and school observations, (3) official records, (4) review of teachers and student works products, (5) third-party review, (6) written open-ended survey, (7) check and ranking list surveys, (8) the Delphi technique, and (9) the nominal group technique.
Eyes and Ears
Talk to teachers, administrators, aides, and anyone else who works directly with the task under consideration. In this case, the supervisor should ask the teacher and aides individually and in small groups what they believe are the strengths and weaknesses of the curriculum guide. How is it being used? It it helpful, and in what ways? Where does it break down? When is it not useful?
Systematic Classroom and School Observations
This type of assessment goes beyond informal discussion and observation. It consists of the systematic gathering of quantitative or qualitative observation data. Chapter 13 provides a variety of classroom observation instruments. Such instruments can be used to assist the individual teacher, but also can be used to gather data from many classrooms and identify schoolwide instructional needs. For example, in a scholwide needs assessment one of the authors coordinated, observers used the same observation instruments to gather data on instructional methods used in classroom throughout the school. The observers found that most of the teachers in the school relied on lecture and independent seatwork as their primary instructional methods. It was concluded that there was a need for teachers to develop a wider variety of instructional methods. Observations also can be carried out in common areas of the school.
Official Records
Look at any documents that indicate the current use and effect of the task under consideration. In this case, what do achievement test score show? How about diagnostic tests? Are students mastering skills, or are there certain areas that are considered out of line with others? What about the curriculum guide? When was it revised? What recent knowledge about writing curriculum guides, instructional approaches, and topics are not reflected in the current curriculum?
Review of teacher and student work products
Assessors can review teacher and student work products to assess need. Examples of teacher work products are unit and lesson plans, videos of teaching, and teacher portfolios. Examples of student work products include daily assignments, video of student presentations, student work products include daily assignments, videos of student presentations, student projects, and student portfolio. Samples of work produced by several teachers or students can be reviewed to identify common instructional needs. In some schools, teachers build classroom and school portfolios that can be reviewed to analyze representative teacher and student work products.
Third-Party Review
Having a neutral outside person review the task area can be helpful. The supervisor might contact a university or central office consultant, a graduate doctoral student, or some other person with expertise to do an investigation and write a report. The third-party person should be given a clear description of the task (to look at the strengths and weaknesses of the curriculum guides), and care should be taken not to bias the third-party person’s judgment. The report can then serve as an additional sources of objective knowledge, not tied to any special interest in the forthcoming project.
Written Open-Ended Survey
To document and add to the information already received through eyes and ears and official records, a written survey can be administered. Send out a brief questionnaire that asks teachers, aides, administrators, and parents what they think about the current reading curriculum. Keep the survey brief, and word the questions simply, without educational jargon.
No comments:
Post a Comment