Rater Agreement in Washington State’s Evaluation System


Contents

  • Rater Agreement Background
  • Overarching Roles and Responsibilities: District Office, Principals, and Teachers
  • Rater Agreement Definition and Stages
  • Rater Agreement Glossary

Download the Rater Agreement document as a PDF.


Rater Agreement Background

The new Washington State Evaluation System for teachers and principals will require new learning and ongoing support to increase the effectiveness of implementation. The four framework authors would like to underscore the importance of the shared experience between evaluator and evaluatee. To that end, the new rules (WAC 392-191A-050) support this intent:

The purposes of evaluations of certificated classroom teachers, certificated principals, and assistant principals shall be, at a minimum:

  1. To acknowledge the critical importance of teacher and leadership quality and support professional learning as the underpinning of the new evaluation system.
  2. To identify, in consultation with classroom teachers, principals, and assistant principals, particular areas in which the professional performance is distinguished, proficient, basic or unsatisfactory, and particular areas in which the classroom teacher, principal, or assistant principal needs to improve his or her performance.
  3. To assist classroom teachers and certificated principals and assistant principals, who have identified areas needing improvement, in making those improvements.

Overarching Roles and Responsibilities: District Office, Principals, and Teachers

Topic
District Office
Principal
Teacher
Rater Agreement Implementation Responsibilities
  • Ensures fidelity of implementation of the chosen framework

  • Leads the oversight of ongoing evaluator training and capacity-building across a district

  • Moves evaluators toward maximizing rater agreement

  • Provides the structures (e.g., principal PLC time) and resources for teachers and principals to learn, implement, and sustain growth- oriented evaluation
  • Leads the implementation of the growth-oriented teacher evaluation processes.

  • Takes a collaborative learning stance with teachers in the evaluation process.

  • Takes responsibility for learning and sustaining their own rater agreement.

  • Provides the structures and resources for building level staff to learn, implement, and sustain the growth-oriented evaluation system.
  • Engages in the implementation of the growth-oriented evaluation processes.

  • Takes a collaborative learning stance with the principal and colleagues in the evaluation process.

  • Takes responsibility for learning and applying the frameworks and rubrics to continually improve practice.

  • Communication
  • Provides communication and sponsorship for immediate and long-term implementation.
  • Provides communication and feedback to support immediate and long-term implementation.
  • Provides communication and feedback to support immediate and long-term implementation.

  • Rater Agreement Definition and Stages

    The TPEP project is guided by the RCW 28A.405.100 and WAC 392-191A. The law requires that “before school district implementation of the revised evaluation systems,” that evaluators of both teachers and principals “must engage in professional development designed to implement the revised systems and maximize rater agreement” (RCW 28A.405.120 and RCW 28A.405.130).


    Rater Agreement

    The extent to which the scores between the raters have consistency and accuracy against predetermined standards. The predetermined standards are the instructional and leadership frameworks and rubrics that define the basis for summative criterion level scores.

    The research and practical application of implementing this law is of primary concern for the Office of Superintendent of Public Instruction (OSPI), TPEP Steering Committee organizations, and the instructional and leadership framework authors.

    In partnership with the three instructional framework authors and two leadership frameworks, OSPI will use the following working definition and steps to guide the 2014–15 implementation year:



    Rater Agreement Glossary

    ICFFS/LCFFS (Instructional or Leadership Criteria and Framework Feedback Specialist): A team of Washington State practitioners who are trained by framework author designees to provide facilitation around Stage 1 and 2 on the specific instructional or leadership framework chosen by OSPI as the “preferred instructional or leadership frameworks.” These practitioners applied and were selected by a state-level committee through a rigorous process. The ICFFs and LCFFs are not experts in the new evaluation law. They are gaining expertise around the frameworks by the framework authors as the foundational tool for the new evaluation systems for teachers and principals.

    Accuracy: A measure of observer data quality indicating the extent to which an observer is assigning scores that agree with scores assigned to the same observation by an expert rater; the extent to which rater’s scores agree with the true or “correct” score for the performance.

    Consistency: A measure of observer data quality indicating the extent to which an observer is assigning scores that agree with scores assigned to the same observation of practice by another typical observer. Consistency among the untrained is not what we are looking for. A goal of rater agreement is to ensure both accuracy and consistency.

    Calibration: A process by which the regular practice of an observer’s scoring is monitored and verified that the observer is still scoring accurately and consistently according to the standards and definitions of the framework/rubrics.

    Artifact: Observed practice, products, or results of a certificated classroom teacher or certificated principal’s work.

    Evidence: Observed practice, products, or results of a certificated classroom teacher or certificated principal’s work that demonstrates knowledge and skills of the educator with respect to the four-level rating system. (392-191A-030)

    Feedback: Information aligned with a rubric provided to reduce discrepancies between current performance and desired performance. Effective feedback answers three questions:

    • Where am I? (What are the performance goals based on a self-assessment of the rubrics?)
    • Where am I going? (How is my performance related to the rubrics?)
    • Where to next? (What actions do I need to take next to increase my performance?)

    Criterion: The standards for teaching and school leadership as defined by RCW 28A.405.100.

    Instructional framework: One of the preferred instructional frameworks adopted by the Office of Superintendent of Public Instruction to support the new evaluation system pursuant to RCW 28A.405.100.

    The preferred instructional frameworks chosen by OSPI:

    Leadership framework: One of the preferred leadership frameworks adopted by the superintendent of public instruction to support the new evaluation system pursuant to RCW 28A.405.100.
    (392-191A-030)

    The preferred leadership frameworks chosen by OSPI:

    Rubrics: The progression/description of practice used during an observation to capture evidence and classify teaching or leadership practice into differentiated aspects and performance levels. Typically consists of:

    • Several Scales (components, domains, dimensions, indicators – there are numerous terms). See Architecture and Alignment Documents.
    • A set of score levels applied within each scale to classify performance. The score levels are described in Washington State as: Distinguished (4), Proficient (3), Basic (2), Unsatisfactory (1)

    Observe or observation: The gathering of evidence made through classroom or worksite visits for the purpose of examining evidence over time against the instructional or leadership framework rubrics. (392-191A-030)

    Summative Criterion Scoring: Rating given to performance based on Washington State Criterion (see definition above). These scores will be based on an ongoing and varied process using a preponderance of evidence to determine final summative scores that promotes and recognizes growth.

    This process is determined at the district level; guidance is available through the TPEP Training Modules.

    Final Summative Scoring: Aggregation of the summative criterion scores. Final Summative Scoring Methodology approved by OSPI to determine final summative ratings. Raw Score Model

    This process is determined at the state level; guidance written in WAC 392-191A.