Rater Agreement Graphic and Stages


Screen Shot 2013-08-27 at 11.01.43 AM

rateragreement
Rater Agreement is the extent to which the scores between the raters have consistency and accuracy against predetermined standards. The predetermined standards are the instructional and leadership frameworks and rubrics that define the basis for summative criterion level scores.
Stage 1
ESDs, through the services of the Framework Specialists, will provide a two-day Stage 1 training for all who evaluate classroom teachers. This two-day overview provides an understanding of the “Big Ideas” of the Instructional or Leadership Frameworks and the inter-dependency of the frameworks, rubrics, and state criteria.
Evaluators should know and be able to: Understand the “Big Ideas” of the Instructional or Leadership Frameworks and the inter-dependency of the frameworks, rubrics, and state criteria. Accuracy in scoring depends on this foundational level. This foundational level understanding is the key to future calibration and application. It is recommended that this training occur prior to commencing the evaluation cycle.
Success Indicators Understanding of:
  • The appropriate use of the instructional language and framework vocabulary
  • The alignment of framework to state criteria
  • The interdependence of Dimensions/Indicators (CEL), Domains/Components (Danielson & Marzano)
  • The definition of essential aspects of the framework
  • The possible evidence for aspects of the framework
Districts should be creating plans to: Provide the Stage 1 training to all evaluators before evaluating staff.
Stage 2
ESDs, through the services of the Framework Specialists, will provide 30 hours* of ongoing training on the frameworks for all who evaluate classroom teachers, principals, and assistant principals.
Evaluators should know and be able to: Apply the framework and rubric in a formative process. This includes the capacity to give feedback, provide immediate support for implementing the feedback, and provide long-term professional development. Real life observation must be included in the application of this stage. It is recommended that this training occur during the year-long evaluation cycle.
Success Indicators: Formative Application of:
  • The appropriate use of language of instruction and framework vocabulary
  • The interdependence of dimensions/domains and indicators/components
  • The key ideas in each criteria indicators/components and what evidence would look like/sound like
  • The essential aspects of the framework
  • The evidence for all aspects of the framework
  • Develop expertise to provide formative feedback through professional conversations/coaching on the framework and rubrics to foster teacher growth development and implementation of short-term professional development plans

Summative Application of:

  • Analysis of evidence over time for the purpose of summative scoring
  • Utilizing the criterion scoring document, guiding principles, and scoring matrix
  • Utilize summative scoring methodology from WAC 392-191A. Develop and sustain rater agreement, including summative scoring against pre-determined standard identified in the framework and rubric.
Districts should be creating plans to: Provide Stage 2 training to all evaluators prior to summative scoring in the evaluation cycle. Provide for teachers to be supported in the formative evaluation process through framework training, self-assessment/goal setting, observation cycles, professional conversations, instructional coaching, billing and district professional development, and collaborative groups like professional learning communities or grade level/content teams.
Stage 3
Districts will continuously move evaluators to accuracy and consistency through ongoing calibration conversations involving real-life or video-based observation. Framework Specialists do not provide training for this stage although portions of Stage 2 training do have applicability toward rater agreement. Stage 3 should be integrated into the evaluation of principals (Criterion 5) and district administrators.
Evaluators should know and be able to: Accurately align and assess observation data to the standards in the rubric within the selected framework.
Success Indicators: Summative Application of:

  • Appropriately use language of instruction and framework vocabulary
  • The evidence for all aspects of the framework and ability to determine summative indicator or component level scores using that evidence.
  • The evidence for all aspects of the rubric and ability to determine summative criterion level scores for each state criteria.
Districts should be creating plans to: Provide for the ongoing professional development for all evaluators to demonstrate rater assurance.