ERIKA HALL CENTER FOR ASSESSMENT PRESENTATION AT THE 2014 NATIONAL CONFERENCE ON STUDENT ASSESSMENT NEW ORLEANS JUNE 25, 2014 The Role of a Theory of Action for Educator Evaluation Systems

Common Design for Educator Evaluation Systems Hall, Theory of Action for EE Systems, NCSA Educator Effectiveness Rating Student Outcomes (Growth or Performance) Growth or Performance on State Summative Assessments Growth or Performance on Other Assessments Teaching Practices Student Perception Surveys Observations Teacher Portfolios or Reflections Professional Development Participation Figure from: Diaz-Bilello & Hall, 2014

Ways in Which Systems Might Vary Hall, Theory of Action for EE Systems, NCSA How measures are operationalized and calculated The manner and precision which the results associated with a measure are represented to support reporting and aggregation The process used to combine measures for the purpose of establishing a component or final effectiveness rating The weights given to different types of measures when defining a component or final effectiveness rating; Rules applied to teachers associated with tested vs. non- tested grades and courses; and The flexibility afforded to districts in terms of how different components of the system are selected and implemented

What is a Theory of Action Hall, Theory of Action for EE Systems, NCSA An empirically and logically stated argument A set of underlying assumptions A testable hypothesis …that outlines how and why a given system, as designed, will support the achievement of specified goals.

Pieces of the TOA Puzzle Goals/Expected Outcomes Intended Uses of System Results Theory as to manner in which the system will bring about desired change (System-Level TOA) Components of the system, including:  the rationale for their selection  the goal(s) they are intended to support  How they are intended to relate to one another Mechanism by which components are intended to provide for specified goals. (Component Level TOA) Inferences/assumptions underlying the system working as intended. (Interpretive Argument). 5 Hall, Theory of Action for EE Systems, NCSA. 2014

Why is it Important to Develop a TOA Supports the development of a common language & shared understanding. Clarifies the rationale for the system design Helps to identify areas of inconsistency/lack of coherence Provides for a document that can be used to “defend” the thoughtfulness and coherence of the system. Supports the specification of a plan for validation that is appropriate for your system. 6 Hall, Theory of Action for EE Systems, NCSA. 2014

Figure from Hall,

Simplified Example Hall, Theory of Action for EE Systems, NCSA Purpose: Administrative – Support administrators in making retention, promotion, or dismissal decisions Goal: Remove all non-effective teachers from the classroom Use of Results : Identify high and low performing teachers Claims to be supported:  Score-Based: Teachers who receive an Overall Rating of “Not Effective” do not display the skills and practices expected  Impact-Based: System will provide administrators with the data to inform decision making

Simplified Example Hall, Theory of Action for EE Systems, NCSA Inferences/Assumptions  Overall Effectiveness Rating is an accurate, reliable measure of expected skills and practices  Administrators are provided with system data in time to support decision making Evidence  Feedback from evaluators regarding the perceived appropriateness of the procedure used to define and establish the overall effectiveness rating  Feedback from administrators regarding the availability of data to support decision making

Additional Benefit of TOA Hall, Theory of Action for EE Systems, NCSA States differ with respect to manner and degree to which there is local control over EE system design. A clearly defined TOA supports states in specifying rules related to the development and evaluation of alternate evaluation systems, or system components.  Supports states in identifying what is/is not eligible for modification  Articulates, by example, the type of documentation necessary to support the development/use of an alternate system.

Flexibility Continuum Relative to TOA Hall, Theory of Action for EE Systems, NCSA

Thank You!!! Hall, Theory of Action for EE Systems, NCSA References: Hall, E. (2014). A Framework to Support the Validation of Educator Evaluation Systems. A paper presented at the National Council on Measurement in Education. Philadelphia, PA. Diaz-Bilello, E. & Hall, E. (2014). Considerations for Establishing Performance Standards for Educator Evaluation Systems. A paper presented at the National Council on Measurement in Education. Philadelphia, PA. Erika Hall