Penny Craig, Ruth Jones, & Lisa Pufpaff, Department of Special Education

 Examined inter-rater agreement pre and post rater training ◦ 10 faculty raters ◦ 3 different assessments ◦ 2 exemplars for each assessment ◦ Inter-rater agreement very poor BSU Assessment Forum April 20152

 Online, self-guided rater training with expanded rubrics and narrated PPTs ◦ Little change in inter-rater agreement ◦ Social validity among raters very high  Recently published: ◦ Pufpaff, L. A., Clarke, L., & Jones, R. E. (2015). The effects of rater training on inter-rater agreement. Mid-Western Educational Researcher, 27(2), BSU Assessment Forum April 20153

 Selected one assessment from previous summer  3 faculty and 2 course instructors  Baseline inter-rater agreement on 5 exemplars  Discussed differences and revised rubric  Repeated inter-rater agreement on 5 different exemplars  Discussed differences and revised rubric BSU Assessment Forum April 20154

UnsatisfactoryBasicProficientDistinguished Does not establish the value of the paper topics as regards the legal, theoretical, and historical bases of special education practices. Establishes the relationship of the paper topic(s) to the identification and support of students with ELN. Describes legal foundations and past practices contributing to the establishment of requirements for special education identification and services. Meets Basic requirements and presents the models, theories, philosophies, and research methods that form the basis for special education practice, including FAPE, LRE, the continuum of placement options, and the provision of related services such as specialized health care. Meets Basic and Proficient requirements and describes how students are identified with disabilities and the educational needs they present, including considerations for culturally and linguistically diverse students. Statutory law, eight examples of case law, or corresponding administrative regulations are not presented or are not relevant to topic. Presents current statutory law, a minimum of eight court cases (total), and current requirements as per state regulations relevant to the paper topics Meets Basic requirements and explains relationship of case law to statute(s) and administrative regulations. Meets Basic and Proficient requirements and cites court cases specifically with a summary of plaintiff and defendant positions, judicial ruling, and relevance to topic explained. Clarifies progression of the litigation through the court system. Omits or minimizes the rights, roles, and responsibilities of families in collaboration with schools to provide meaningful access to educational opportunity for students with ELN. Describes the effects of law on student, parents, and school staff and the role of the family with the school, citing a minimum of five scholarly (peer-reviewed) journals. Meets Basic requirements and describes the rights, responsibilities, and outcomes as per topic, including the impact of the dominant culture and potential differences between school and home. Meets Basic and Proficient requirements and provides examples of best practice recommendations for disability area for support to students with ELN, including laws, policies, and ethical principles of behavior management and reinforcement and models and theories of deviance and behavior problems. BSU Assessment Forum April 20155

 Completed this process 5 times before reaching >90% agreement  This process resulted in a “rubric checklist” to be used by both students and instructors BSU Assessment Forum April 20156

UnsatisfactoryBasicProficient (Meets Basic) Distinguished (Meets Basic and Proficient) ________Correctly identifies the legal foundations of IDEA or Indiana’s Article 7. ________Fails to completely or correctly address the six components of IDEA. Correctly identifies the legal foundations of: ________Constitution ________IDEA ________Article 7 ________Correctly lists social and legal history ________Correctly addresses six components of IDEA: ___IEP ___FAPE ___LRE ___Eval process ___Parent involvement ___Procedural safeguards _______Correctly addresses: ___Continuum of placements ___Related services Correctly explains legal foundations from: ________Constitution ________IDEA ________Article 7 ________social and legal history Correctly explains: ________IEP ________FAPE ________LRE ________Eval process ________Parent involvement ________Procedural safeguards ________RtI Synthesizes and interrelates corresponding ideas relative to: ________how SWD are identified ________RtI and traditional model ________specific example of educational needs of a SWD related to law ________needs of culturally and linguistically diverse individuals ________Correctly describes one court case ________Relates the case to incorrect components of IDEA ________Describes at least 2 court cases ________Relates the cases to correct key component(s) of IDEA ________Correctly relates the 2 court cases to specific regulation(s) within Article 7 ________Court cases are correctly explained as they progress through court system: ________plaintiff ________defendant ________final ruling ________progression through the court system ________Correctly discusses relevance of the court cases to both: ________IDEA ________Article 7 ________Fails to correctly address the role of the family in regards to discipline Comment pertaining to all cells in this rubric row: Simple mention of a concept or topic does not meet criteria. Be sure that topic is “Correctly identified…correctly explained and correctly discussed…” ________Correctly identifies and defines the role of the family in regards to: ________discipline ________procedural safeguards ________resolving conflicts ________right of the family to be involved ________Correctly discusses components of discipline including all of the following: ________FBA ________BIP ________Relates the provision of discipline and procedural safeguards to both: ________IDEA ________Article 7 ________Correctly discusses components of discipline including all of the following: ________manifestation det. ________suspension ________expulsion ________Relates discipline and procedural safeguards to a court case or peer-reviewed article BSU Assessment Forum April 20157

 Digital Portfolio is a required assessment of all teacher education candidates at 4 points in their program  Normally assessed by course instructor  SPCE created a Portfolio Assessment Committee made up of faculty BSU Assessment Forum April 20158

 Revised the Unit rubric into a rubric checklist with increased specificity of performance- based verbs and quantifiers at each performance level  Began creating departmental policies and procedures for uniformity in administration and assessment of portfolios across all SPCE majors BSU Assessment Forum April 20159

 Rater training among PAC members  Randomly divided 117 portfolios among three PAC members for evaluation  27 portfolios evaluated by a second rater  Have not done statistical analysis yet BSU Assessment Forum April

 Randomly divided 65 portfolios across 4 PAC members  All 4 members assessed 13 (20%) portfolios  Have not run statistical data yet BSU Assessment Forum April

Evaluator #1 Evaluator #2 Evaluator #3 Evaluator #4 Student A.H. ReflectionDPDD RationalePPPP DesignDPDD Environmen t DPDD MechanicsPDDP Professiona lism DDPD ArtifactDDPD SummaryDPPD BSU Assessment Forum April

Evaluator #1 Evaluator #2 Evaluator #3 Evaluator #4 Student E.M. ReflectionDDDD RationalePPPD DesignDDDD Environmen t DPDD MechanicsPPPD Professional ism DDPD ArtifactDDDD SummaryDDDD BSU Assessment Forum April

Evaluator #1Evaluator #2Evaluator #3Evaluator #4 Student B.T. ReflectionPPPD RationaleDPDD DesignDDDD EnvironmentPDDD MechanicsPPPP Professionali sm PDPD ArtifactDDPD SummaryPPPD BSU Assessment Forum April

Evaluator #1 Evaluator #2 Evaluator #3 Evaluator #4 Student E.S. ReflectionPBPP RationaleBBBB DesignPBBP Environmen t PDPD MechanicsBPDP Professional ism PPPP ArtifactPDPD SummaryPPPP BSU Assessment Forum April

 Decision Point 1   Decision Point 2   Decision Point 3  BSU Assessment Forum April