1. STEPHANIE CHENEY EDU 547 DR. ORDU Validity and Reliability in Competency Based Assessment

2. Background + Context Images from: http://collegeforamerica.org/

3. Program Design: Assessment Sub-Group Assessment questions:  What types of knowledge are being assessed? Foundational and/or Applied  What are the means of assessment? Objective, essay, performance based, project based, etc.  How are assessments scored? Rubrics, computer-scored, human-scored, etc.  Why did you choose this type of assessment strategy?

4. Research Question Based on coding transcripts, one area of uncertainty is the validity and reliability of assessments. Research Question: How does a school ensure that competency based assessments are valid and reliable?

5. What are Validity and Reliability? The validity of an assessment tool is the extent to which it measures what it was designed to measure. Four considerations:  Content validity  Construct validity  Criterion-related validity  Consequential validity Reliability refers to the consistency of the results we obtain from an assessment. This may impact:  Consistency across time  Consistency across tasks  Consistency across markers Darr, C. (2005, September 1). A hitchhiker's guide to validity. Retrieved March 21, 2015, from http://toolselector.tki.org.nz/content/download/371/4475/version/2/file/Validity -Hitchhikers Guide.pdf

6. Strategies that Support Validity and Reliability Assessment that covers wide range of content increases validity Tasks that require use of desired skills and reasoning processes improves validity Clear assessment directions supports validity More tasks on an assessment improves reliability The larger the spread of scores produced by the assessment, the higher the reliability Well developed rubrics make it easier to identify achievement areas, increasing reliability Darr, C. (2005, September1). A hitchhiker's guide to reliability. Retrieved March 21, 2015, from http://toolselector.tki.org.nz/content/download/371/4475/version/2/file/Reliability-Hitchhikers Guide.pdf

7. CBE: Sharpening Our Focus on Learning Must determine whether the student can apply what he or she knows to real life programs and situations. Instead, assignments should present tasks or situations that students will encounter in life and in the workplace. Klein-Collins, R. (2013, November 1). Sharpening Our Focus on Learning: The Rise of Competency-Based Approaches to Degree Completion. Retrieved March 5, 2015, from http://www.learningoutcomeassessment.org/documents/Occasional Paper 20.pdf Image from: http://bit.ly/1LKiteg

8. Five-Dimensional Framework for Authentic Assessment Task Physical Context Social Context Assessment Form Criteria + Standards Gulikers, J., Bastiaens, T., & Kirschner, P. (2004). A Five-Dimensional Framework for Authentic Assessment. Educational Technology Research & Development, 52(3), 67-86.

9. Five-Dimensional Framework: Findings Research showed that task, form and criteria were most important for authenticity. Older students and teachers agreed that an authentic task combined workplace knowledge, skills and attitudes and mirror real-life complexity. People with professional practice in the field should be involved in identifying the criteria and standards and evaluating student work. Gulikers, J., Bastiaens, T., & Kirschner, P. (2004). A Five-Dimensional Framework for Authentic Assessment. Educational Technology Research & Development, 52(3), 67-86.

10. Torpey, E. (2011, September 1). You're a What? Psychometrician. Retrieved March 22, 2015, from http://www.bls.gov/careeroutlook/2011/fall/yawhat.pdf Promising Practices Subject Matter Experts Pilot TestAnalyze Results Calibration w/Evaluators Psychometricians on Staff

11. References Bushway, D., & Everhart, D. (2014, December 8). Investing in Quality Competency-Based Education. Retrieved March 5, 2015, from http://www.educause.edu/ero/article/investing-quality-competency-based-educationhttp://www.educause.edu/ero/article/investing-quality-competency-based-education Darr, C. (2005, June 1). A hitchhiker's guide to validity. Retrieved March 21, 2015, from http://toolselector.tki.org.nz/content/download/371/4475/version/2/file/Validity -Hitchhikers Guide.pdf http://toolselector.tki.org.nz/content/download/371/4475/version/2/file/Validity -Hitchhikers Guide.pdf Darr, C. (2005, September1). A hitchhiker's guide to reliability. Retrieved March 21, 2015, from http://toolselector.tki.org.nz/content/download/371/4475/version/2/file/Reliability-Hitchhikers Guide.pdf http://toolselector.tki.org.nz/content/download/371/4475/version/2/file/Reliability-Hitchhikers Guide.pdf Gulikers, J., Bastiaens, T., & Kirschner, P. (2004). A Five-Dimensional Framework for Authentic Assessment. Educational Technology Research & Development, 52(3), 67-86. Jones, E. (2002, September 1). Defining and Assessing Learning: Exploring Competency-Based Initiatives. Retrieved March 7, 2015, from http://nces.ed.gov/pubs2002/2002159.pdfhttp://nces.ed.gov/pubs2002/2002159.pdf Klein-Collins, R. (2013, November 1). Sharpening Our Focus on Learning: The Rise of Competency-Based Approaches to Degree Completion. Retrieved March 5, 2015, from http://www.learningoutcomeassessment.org/documents/Occasional Paper 20.pdfhttp://www.learningoutcomeassessment.org/documents/Occasional Paper 20.pdf Torpey, E. (2011, September 1). You're a What? Psychometrician. Retrieved March 22, 2015, from http://www.bls.gov/careeroutlook/2011/fall/yawhat.pdf http://www.bls.gov/careeroutlook/2011/fall/yawhat.pdf Wijekumar, K., Ferguson, L., & Wagoner, D. (2006). Problems with Assessment Validity and Reliability in Web-Based Distance Learning Environments and Solutions. Journal of Educational Multimedia and Hypermedia, 15(2), 199-215.