Education 3504 Week 3 reliability & validity observation techniques checklists and rubrics

Reliability and Validity validity –extent to which you are measuring what you think you are measuring reliability –extent to which information you are gathering is consistent and unambiguous

Reliability chance factors –characteristics of student –characteristics of test / evaluation –conditions of administration –variations in scoring

Validity danger of thinking you are measuring one thing, actually measuring another –trivial knowledge –lowest-common denominator –regional knowledge/one culture/gender

Validity face validity –appears important, relevant, interesting construct validity –appears to capture intended quality or behaviour - e.g. what does “good citizenship” consist of?

validity and reliability are related but can vary independently

Checklists and Rubrics

checklist construction Checklist must be based on objectives of the course, unit, or lesson each item should be clear and precise each item should avoid generalities and focus on specifics

checklist construction checklist should be designed for feedback to students and parents –so need to be understandable –might want to distribute to students before hand must do so if intended for grading

checklist construction if students can use them, can be used for peer and self- evaluation focus on a few specifics, so keep checklist short –if list needs to be longer, focus on only small section at a time anecdotal summary comments can be tied to checklists

rating scales 5 -Extremely active 4 -somewhat active 3 -average active 2 -somewhat active 1 -extremely active

rating scales Extremely Active (5 points) always out of seat rarely sits without squirming makes noise, yells a good deal of the time

Rubrics scoring schemes

clear expectations Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A ( ) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B ( ) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C ( ) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

…but flexible Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A ( ) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B ( ) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C ( ) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

levels of mastery Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A ( ) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B ( ) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C ( ) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

multi-dimensional Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A ( ) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B ( ) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C ( ) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

parallelism Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A ( ) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B ( ) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C ( ) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

high but attainable standards: cumulative criteria Generic Essay OriginalitySubject content awareness of issues critical thinking EXCELLENT A ( ) A Markedly Exceptional Performance originality, insight, and creativity are demonstrated; the paper goes beyond repeating what others have said and contributes something new to our understanding of the topic a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information SUPERIOR B ( ) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation SATISFACTORY C ( ) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed

allowance for part marks Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A ( ) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B ( ) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C ( ) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

allowance for part marks

correspondence to real world Generic Essay Subject content awareness of issues critical thinking Takes a position EXCELLENT A ( ) A Markedly Exceptional Performance a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion SUPERIOR B ( ) Clearly Above Average Performance a thorough grasp of the subject matter is demonstrated an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied SATISFACTORY C ( ) A Fully Competent Paper a basic grasp of the subject matter is demonstrated asserts viewpoint without acknowledging alternative viewpoints accurate information incorporating relevant sources and references is conveyed a position is adopted and logically argued

evolving changes if you discover papers that do not fit criteria –add descriptors that accurately describe what you are seeing

evolving changes as your teaching changes changes as students change changes as environment changes

ease of use

rubrics workshop Objective: Produce the best cookie possible

rubrics workshop Objective: Produce the best cookie possible Was it difficult to come to a consensus on criteria for “best”?

why we evaluate motivation administration accountability certification

motivation problems: –test anxiety

Norm referenced describes student performance relative to other students –e.g.: ranks 11th in group of 30

Criterion referenced describes student performance compared to a clearly defined criteria or task –e.g.: “adds single-digit whole numbers”

motivation problems: –overemphasis distorts learning process emphasis on external motivation

motivation problems: –overemphasis distorts learning process not everything we teach can be assessed (e.g., non-cognitive skills, attitudes, etc.)

administration general trend towards greater emphasis on assessment general shift from selection to accountability

accountability government to taxpayers –2003, K-12 = 3.8 billion dollars…

accountability teachers –accountable for mandated curriculum –accountable for student learning –to oneself over time

accountability students to their parents

certification right of passage selection and allocation (gatekeeper)