1 Issues in Assessment in Higher Education: Science Higher Education Forum on Scientific Competencies Medellin-Colombia Nov 2-4, 2005 Dr Hans Wagemaker INTERNATIONAL ASSOCIATION FOR THE EVALUATION OF EDUCATIONAL ACHIEVEMENT

Theme: Validity in Assessment What can/should be Assessed. What can/should be Assessed. How can it be Assessed How can it be Assessed Frameworks, Curriculum and Trajectories Frameworks, Curriculum and Trajectories From TIMSS to Higher Education From TIMSS to Higher Education

Purpose of Assessment: Assessment for Learning/Assessment of Learning Validity/Reliability- Primacy of Validity Validity/Reliability- Primacy of Validity Construct Validity Construct Validity Consequential Validity Consequential Validity

TIMSS what is Assessed: Content Domains Grade 4 – Life, Physical and Earth Science Grade 4 – Life, Physical and Earth Science Grade 8- Biology, Chemistry, Physics, Earth Science TIMSS Design Grade 8- Biology, Chemistry, Physics, Earth Science TIMSS Design

What is Assessed: Cognitive Assessment Domains Knowing Knowing Applying Applying Reasoning Reasoning Scientific Inquiry Scientific Inquiry Implies different strategies Implies different strategies

Knowing 1. Recall/Recognize- statements, facts, relationships, processes, concepts. 1. Recall/Recognize- statements, facts, relationships, processes, concepts. 2. Define- provide or identify definitions of scientific terms, vocabulary, symbols, abbreviations, units, scales in appropriate contexts. 2. Define- provide or identify definitions of scientific terms, vocabulary, symbols, abbreviations, units, scales in appropriate contexts. 3. Describe- organisms, physical materials, scientific processes that demonstrate knowledge of properties, structure and relationships. 3. Describe- organisms, physical materials, scientific processes that demonstrate knowledge of properties, structure and relationships.

Knowing 4. Illustrate with Examples – support or clarify statements of facts or concepts with appropriate examples- provide specific examples 4. Illustrate with Examples – support or clarify statements of facts or concepts with appropriate examples- provide specific examples 5. Use tools and procedures – demonstrate knowledge of the use of science apparatus, equipment, tools, procedures, measurement devices. 5. Use tools and procedures – demonstrate knowledge of the use of science apparatus, equipment, tools, procedures, measurement devices.

Applying 1. Compare contrast classify- identify or describe similarities/differences between groups of organisms, materials, processes –distinguish, classify or order objects, materials or organisms and processes based on given characteristics and properties. 1. Compare contrast classify- identify or describe similarities/differences between groups of organisms, materials, processes –distinguish, classify or order objects, materials or organisms and processes based on given characteristics and properties.

Applying 2. Use Models – use a diagram or model to demonstrate science concept, structure, relationship, process, biological or physical system or cycle ( e.g.. Atomic structure, electrical circuit, water cycle)

Applying 3. Relate – relate knowledge of an underlying biological or physical concept to an observed or inferred property, behavior, or use of objects, organisms or materials 3. Relate – relate knowledge of an underlying biological or physical concept to an observed or inferred property, behavior, or use of objects, organisms or materials 4. Interpret information - interpret textual, tabular or graphical information in light of a scientific concept or principle 4. Interpret information - interpret textual, tabular or graphical information in light of a scientific concept or principle

Applying 5. Find solutions – use a science relationship, equation, or formula to find a qualitative or quantitative solution involving the direct application/demonstration of a concept 5. Find solutions – use a science relationship, equation, or formula to find a qualitative or quantitative solution involving the direct application/demonstration of a concept 6. provide and explanation for an observation demonstrating understanding of the underlying science concept, principle, law or theory. 6. provide and explanation for an observation demonstrating understanding of the underlying science concept, principle, law or theory.

Reasoning 1. Analyze problems to determine the relevant relationships, concepts and problem solving steps – develop problem solving strategies 1. Analyze problems to determine the relevant relationships, concepts and problem solving steps – develop problem solving strategies 2. Integrate/Synthesize – provide solutions to problems that require consideration of a number of different factors or related concepts – integrate mathematical concepts 2. Integrate/Synthesize – provide solutions to problems that require consideration of a number of different factors or related concepts – integrate mathematical concepts

Reasoning 3. Hypothesize/predict – combine science knowledge with information from observation/experience or formulate questions that can be answered by investigation – formulate testable hypotheses – make predictions about effects of changes in biological or physical conditions in light of evidence and scientific understanding 3. Hypothesize/predict – combine science knowledge with information from observation/experience or formulate questions that can be answered by investigation – formulate testable hypotheses – make predictions about effects of changes in biological or physical conditions in light of evidence and scientific understanding

Reasoning 4. Design Plan investigations appropriate for answering questions 4. Design Plan investigations appropriate for answering questions 5. Draw conclusions – detect patterns in data trends – interpolate 5. Draw conclusions – detect patterns in data trends – interpolate 6. Generalize – make general conclusions that go beyond experimental or given conditions 6. Generalize – make general conclusions that go beyond experimental or given conditions 7. Evaluate – weigh advantages/disadvantages about alternative processes, materials, alternative explanations etc. 7. Evaluate – weigh advantages/disadvantages about alternative processes, materials, alternative explanations etc.

Reasoning Justify – use evidence and scientific understanding to justify explanations and problem solutions… Justify – use evidence and scientific understanding to justify explanations and problem solutions…

Scientific Inquiry 1. Demonstrate a formalized approach to scientific investigation 1. Demonstrate a formalized approach to scientific investigation

Core Principles of Effective Assessment 1. assessment that guides and encourages to effective approaches to learning 1. assessment that guides and encourages to effective approaches to learning 2. assessment that validly and reliably measures expected learning outcomes, in particular higher-order learning that characterizes higher education 2. assessment that validly and reliably measures expected learning outcomes, in particular higher-order learning that characterizes higher education 3. assessment and grading that defines and protects academic standards 3. assessment and grading that defines and protects academic standards

Assessment for Learning- Formative Validity of Assessment for learning is high when assessment procedures help the student toward achieving the real goals of the learning activity Validity of Assessment for learning is high when assessment procedures help the student toward achieving the real goals of the learning activity Requires, committed participants, trust, understanding of what is being asked, effective feedback Requires, committed participants, trust, understanding of what is being asked, effective feedback

Assessment for Learning – Formative Assessment Task Factors – Knowledge of key aspects and difficulties of task Task Factors – Knowledge of key aspects and difficulties of task Criteria – students told ( and understand) clearly what is needed. Criteria – students told ( and understand) clearly what is needed. Standards – set standards are appropriate to students Standards – set standards are appropriate to students Feedback – balance of attention – both strengths and weaknesses- is useful for student Feedback – balance of attention – both strengths and weaknesses- is useful for student

Assessment of Learning – Summative Assessment Done under conditions that give confidence it is students own work. Done under conditions that give confidence it is students own work. Marking is standardized across several of many teachers Marking is standardized across several of many teachers Most up to date information possible Most up to date information possible

Assessment of learning Summative Assessment Quality ( and appropriateness) should take precedence over quantity Quality ( and appropriateness) should take precedence over quantity Things that can be assessed in final exam should be - those that can not should not. Things that can be assessed in final exam should be - those that can not should not. If multiple assessments of same material – final grade should reflect final level of performance If multiple assessments of same material – final grade should reflect final level of performance

Assessment of Learning - Summative Assessment Disadvantage – time sampling problem Disadvantage – time sampling problem May not be valid for admitting student to next stage of education May not be valid for admitting student to next stage of education

Indicators of affective assessments 1. assessment which is integral component of teaching and learning process 1. assessment which is integral component of teaching and learning process 2. multiple roles of assessment recognized. 2. multiple roles of assessment recognized. 3. departmental/faculty policy as guide 3. departmental/faculty policy as guide 4. alignment between what is taught and what is assessed. 4. alignment between what is taught and what is assessed.

Indicators.. 5. Assessment tasks assess capacity to analyze and synthesize new information. 6. Variety of assessments used 7. Assessment designed to assess generic skills as well as subject specific skills

Indicators 8. steady progression of complexity and demands in later years 9. student choice (paper/exam/timing) 10. workloads considered 11. not excessive assessment 12. task weighted to balance formative and summative roles

Indicators 13. Grades calculated on basis of clearly articulated learning outcomes. 14. Students provided with explanatory feedback 15. Tasks checked for bias 16. Plagiarism minimized

Assessment formats Multiple choice Multiple choice Extended Response Extended Response Individual Project Individual Project Thesis Thesis Laboratory work Laboratory work Group work Group work Computer based Computer based Presentations Presentations Many others: oral, portfolio Many others: oral, portfolio