Educational Development Centre May 2015 Assessment Planning: Why,When,What? Maristela Petrovic-Dzerdz, Instructional Designer Including Q&A Session with Dr. Anthony Marini, Senior Teaching Development Associate Educational Development Centre May 2015 Educational Development Centre May 2015

Maristela Petrovic-Dzerdz, MEng, BEd, PBCID Instructional Designer Assessment Planning: Why,When,What? Educational Development Centre May 2015 Educational Development Centre May 2015 Maristela Petrovic-Dzerdz, MEng, BEd, PBCID Instructional Designer

This presentation is based on: Smith P. L. & Ragan T. J This presentation is based on: Smith P.L. & Ragan T. J. (2005) Instructional Design (3rd ed.). Hoboken, NJ: Wiley Jossey-Bass Education.

Post-workshop Bring in 2-3 assessment items you use in your course (example of an essay question, multiple choice question, problem item, etc.) including the description of the assignment, for a more detailed analysis and group discussion.

Agenda Why to assess? When to design assessment? What to consider when designing assessment? When to assess? What to assess?

Case study: Training on creating effective visuals

Case study: Learning Outcomes Given a description of the audience characteristics, purposes of the presentation, and a source materials, the learner will be able to develop a presentation that incorporates the principles of visual design and good lettering. Given a description of the audience characteristics, purposes of the presentation, and a handout developed for a presentation, the learner will be able to evaluate the handout based upon the principles of visual design, effective lettering, and appropriate use of the medium, and to suggest revisions.

Case study: One summative assessment 50-item test, open-ended and multiple-choice questions List and discuss the eight principles of visual design Which of the following is not a characteristic of the design element “surface”? A. texture B. color C. value D. form

Case study: Test results Instructor was happy with grade distribution

WHY (to assess)? Stakeholders Teacher/course designer Students Others (institution, employers, parents, etc.)

Student perspective Student behaviour is driven largely by assessment “How can I meet the assessment requirements most effectively (in the least amount of time and with the least effort)”?

Course design questions Where are we going? How will we get there? How do we know when we’re there?

WHEN (to design assessment)? Learning outcomes Instructional strategies

Course design sequence 1. Write learning objectives 2. Design assessment items (part of task analysis) What will constitute evidence that the learners have learned? How should students demonstrate their knowledge? Are there skills to assess together with content? Learning objectives Assessment

Why writing assessment items immediately after writing objectives? The intentions of the objectives will be fresh in your mind If you cannot write an item for the objective, maybe you need to reexamine the objective Learning objectives Assessment

Robert Gagne: “Conditions of Learning” (1965). McTighe and Wiggins: “Backward design” (1988). John Biggs: “Constructive alignment” (1999). Dee Fink : “Significant learning” (2003)

Takeaway 1 Try to think of assessment as soon as you know the course goals and learning outcomes

Adequate or passing performance % of items learners must get correct “Learners must get at least one correct out of each of the easy, medial and difficult categories”

Designing assessment blueprint:

Takeaway 2 As you start designing assessment, think about the desired level of performance. Is the required passing level ensuring the minimal acceptable performance?

WHAT & WHEN (to assess) A1 A2 A3

Types of assessment - WHEN Entry Skill Assessments Skills and knowledge one must have to begin instruction Preassessment What learners already know about the topic that they are to learn about though instruction Can be used to gain students’ attention and to inform of the objective(s) of the lesson Focus on the portion of the instruction not previous learned Postassessment

Types of assessment – FOR WHAT PURPOSE Formative Assessment How is the student progressing? Combine with feedback – corrective measures. The purpose is LEARNING Summative Assessment What did the student learn in this course? Can student combine the knowledge and skills and demonstrate performance at the expected level? The purpose is to determine the LEVEL of COMPETENCE

WHAT to assess?

Learning to drive

Learning Domains based on the characteristics of the “content” that a learner must learn Verbal information (declarative knowledge) Intellectual skills individual competence and ability to respond to stimuli Cognitive strategies capability to learn, think, and remember Attitudes ingrained bias towards different ideas, people, situation Psychomotor skills capability to learn to drive, ride a bike, draw a straight line, etc. Robert Gagné 1916-2002 “Conditions of Learning” 1965. “Most fundamental, most widely used and most useful in designing instructional materials” (Smith & Ragan, 2005.)

WHAT (to assess)? Verbal information (declarative knowledge) Intellectual skills (procedural knowledge) Cognitive strategies Attitudes Psychomotor skills + Enterprise schema and Integrated goals (With D. Merrill)

WHAT (to teach and assess)? Verbal information (declarative knowledge) Intellectual skills (procedural knowledge) Cognitive strategies Attitudes Psychomotor skills

Declarative Knowledge: “Knowing that…” Recall in verbatim, paraphrased, or summarized form facts, lists, names, or organized information Recall, recognize, state in own words Knowing declarative knowledge helps students learn higher-order, more complex objectives, such as intellectual skills

WHAT (to teach and assess)? Verbal information (declarative knowledge) Intellectual skills Cognitive strategies Attitudes Psychomotor skills

Intellectual skills: “ Knowing how…” The application of rules to previously unencountered examples

Intellectual Skills: Subcategories Discrimination Concepts Principles (if-then) Procedures Problem Solving

Discrimination: learning to differentiate, to perceive that something either matches or differs from other things

Which of a number of figures is a rhombus? The acquisition of concepts helps the learner to simplify the world (responds to things as members of “groups”) Which of a number of figures is a rhombus?

Principles (if-then) help us predict, explain, control “Any body completely or partially submerged in a fluid (gas or liquid) at rest is acted upon by an upward, or buoyant, force the magnitude of which is equal to the weight of the fluid displaced by the body.”

Procedures: In what order certain steps should be taken (and why)

Problem Solving: Learned capability involving selection and application of multiple rules Proving a geometric theorem (learning how to respond to a class of problems)

Takeaway 3 Are you assessing “Knowing that” or “Knowing how” (or both)? If you are assessing “knowing how”, are you PROGRESSIVELY assessing skills and knowledge in Discrimination Concepts Principles Procedures Problem solving

WHAT (to assess)? Verbal information (declarative knowledge) Intellectual skills Cognitive strategies Attitudes Psychomotor skills

Cognitive strategies (knowing how to learn) Rehearsal strategies (select, recall) Elaboration strategies (tie new information to prior knowledge) Organizational strategies (define the relationship among information) Metacognition (knowledge about own cognitive processes and ability to control them) Affective strategies (for focusing attention, maintaining concentration, establish and maintain motivation, manage time)

WHAT (to assess)? Verbal information (declarative knowledge) Intellectual skills Cognitive strategies Attitudes Psychomotor skills

Attitudes (“How to behave”) Attitudes influence the choices that learners make Instruction in attitudes often subtle and indirect Simulation games

WHAT (to assess)? Verbal information (declarative knowledge) Intellectual skills Cognitive strategies Attitudes Psychomotor skills

Psychomotor Skills Coordinated muscular movements that are typified by smoothness and precise timing

Learning to drive

Takeaways Plan for assessment as soon as you know the course goals and intended learning outcomes Consider the purpose of assessment and when in the course it should happen Think about the expected “performance” – what is a desired level? What is a passing level?

Takeaway (cont’d) Design “assessment blueprint” aligned with intended learning outcomes There are different domains of learning and they should be assessed accordingly

References Smith P.L. & Ragan T. J. (2005) Instructional Design (3rd ed.). Hoboken, NJ: Wiley Jossey-Bass Education. Gagné, R.M. (1985). The Conditions of Learning and Theory of Instruction (4th ed.). New York: CBS College Publishing. Bloom B.S. , Hastings, J.T., Madaus, G.F. , (1971) Handbook on formative and summative evaluation of student learning. McGraw-Hill, New York. Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education 32:347- 364. Wiggins G. and McTighe J., (2005). Understanding by Design. Alexandria, VA : Association for Supervision and Curriculum Development.

Assessment Planning: Why,When,What? May 2015, EDC, Carleton University Thank You Assessment Planning: Why,When,What? May 2015, EDC, Carleton University Maristela Petrovic-Dzerdz, MEng, BEd, PBCID Instructional Designer

Q&A Session with Dr. Anthony Marini 3M Teaching Fellow Q&A Session with Dr. Anthony Marini 3M Teaching Fellow EDC Senior Teaching Development Associate May 2015

Question 1 “I have a question about the expected performance and desired level. Today we hear students say that if you have paid your tuition, you should expect to pass the course. We had in the past cases when (and it was also in newspapers) too many students fail the course, so the question was raised if the ‘clients’ pay, yes they should expect a good course design, but should they also expect to pass the course because they paid for it?”

Question 2 “What to do if you have a graduate student who, you realize, does not have skills necessary at the graduate level? We know the rule that you should not get less than B- in a graduate course. A couple of times in my career I had to give a graduate student less than B- , and in one particular case I had to give student, who was about to graduate, a D- ; I was very concerned about this person going into the world with masters degree and Carleton University credentials and I wanted to send a strong message. The student complained, appealed…It is very difficult. What should be the passing level? Maybe I have not been explicit about the passing level up front? “

Question 3 “The ‘safe and effective’ thing resonates with me, since I teach engineering. If someone gets D- in my engineering design course, I wouldn’t trust him to design a house for my dog. From course design, from best educational practices perspective, it makes a lot of sense; but in university environment, do I have a freedom to choose what the passing level should be and not allow for a 50% to be a passing level? “

Question 4 “In our department we had a discussion because it seems that the grades are too high. Some instructors say that the students put so much effort, came to all the classes, all the office hours…How much can we justify grading the effort as to grading the actual level of performance in the course? “

Question 5 “In Europe, we have that maybe 20% of your mark is effort and participation and the 80% is actual performance. Maybe this is a solution?“