1. Identification and Mapping of Student Learning Outcomes
Karl A. Smith
STEM Education Center / Technological Leadership Institute / Civil Engineering – University of Minnesota &
Engineering Education – Purdue University -
King Fahd University of Petroleum and Minerals
Design and Implementation of Cooperative Learning
August 19-21, 2013

2. Session 2 Layout Reflection on Session 1
Enduring student learning outcomes (BIG ideas)
Taxonomies of learning outcomes
Connecting outcomes and assessment strategies 2

3. Five Minute University
Streveler and Smith

4. Content-Assessment-Pedagogy (CAP) Design Process Flowchart
Understanding by Design (Wiggins & McTighe, 2005)
Context
Content
Assessment
Pedagogy
C & A & P
Alignment?
End
Start
Yes No
Backward Design
Streveler, Smith & Pilotte (2012) 4

5. 3 Stages of Backward Design
What should learners know, understand, and be able to do?
Identify the Desired Results
Determine Acceptable Evidence
Plan Learning Experiences
Are the desired results, assessments, and learning activities ALIGNED?
Streveler and Smith

6. 3 Stages of Backward Design
Identify the Desired Results
How will we know if the learners have achieved the desired results? What will be accepted as evidence of learners’ understanding and proficiency?
Determine Acceptable Evidence
Plan Learning Experiences
Are the desired results, assessments, and learning activities ALIGNED?
Streveler and Smith

7. 3 Stages of Backward Design
Identify the Desired Results
Determine Acceptable Evidence
What activities will equip learners with the needed knowledge and skills? What materials and resources will be useful?
Plan Learning Experiences
Are the desired results, assessments, and learning activities ALIGNED?
Streveler and Smith

8. Elements of Content Knowledge-centered aspects of Content
Focuses on the information, topics, “stuff” of the learning unit
Curricular priorities
Learning-centered aspects of Content
Focuses on how the learner interacts with the content
Streveler and Smith

9. Designing Learning Environments Based on HPL (How People Learn)
Streveler and Smith

10. Knowledge-Centered What do you want learners to learn?
Streveler and Smith

11. Understanding Big Ideas
To understand a topic or subject is to use knowledge in sophisticated, flexible ways. Knowledge and skill are necessary elements of understanding, but they are not synonymous with understanding. Matters of understanding require more: [Learners] need to make conscious sense and apt use of the knowledge they are learning and the principles underlying it.
-Understanding by Design
Wiggins and McTighe (1998)
Streveler and Smith

12. Wiggins & McTighe Curricular Priorities
Good to be familiar with
Important to know
Enduring outcomes
Streveler and Smith

13. Streveler and Smith

14. Understanding Misunderstanding
A Private Universe – 21 minute video available from
Also see Minds of our own (Annenberg/CPB Math and Science Collection –
Can we believe our eyes?
Lessons from thin air
Under construction

15. Exercise Determine for your curriculum (re)design
Enduring outcomes
Important to know or understand
Good to be familiar with
Streveler and Smith

16. How do people learn?
Learner-centered
Streveler and Smith

17. Information Processing Model of Learning
Input via attention – to short term memory – to long term memory – retention and retrieval
Streveler and Smith

18. Information Processing Model of Learning
Key areas for instruction
Attention and processing power = cognitive load (bandwidth)
LIMITED – need to be careful how one uses the learner’s bandwidth
Connected nature of memory
Link to what is already known
PRIOR KNOWLEDGE
Need for organization
Structural knowledge – ADVANCED ORGANIZERS
Multiple inputs (both in timing and in modes of input) = number and strength of connections = richer learner that will be retained longer
PRACTICE
MULTIPLE MODES OF INPUT (visual, audio, explanation to yourself and others)
Streveler and Smith

19. Exercise
What are your initial ideas about how you can help your learners achieve the enduring outcomes of the course you are redesigning?
Streveler and Smith

20. How will you know learners have learned what you want them to learn?
Assessment
Streveler and Smith

21. Objectives of Assessment What is the Purpose of the Assessment?
We cannot measure learning directly. Instead we must make inferences from evidence.
Sound inferences depend on assessments that are aligned with content.

22. Key questions to ask yourself about assessment
What should I be assessing?
What is the best assessment to use?
How can I be sure I am consistently interpreting the results of the assessments?
Streveler and Smith

23. 1. What should I be assessing?
Assessments should be aligned with your curricular priorities.
Enduring outcomes should ALWAYS be assessed.
Important to know outcomes are usually assessed.
Good to be familiar with information might not be assessed.
Streveler and Smith

24. 2. What is the best assessment to use
2. What is the best assessment to use? Assessments Aligned to Curricular Priorities
Traditional Quizzes and Tests
Selected Response
Closed-ended, convergent
Well-structured
Scored via “answer key”
Familiar with
Academic Prompts
Constructed Response
Open-ended, ill-structured, divergent
Academic conditions (e.g. exams, drills)
Requires analysis, evaluation, and/or synthesis
Judgment-based scoring (e.g., rubrics)
Important to know
Understanding
Enduring
Performance Tasks
Papers, projects, design tasks, etc.
Open-ended, complex, ill-structured, divergent
Approximation of practice, specific audience
Higher autonomy, more personalized
Judgment-based scoring (e.g., rubrics)
Adapted from: Wiggins, G. & McTighe, J. (1997). Understanding by Design. Alexandria, VA: ASCD

25. 3. How can I be sure I am consistently interpreting the results of the assessments?
For close-ended and well-structured assessment prompts.
Construct an accurate grading key
Determine what level of performance is acceptable for demonstrating “mastery” [i.e. “pass”]
For more open-ended prompts and for performance tasks
Construct a matrix what will link key features with how those features will be demonstrated to show mastery. This is called a RUBRIC.
Determine what level of performance is acceptable for demonstrating “mastery.”
Streveler and Smith

26. Some steps for creating a rubric
Look at work you feel is of varying levels of quality.
What characteristics make some work “better” than others?
What are markers within the work point to better performance?
Exercise: Work together to create a rubric for assessing
A memo
A weld
Streveler and Smith

27. Learning Objectives and Taxonomies
Learning objectives are the bridge between what you want learners to learn and how you know they learned it.
Learning Objectives and Taxonomies
Streveler and Smith

28. Constructing Learning Objectives Learning Activities vs
Constructing Learning Objectives Learning Activities vs. Learning Objectives
Streveler and Smith

29. Constructing Learning Objectives Using Verb-Noun Format
Streveler and Smith

30. Mapping Learning Objectives to Curricular Priorities
Annotated Example: Ruth Wertz - High Level Weekly Planning (Annotated Example).docx
Streveler and Smith

31. Taxonomies
Bloom’s taxonomy of educational objectives: Cognitive Domain (Bloom & Krathwohl, 1956)
A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (Anderson & Krathwohl, 2001).
Taxonomy of significant learning (Fink, 2003)
Evaluating the quality of learning: The SOLO taxonomy (Biggs & Collis, 1982)
Facets of understanding (Wiggins & McTighe, 1998)
Streveler and Smith

32. Revised Bloom’s Learning Taxonomy
Source:
Streveler and Smith

33. Revised Bloom’s Learning Taxonomy
Source:
Streveler and Smith

34. Revised Bloom’s Learning Taxonomy
Source:
Streveler and Smith

35. Mapping Learning Objectives Range of Low-Order & High-Order Objectives
Annotated Example: Ruth Wertz - Map of Weekly Learning Objectives (Annotated Example).docx
Streveler and Smith

36. Exercise For the course you are (re)designing:
Starting with your enduring outcomes – write learning objectives for each of your 5 most important curricular priorities.
Place those learning objectives in Bloom’s revised taxonomy.
Streveler and Smith

37. Session 3-4 Preview
Pedagogies of Engagement – Cooperative Learning and Challenge Based Learning
Informal – Bookends on a Class Session
Formal Cooperative Learning
Key Resource
Review Smith, Sheppard, Johnson & Johnson, “Pedagogies of engagement.” 37

38. Kinds of Assessment
Streveler and Smith

39. Objectives of Assessment What is the Purpose of the Assessment?
Formative
Summative
Provide Quantitative Feedback (e.g., score or grade)
Evaluate Completion of Learning Objectives
Summarize Learning Achievement
Provide Qualitative Feedback
Evaluate Learning Progress
Improve/Redirect Learning Activities
We cannot measure learning directly. Instead we must make inferences from evidence. Sound inferences depend on assessments that are aligned with content.

40. Assessment Examples Formative and Summative Assessments
Tests & Quizzes
Self-test
Quiz
Written Exam
Academic Prompts
Group Discussion
Reflection Paper
Peer Review
Written Assignment
Practical Exam
Performance Tasks
Oral Exam
Practice lab practical with feedback
Design Project
Research Paper
Case Study
Graded lab practical
Observation & Dialogue
Informal Check for Understanding
Performance Evaluation
Examples above are not exhaustive.

41. Glossary of important terms in summative assessment
Criterion model of assessment
Determine a level of learning considered acceptable and then measure every learner against that criterion.
All could pass, none could pass.
Normative model of assessment
Learners are measured against each other.
“Grading on a curve.”
Rubrics
Streveler and Smith

42. Examples of formative feedback
Streveler and Smith

43. Quick Thinks Reorder the steps Paraphrase the idea Correct the error
Support a statement
Select the response
Johnston, S. & Cooper,J Quick thinks: Active- thinking in lecture classes and televised instruction. Cooperative learning and college teaching, 8(1), 2-7
Streveler and Smith

44. Minute Paper
What was the most useful or meaningful thing you learned during this session?
What question(s) remain uppermost in your mind as we end this session?
What was the “muddiest” point in this session?
Give an example or application
Explain in your own words . . .
Angelo, T.A. & Cross, K.P Classroom assessment techniques: A handbook for college teachers. San Francisco: Jossey Bass.
Streveler and Smith

45. Session Summary (Minute Paper)
Reflect on the session
Most interesting, valuable, useful thing you learned.
Things that helped you learn.
Question, comments, suggestions.
Pace: Too slow Too fast
Relevance: Little Lots
Instructional Format: Ugh Ah
Streveler and Smith

46. MOT 8221 – Spring 2009 – Session 1
Q4 – Pace: Too slow Too fast (3.3)
Q5 – Relevance: Little Lots (4.2)
Q6 – Format: Ugh Ah (4.4)
Streveler and Smith

47. Exercise For your 5 most important learning objectives:
What kind of assessments might you use to measure learners’ attained the level of mastery of the learning objectives?
Where can you insert formative assessment to check learners’ progress?
Streveler and Smith

48. Taxonomies of Types of Learning
Bloom’s taxonomy of educational objectives: Cognitive Domain (Bloom & Krathwohl, 1956)
A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (Anderson & Krathwohl, 2001).
Facets of understanding (Wiggins & McTighe, 1998)
Taxonomy of significant learning (Fink, 2003)
Evaluating the quality of learning: The SOLO taxonomy (Biggs & Collis, 1982) 49

49. The Six Major Levels of Bloom's Taxonomy of the Cognitive Domain (with representative behaviors and sample objectives)
Knowledge. Remembering information Define, identify, label, state, list, match
Identify the standard peripheral components of a computer
Write the equation for the Ideal Gas Law
Comprehension. Explaining the meaning of information Describe, generalize, paraphrase, summarize, estimate
In one sentence explain the main idea of a written passage
Describe in prose what is shown in graph form
Application. Using abstractions in concrete situations Determine, chart, implement, prepare, solve, use, develop
Using principles of operant conditioning, train a rate to press a bar
Derive a kinetic model from experimental data
Analysis. Breaking down a whole into component parts Points out, differentiate, distinguish, discriminate, compare
Identify supporting evidence to support the interpretation of a literary passage
Analyze an oscillator circuit and determine the frequency of oscillation
Synthesis. Putting parts together to form a new and integrated whole Create, design, plan, organize, generate, write
Write a logically organized essay in favor of euthanasia
Develop an individualized nutrition program for a diabetic patient
Evaluation. Making judgments about the merits of ideas, materials, or phenomena Appraise, critique, judge, weigh, evaluate, select
Assess the appropriateness of an author's conclusions based on the evidence given
Select the best proposal for a proposed water treatment plant 50

50. The Cognitive Process Dimension
Remember
Understand
Apply
Analyze
Evaluate
Create
Factual Knowledge – The basic elements that students must know to be acquainted with a discipline or solve problems in it.
a. Knowledge of terminology
b. Knowledge of specific details and elements
Conceptual Knowledge – The interrelationships among the basic elements within a larger structure that enable them to function together.
a. Knowledge of classifications and categories
b. Knowledge of principles and generalizations
c. Knowledge of theories, models, and structures
Procedural Knowledge – How to do something; methods of inquiry, and criteria for using skills, algorithms, techniques, and methods.
a. Knowledge of subject-specific skills and algorithms
b. Knowledge of subject-specific techniques and methods
c. Knowledge of criteria for determining when to use appropriate procedures
Metacognitive Knowledge – Knowledge of cognition in general as well as awareness and knowledge of one’s own cognition.
a. Strategic knowledge
b. Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge
c. Self-knowledge
The Knowledge Dimension 51
(Anderson & Krathwohl, 2001).

51. 52

52. http://www. celt. iastate. edu/pdfs-docs/teaching/RevisedBloomsHandout 53

53. 54

54. Cognitive
Affective Me t a

55. SOLO Taxonomy
The Structure of Observed Learning Outcome (SOLO) model consists of 5 levels of understanding
Pre-structural - The task is not attacked appropriately; the student hasn’t really understood the point and uses too simple a way of going about it.
Uni-structural - The student's response only focuses on one relevant aspect.
Multi-structural - The student's response focuses on several relevant aspects but they are treated independently and additively. Assessment of this level is primarily quantitative.
Relational - The different aspects have become integrated into a coherent whole. This level is what is normally meant by an adequate understanding of some topic.
Extended abstract - The previous integrated whole may be conceptualised at a higher level of abstraction and generalised to a new topic or area.

56. Teaching Teaching and Understanding Understanding
Biggs SOLO taxonomy