1. How to make sure your Students Learn what you want them to?
Claus Brabrand
((( )))
((( )))
Associate Professor, Ph.D. Department of Computer Science IT University of Copenhagen, Denmark

2. How to apply the theory of Constructive Alignment & The SOLO Taxonomy
Claus Brabrand
((( )))
((( )))
Associate Professor, Ph.D. Department of Computer Science IT University of Copenhagen, Denmark

3. Outline (Feb 14, 2017) 1) Introduction 2) From Content to Competence
Constructive Alignment
The SOLO Taxonomy
2) From Content to Competence
Advocate a shift in perspective
Motivation for "competence"
3) In Practice
Concrete recommendations
How to implement Alignment

4. Constructive Alignment
Make explicit ILO's ( )
(…and tell this to students)
Intended
Learning
Outcomes
= ILO's =
Teaching
Learning Activities
Assessment
vs.
SUSAN:
intrinsically motivated
ROBERT:
extrinsically motivated

5. Outline (Feb 14, 2017) 1) Introduction 2) From Content to Competence
Constructive Alignment
The SOLO Taxonomy
2) From Content to Competence
Advocate a shift in perspective
Motivation for "competence"
3) In Practice
Concrete recommendations
How to implement Alignment

6. From Content to Competence
My old course descriptions (Concurrency 2004):
Given in terms of a 'content description'.
Essentially:
Goal is…:
To understand:
deadlock
interference
synchronization
...
This is a bad idea
for two reasons...!

7. research-based tradition
Problem 1 !
Problem with 'content' as goals !
analyze ...
theorize ...
analyze systems
explain causes
define deadlock
describe solutions
agreement
research-based tradition
tacit knowledge from a
Goal is…:
To understand:
deadlock
interference
synchronization
...
Stud. C
not known by student
Teacher
name solutions
recite conditons
analyze systems
explain causes
Stud. B 
Stud. A
External
Examiner

8.  Problem 2 ! Problem with 'understanding' as goals !
Goal is…:
To understand:
deadlock
interference
synchronization
...
'concept of deadlock' ?! 
The answer is simple:
It cannot be measured !

9. Note: 'understanding' is of course pre-requisitional !
Competence !
'Competence' as goals !
Competence := knowledge capacity to act upon it
Have the student do something;
and then observe (evaluate) the product and/or process
Objective !
To learn how to:
analyze systems for...
explain cause/effects...
prove properties of...
compare methods of...
...
Note: 'understanding' is of course pre-requisitional ! 
Note': inherently operational (~ verbs)
'SOLO' = Structure of the Observed Learning Outcome

10. "How does this content vs competence relate to YOUR courses?"
Neighbour Discussion
"How does this
content vs competence
relate to YOUR courses?"

11. Outline (Feb 14, 2017) 1) Introduction 2) From Content to Competence
Constructive Alignment
The SOLO Taxonomy
2) From Content to Competence
Advocate a shift in perspective
Motivation for "competence"
3) In Practice
Concrete recommendations
How to implement Alignment

12. Implementation Process
1) Think carefully about:
overall goal of course
(what students learn to do?)
2) Operationalize these goals
and formulate them as SOLO
intended learning outcomes
alignment
learning incentive
learning support
3) Choose carefully the
form(s) of assessment
(~ intended learning outcomes)
4) Choose carefully the
form(s) of teaching
(~ intended learning outcomes)

13. Concrete Recommendations
1) Use 'standard formulation':
a) puts learning focus on the student
b) competence formulation: "to be able to"
Intended Learning Outcomes [Genetics 101]
After the course, the students are expected to be able to:
locate genes on chromosomes
do simple calculations : (e.g., recombination frequencies, in-breeding coefficients, Hardy-Weinberg, evolutionary equilibria).
describe and perform connexion-analysis
describe fundamental genetic concepts: (e.g., mutation variation, in-breeding, natural selection).
describe and analyze simple inheritancies
analyze inheritance of multiple genes simultaneously
4) Avoid 'understanding-goals':
"To understand X",
"Be familiar with Y",
"Have a notion of Z" ! V N V N V V N V V V N V N
3) Use 'Verb + Noun' formulation:
What the student is expected to
do with a given matter .
2) List sub-goals as 'bullets':
Clearer than text V N

14. Implementation Process
1) Think carefully about:
overall goal of course
(what students learn to do?)
2) Operationalize these goals
and formulate them as SOLO
intended learning outcomes
alignment
learning incentive
learning support
3) Choose carefully the
form(s) of assessment
(~ intended learning outcomes)
4) Choose carefully the
form(s) of teaching
(~ intended learning outcomes)

15. Implementation Process
1) Think carefully about:
overall goal of course
(what students learn to do?)
2) Operationalize these goals
and formulate them as SOLO
intended learning outcomes
alignment
learning incentive
learning support
3) Choose carefully the
form(s) of assessment
(~ intended learning outcomes)
4) Choose carefully the
form(s) of teaching
(~ intended learning outcomes)

16. Teaching/Learning Activities
functional knowledge
(problem oriented)
Project work
Case teaching
teacher centric
student centric
Exercise class
Lecture
declarative knowledge
(discipline oriented)

17. Lecture (about Application)
Teacher activity:
Introduce
Explain
Elaborate
Discuss application
Give examples
Show PPT slides
Questions on slides
Winding up
vs.
Student activity:
Listen
Listen (maybe take notes)
Understand? (correctly? deeply?)
Watch (maybe note points)
Write answers to questions
Possibly ask a question
active
teacher
[ Biggs & Tang 2007, p.137 ]
vs.
passive
student

18. Learn about vs. Learn to do
Learning (about):
about application
about cooking
about programming
about designing
about analysis
about construction
about relating
...
Learning (to do):
to apply
to cook
to program
to design
to analyse
to construct
to relate
...
student listening
(to something
about something)
vs.
student
doing !!

19. Student Activation "The (in-famous) Learning Pyramid": passive student
Average
retention rate
Note: the percentages
are completely bogus
Lecture 5%
Listening:
Learning about
passive
student
Reading
10%
Audiovisual
20%
student
activation
Demonstration
30%
active
student
Discussion group
50%
Practice by doing
75%
Doing:
Learning to do
Teaching others
80%
[ NTL Institute for Applied Behavioral Science, Bethel, Maine ]

20. Constructive Alignment
Make explicit ILO's ( )
(…and tell this to students)
Intended
Learning
Outcomes
= ILO's =
Teaching
Learning Activities
Assessment
vs.
SUSAN:
intrinsically motivated
ROBERT:
extrinsically motivated

21. Teaching / Learning Activities Students at University
Questions... R R' Q
Cognitive processes
My research
and teaching
"understanding"
content  competence
Course descriptions
Association
new ~ old
The SOLO Taxonomy
'TLA'
Teaching / Learning Activities
Teacher models
levels
The Film
Susan & Robert
The Book ?
Tips'n'Tricks
Student activation
CS v. NAT v. MAT
recite
generalize
15% programming
Students at University
"What is good teaching?"
John Biggs
Top Competences
Constructive Alignment

22. Now, please: "2-minute recap"
Please spend 2' on thinking about and writing down the most important points from the talk – now!:
Immediately
After 1 day
After 1 week
After 2 weeks
After 3 weeks

23. Key References
”Teaching for Quality Learning at University” John Biggs & Catherine Tang Society for Research into Higher Education, McGraw-Hill.
”Evaluating the Quality of Learning: The SOLO Taxonomy” John Biggs & Kevin F. Collis London: Academic Press, 1982
”Teaching Teaching & Understanding Understanding” Claus Brabrand & Jacob Andersen 19 minute award-winning short-film (DVD) Aarhus University Press, Aarhus University, 2006
"Constructive Alignment & The SOLO Taxonomy: a Comparative Study of University Competencies in Computer Science vs. Mathematics" Claus Brabrand & Bettina Dahl CRPIT, Vol. 88, ACS 3-17, R. Lister & Simon, Eds., 2007

24. Thank You! ((( http://www.daimi.au.dk/~brabrand/short-film/ )))
Film's homepage:
((( )))

25. T
Exercise:
"What is good teaching?"

26. Neighbour Discussion:T
Neighbour Discussion:
"What are the film's main messages
(...in your opinion)"?

27. Introduction to… Constructive Alignment & SOLO Taxonomy:
John Biggs’ popular and heavily cited book:
“Teaching for Quality Learning
at University
- What the student does”
13,410 citations!
[J. Biggs & C. Tang, 2007]
“Teaching Teaching
&
Understanding Understanding”
300,000+ views!
19 min award-winning short-film on Constructive Alignment
(available on DVD in 7 languages, epilogue by John Biggs)

28. SOLO (elaborated) QUANTITATIVE QUALITATIVE SOLO 2 ”uni-structural”
Note: the list is non-exhaustive
SOLO (elaborated)
Graphic Legend
problem / question / cue
known related issue - given!
hypothetical related issue - not given!
student response Q R
QUANTITATIVE
QUALITATIVE
SOLO 2
”uni-structural”
SOLO 3
“multi-structural”
SOLO 4
“relational”
SOLO 5
“extended abstract”
define
identify
count
name
recite
paraphrase
follow (simple) instructions …
combine
structure
describe
classify
enumerate
list
do algorithm
apply method …
analyze
compare
contrast
integrate
relate
explain causes
apply theory (to its domain) …
theorize
generalize
hypothesize
predict
judge
reflect
transfer theory (to new domain) … R R' Q R Q R Q R Q

29. Common SOLO Competences

30. SOLO Advantages Advantages of The SOLO Taxonomy:
Linear hierarchical structure (good for progression)
Aimed at evaluating student learning
Converges on research (at SOLO 5)
Research: Production of new knowledge

31. "Key competences in YOUR course?" Exercise T Concurrency:
analyze systems
compare models

32. Tips'n'Tricks (activation)
Neighbour discussions:
Post-It exercise:
more questions (students dare ask them)
better questions (students had a chance to discuss)
focus: zoom in
anonymous (!)
swap'able
everyone will engage
empathetic control
shared knowledge pool
1-2 min timeout
[Phil Race]
Form variation:
Frequent breaks:
pulse reader measurements:
lecturing blended with in-class activation exercises

33. Tips'n'Tricks (cont'd)
Use many examples: (build on student pre-knowledge)
Explicit structure:
1. xxxxxxxxxx
2. yyyyyyyyyy
3. zzzzzzzzzz
4. wwwwwww
1. xxxxxxxxxx
2. yyyyyyyyyy
3. zzzzzzzzzz
4. wwwwwww
1. xxxxxxxxxx
2. yyyyyyyyyy
3. zzzzzzzzzz
4. wwwwwww
1. xxxxxxxxxx
2. yyyyyyyyyy
3. zzzzzzzzzz
4. wwwwwww
NEW
OLD 
self evident to you [ teacher ]
not to a learner [ student ]
(esp. during learning process)
"Less-is-more":
Student 'recap' at end:
analyze
compare
relate
common deadlock, uncommon deadlock, A-synchronization, B-synchronization, hand-shake, multi-party synchronization, multi-party hand-shake, binary semaphores, generalized semaphores, blocking semaphores, recursive locks, ...
vs.
now
after 1 day
after 1 week
after 2 weeks
after 3 weeks
Emphasize depth over breadth (coverage)

34. ” The BLOOM Taxonomy (1956) The BLOOM Taxonomy: Analysis Synthesis
Evaluation
SOLO 4+5
Qualitative
Application
Comprehension
SOLO 2+3
Quantitative
Knowledge ”
”[…] really intended to guide the selection of items for a test rather than to evaluate the quality of a student’s response to a particular item”
-- (Biggs & Collis, 1982)

35. Problematic Courses E.g. course: ”Databases” (at RUC/Roskilde):
Note: almost entirely non-operational(!)
i.e. measure how?!
obtain knowledge about the structure of database systems;
be familiar with design of databases by use of special notations like E/R and analysis through normalization;
get an overview of the most important database models and a detailed knowledge about the most important model - the relational model as well as the language SQL;
get an overview of database indexing and query processing;
obtain knowledge about application programming for DB systems.
Familiar with ?!

36.  UNALIGNED COURSE Teacher’s intention Student’s activity Exam’s
e.g.
explain
relate
prove
apply
"Dealing with the test"
Exam’s
assessment
e.g.
memorize
describe
e.g.
memorize
describe

37.  ALIGNED COURSE Teacher’s intention Student’s activity Exam’s
e.g.
explain
relate
prove
apply
e.g.
explain
relate
prove
apply
e.g.
explain
relate
prove
apply
Exam’s
assessment
e.g.
explain
relate
prove
apply
e.g.
explain
relate
prove
apply

38. Student Motivation Susan: (”intrinsic motivation”)
- wants to…: learn !
Robert: (”extrinsic motivation”)
- to…: pass exams !

39. Constructivism ”Transmission is Dead…” : (lectures = )
Knowledge is… Actively Constructed ! !
active teacher &
passive students
risk

40. SOLO Taxonomy Hierarchy for Competences: Deep learning (not surface) !
5: generalize, theorize, predict, …
4: explain, analyze, compare, …
3: describe, combine, classify, …
2: recite, identify, calculate, …

41. Stud Learning Focus Focus on Student Learning !
(instead of ”what teacher does”
& labelling students: ’good/bad’)
Student activitation  learning

42. Alignment Make explicit ILO’s (…and tell this to students)
(Intended Learning Outcomes):
(…and tell this to students)
Exam = ILO’s = Teaching

43. On the Role of Examination
Constructive Alignment:
A systemic theory (a teaching system w/ cause/effects)
A theory of planning (over the course of a course)
A theory of motivation (and incentive)
From the exam as a...: ...to:
"The exam does not come after, but before the course!"
"Necessary evil"
application
of alignment
Motivational and learning-guiding
pedagogical tool for the teacher(!)

44. Definition: “Good Teaching”
Good news (we now know how to do this):
Alignment!!!
Explicitly defined course objectives (as verbs)!
Discourage surface-learning!
Encourage depth-learning!
“Less-is-more”: depth rather than breadth of coverage!
”Good teaching is getting most students to use the higher cognitive level processes that the more academic students use spontaneously”
-- “Teaching for Quality Learning at University”, John Biggs, 2003

45. The SOLO Taxonomy (1982) SOLO 1: (Pre-Structural)
Here the subject does not have any kind of understanding but uses irrelevant information and/or misses the point altogether. Scattered pieces of information may have been acquired, but they are unorganized, unstructured, and essentially void of actual content or relation to a topic or problem.
SOLO 2: (Uni-Structural)
The subject can deal with one single aspect and make obvious connections. The subject can use terminology, recite (remember things), perform simple instructions/algorithms, paraphrase, identify, name, count, etc.
SOLO 3: (Multi-Structural)
At this level the subject can deal with several aspects but these are considered independently; i.e., not in connection with one another. Metaphorically speaking; the subject sees the many trees, but not the forest. He or she is able to enumerate, describe, classify, combine, apply methods, structure, execute procedures, etc.
SOLO 4: (Relational)
At level four, the subject may understand relations between several aspects and how they might fit together to form a coherent whole. The understanding forms a structure and now he or she sees how the many trees form a forest. A subject at this level may compare, relate, analyze, apply theory, explain in terms of cause and effect, etc.
SOLO 5: (Extended Abstract)
At this level, which is the highest, the subject may generalize structure beyond what was given, may perceive structure from many different perspectives, and transfer ideas to new areas. He or she may generalize, hypothesize, criticize, theorize, etc.

46. The Danish Grade Scale 12 A 10 B 7 C 4 D 02 E 00 Fx -3 F Excellent
For an excellent performance which completely meets the course objectives, with no or only a few insignificant weaknesses A
Excellent 10
For a very good performance which meets the course objectives, with only minor weaknesses
Very good B 7
For a good performance which meets the course objectives but also displays some weaknesses C
Good 4
For a fair performance which adequately meets the course objectives but also displays several major weaknesses
Fair D 02
Adequate
For a sufficient performance which barely meets the course objectives E 00
Inadequate
For an insufficient performance which does not meet the course objectives Fx -3
Unacceptable
For a performance which is unacceptable in all respects F

47. The Danish Grade Scale 12
For an excellent performance which completely meets the course objectives, with no or only a few insignificant weaknesses A
Excellent 10
For a very good performance which meets the course objectives, with only minor weaknesses
Very good B 7
For a good performance which meets the course objectives but also displays some weaknesses C
Grade := Degree of realization of course objectives!
Good 4
For a fair performance which adequately meets the course objectives but also displays several major weaknesses
Fair D 02
Adequate
For a sufficient performance which barely meets the course objectives E 00
Inadequate
For an insufficient performance which does not meet the course objectives Fx -3
Unacceptable
For a performance which is unacceptable in all respects F

48. Why -3, 00, 2, 4, 7, 10, 12? --
degree of realization of course objectives ++
A: 10%
B: 25%
C: 30%
D: 25%
E: 10%
Not passed
Passed
((( Table 11, p. 27 )))
The ECTS-skale, relative distribution
(among the ‘passed’ grades)
10%
25%
30%
25%
10%
(lowest)
22,5
(interval mid) 50
(interval mid)
77,5
(interval mid)
100
(highest)
divide by 10
and round off 2 5 8 10
symmetric transposition of lowest passed grades -5 -2
add two 02 4 7 10 12
avoid “psychological effect” with negative grades -3 00