1. CONALI ontology A framework for design and evaluation of constructively aligned courses in higher education: putting in focus the Educational Goal Verbs Antonio Maffei , Lorenzo Daghini , Andreas Archenti , Niels Lohse   KTH Royal Institute of Technology. Dept. of production Engineering  Loughborough University, Wolfson School of Mechanical and Manufacturing Engineering,

2. Outline Constructive alignment: international perspective and KTH reception Application and Measurment of constructive alignment

3. Constructive alignment in literature (a) Annual number of paper with CA as main topic in Scopus and Web of Science. (b) Annual number of documents featuring CA in Google Scholar

4. Constructive alignment at KTH LH201V Learning and Teaching. Available: http://www.kth.se/student/kurser/kurs/LH201V?l=en http://www.kth.se/student/kurser/kurs/LH201V?l=en LH202V Knowledge Building in Teachers Practice. Available: http://www.kth.se/student/kurser/kurs/LH203V?l=en http://www.kth.se/student/kurser/kurs/LH203V?l=en LH216V Develop the Learning by Using Grading Criteria. Available: http://www.kth.se/student/kurser/kurs/LH216V?l=en http://www.kth.se/student/kurser/kurs/LH216V?l=en …

5. Problem with Constructive alignment... Problematic areas connected with CA 1. Scarcity of Resource (Biggs) 2. Resistance to Change (Biggs) 3. High interest and consequent risk for uncontrolled and diverging growth (Authors) 4. Assessing CA (Authors plus literature)

6. ...and solution offered by ontological approach Problematic areas connected with CABenefit of producing an ontology 1. Scarcity of Resource (Biggs) - To enable reuse of domain knowledge - To analyze domain knowledge 2. Resistance to Change (Biggs) - To make domain assumptions explicit - To separate domain knowledge from the operational knowledge 3. High interest and consequent risk for uncontrolled and diverging growth (Authors) - To share common understanding of the structure of information among people or software agents 4. Assessing CA (Authors plus literature) - Through analysis of the body of knowledge produced and definition of specific KPI

7. Contribution of this work  Produce the first iteration of an ontological framework model for design and evaluation of constructively aligned courses (Hence CONALI ontology).  Instantiate the defined CONALI ontology on a selected set of courses given at KTH IIP  Define and analyze some KPI connected with the alignment of educational units and interrogate the knowledge base on the performance attained by the represented courses and the areas where improvement is necessary.

8. Main assumption behind CA

9. Intended Learning Outcomes

10. Instantiation of the ontology: EGV Remembering Verbs Understanding Verbs.... Creating Verbs

11. Instantiation on a course ILOEDVTLAATKoKnow 01DescribeLecture Multiple Choice Short Answer Declarative (D) 02ChooseLectureMultiple Choice D 03NameLectureMultiple Choice D 04Plan Learning Cell Group Project Functional (F) Peer TeachingStudent Presentation 05Use Tutorial Case Based Learning Group Assessment F 06Produce Interactive Work in Class Short Practical Exercise F 07DescribeLectureMultiple Choice D 08Understand Learning Cell Group Project F Peer TeachingStudent Presentation

12. KPI#1 %A EdU : Percentage of alignment Scenario Score EGV does not indicate a learner action 0 EGV indicates a learner action Partial alignment  EGV ILO = EGV TLA and EGV ILO = EGV AT  EGV ILO = EGV AT and EGV ILO = EGV TLA 3 Perfect Alignment  EGV ILO = EGV TLA = EGV AT 9 Aggregate KPI Single EGV KPI Single ILO KPI

13. Kind of Knowledge

14. Kind of Knowledge alignment

15. KPI#2 %KA EdU : Kind of Knowledge alignment Scenario Score K EGV /ILO K TLA K AT DDD1 DDF0 DFD0 DFF0 FDD0 FDF0 FFD0 FFF1 Aggregate KPI Single ILO KPI

16. Calculation of the Score ILOEDVTLAATKoKnow Score %AD/F K 01DescribeLecture Multiple Choice Short Answer Declarative (D) 93 02ChooseLectureMultiple ChoiceD93 03NameLectureMultiple ChoiceD93 04Plan Learning Cell Group Project Functioning (F) 93 Peer Teaching Student Presentation 05Use Tutorial Case Based Learning Group AssessmentF93 06ProduceInteractive Work in Class Short Practical Exercise F93 07DescribeLectureMultiple ChoiceD93 08Understand Learning Cell Group Project F00 Peer TeachingStudent Presentation

17. Score

18. Summary of the CONALI ontology class instantiation and KPI scores Course%A%KA MG1026 87.5% MG2100 81%86% MG2033 70%69% MG2040 80%88% MG2028 91%100%

19. Conclusion The case study tested the main intended functions of the ontology: Problematic area 3. The framework model was expressive enough to capture the information required to address the metrics identified as basic requirement for the evaluation of CA. Result: encouraging! The ontology is more expressive than intended Problematic area 1 and 2. The ease of use and understanding of the model. Results: good! It took between 2-4 hours to instantiate a course;

20. Conclusion...some correlation Course%A%KAYears Teachers CAEduc / Teachers total MG1026 87.5% 21/1 MG2100 81%86% 11/1 MG2033 70%69% 11/5 MG2040 80%88% 01/2 MG2028 91%100% 82/2 Problematic area 4. The following correlation between the results of the analysis and two parameters possibly related with CA quality suggest that the chosen KPI and related method are sound

21. Future works  Expand the domain of investigation from single EdU to entire programs. This includes also connection between different EdUs.  Include school in different countries and different discipline to validate the requirements identified in this pilot study and introduce new ones.  Introduce the connection between assessment task and grading scale. As Biggs point out, having a clear alignment among these two entities is fundamental for the alignment of the course. In view of this different classes must be created for formative and summative assessment in future releases.  Introduce capability to track EdU across different years with specific metrics to identify and address the “child diseases”.  Problematic area 4. Integrate the CONALI ontology with valid existing contributions in the domain of CA measurement.

22. Required tools  Ad-hoc system to interrogate the knowledge base  Expert system that can suggest the best way of teaching content based on best practice in the available knowledge base.

23. KTH ROYAL INSTITUTE OF TECHNOLOGY Thank You!