1. An Ontological Framework for Designing the Assessment of Learning Outcomes Arkalgud Ramaprasad Professor & Head, Information and Decision Sciences, University of Illinois at Chicago Jen Sweet Assessment Specialist, University of Illinois at Chicago Shannon Milligan Coordinator of Assessment, Loyola University Chicago June 4, 2012 AALHE_05_31_20121

2. The Six Blind Men and the Elephant AALHE_05_31_20122 Image Author: Pawyi Lee; Image Source: Wikimedia Commons commons.wikimedia.orgcommons.wikimedia.org

3. Making the Assessment “ Elephant ” Visible Amidst the jungle of daily challenges To the university administrators – Presidents, Chancellors, Vice-Chancellors, Provosts, Vice-Provosts, Deans, Heads, etc. To the faculty members To students, parents, and other stakeholders AALHE_05_31_20123

4. Goals of the Assessment Professional Know the “elephant” to show the “elephant” Show the “elephant” to make stakeholders participants Approach assessment systemically Simplify assessment’s complexity Highlight the pixels and the pictures of assessment – to make them visible and accessible Apply assessment systematically AALHE_05_31_20124

5. Development of the Ontological Framework Problem Statement Literature review Review of current practice Review of accreditation requirements Ontological framework construction Interviews with users Ontological framework review and validation AALHE_05_31_20125

6. Dimensions of Assessment of Student Learning Learning outcomes to be assessed [Learning Outcomes] Unit of assessment [College] Information required to assess the learning outcomes [Information] Periodicity of the assessment of learning outcomes/collection of information [Periodic] Assessors of learning outcomes/users of information [Users] Systems for collecting the information [System] AALHE_05_31_20126

7. An Ontological Framework for Assessment - 1 “Ontology” – An “explicit specification of a conceptualization.” (Gruber, 1995, p. 907) Ontological framework for Assessment – Created in Spring 2011 by Arkalgud Ramaprasad, Seth Davidson, Paulanne Green, Tom Idzik, Pranali Lad, Philmore Morrison, Harish Rathnaala, and Ryan Sedivy, with the help of Saleha Rizvi and Jennifer Sweet AALHE_05_31_20127

8. An Ontological Framework for Assessment - 2 AALHE_05_31_20128

9. An Ontological Framework for Assessment - 3 AALHE_05_31_20129

10. An Ontological Framework for Assessment- 4 AALHE_05_31_201210

11. Illustrative Components of Assessment Acquisition of [case study write ups] for [early semester] [undergraduate] assessment of [analytical thinking skills] by [program administration] using [people at workshops] Acquisition of [standardized test scores] for [graduating seniors] [undergraduate] assessment of [core skills] by [Dean’s office] using [Blackboard] Acquisition of [competency documents] for [quarterly] [undergraduate] assessment of [competency] by [instructors] using [pen and paper] Acquisition of [alumni survey results] for [1.5 years after graduation] [undergraduate] assessment of [job placement] by [staff] using [Survey Monkey] AALHE_05_31_201211

12. Example 1: Nursing College AALHE_05_31_201212

13. Example 2: Business College AALHE_05_31_201213

14. Advantages of the Ontological Framework Comprehensive Ability to “drill down” – Conceptualize assessment systems broadly and in detail Highly Flexible – Extensible – Can adjust to changes over time Adaptable Ability to use at any institution to capture its unique systems and relationships AALHE_05_31_201214

15. Application to Assessment of Student Learning Presentation of the ontological framework to the Chicago Area Assessment Group in December, 2011 Loyola University Chicago’s use of the ontological framework to organize its assessment systems and processes at the institutional level in Spring 2012 Application to a co-curricular office at Loyola AALHE_05_31_201215

16. Adapting the Ontology Step 1: Created a table with the headings of the ontological framework – We kept the same headings, but they can be changed Step 2: Reviewed the existing ontological framework and examples from Business and Nursing – Decision: What matches our practices at Loyola? Step 3: Populated our table with relevant information from the ontological framework Step 4: Determined what’s missing – Thinking of our assessment practices, what do we need to add? – Our additions are in red – Lesson learned: You know more about your assessment practices than you might think Step 5: Cultural alignment – Check for any terminology that needs to be changed – Have we included/highlighted what is important to Loyola? – Have we included/highlighted information that is particularly pertinent? AALHE_05_31_201216

17. About Loyola Loyola is a private, Jesuit Catholic university Important to include tenets of Jesuit education in our ontological framework – Example: emphasis on reflection Highlighted areas are our Core Curriculum components – Recently revised Core, currently revisiting Core assessment – Use ontological framework to better piece together and plan assessment AALHE_05_31_201217

18. Loyola University Chicago Ontology AALHE_05_31_201218

19. Loyola University Chicago Illustrative Components of Assessment: – Acquisition of ePortfolio data for graduating History majors assessment of critical thinking skills by History instructors using TaskStream. – Acquisition of reflection for undergraduate student assessment of global and societal awareness by Division of Student Development using TaskStream and focus groups. AALHE_05_31_201219

20. About the Center for Experiential Learning (CEL) Coordinates service learning, internships, student employment, undergraduate research, and ePortfolio program – Used service learning and internships for ontology Why CEL? – To demonstrate applicability to co-curricular offices – To help them organize their assessment plan The unintended jackpot – Accreditation! – Huge advantage: visualizing what is assessed, how, and by who will make evidence-gathering easier AALHE_05_31_201220

21. Ramble On: Loyola University Chicago Center for Experiential Learning AALHE_05_31_201221

22. Loyola University Chicago Center for Experiential Learning Illustrative Components of Assessment: – Acquisition of survey feedback for undergraduate student assessment of skills developed through service-learning course by program administration using Opinio web-based survey tool. – Acquisition of focus group feedback for undergraduate student assessment of skills developed through service-learning course by program administration using past service- learning participants in focus groups. AALHE_05_31_201222

23. Exercise: Creation of an Ontological Framework (Small Group) Each table has a blank ontology; work together to fill in 1 or 2 examples for each of the six sections: Information Periodic College Learning Outcomes Users System AALHE_05_31_201223

24. Exercise: Creation of an Ontological Framework (Large Group) We are now going to use your suggestions, based on your work with the group, to create another example ontology. We will then discuss various ways you might be able to use it to organize assessment processes, in consideration of different: Levels of assessment Types of information sought Time periods for conducting assessment Systems to collect information Stakeholders AALHE_05_31_201224

25. Example Ontology 4: Participants AALHE_05_31_201225

26. Recommendations for Future Use and Development Validate and expand the ontological framework through more user interviews. Use the framework for user education, requirements gathering, design specifications. Use an integration of platforms to meet user requirements. Maintain the framework for continuity and integrity of design. Adapt the framework for different philosophies of Learning Outcome Assessment. AALHE_05_31_201226

27. Questions? We will even take answers! AALHE_05_31_201227

28. Presenter Information Arkalgud Ramaprasad – Professor and Head, Information Decision Sciences, University of Illinois at Chicago; E-mail: prasad@uic.edu prasad@uic.edu Jen Sweet – Assessment Specialist, Office of Programs and Academic Assessment, University of Illinois at Chicago; E-mail: jmsweet@uic.edujmsweet@uic.edu Shannon Milligan – Coordinator of Assessment, Faculty Center for Ignatian Pedagogy, Loyola University Chicago; E-mail: smilligan@luc.edu smilligan@luc.edu AALHE_05_31_201228

29. Works Cited Gruber, T. R. (1995). Toward Principles for the Design of Ontologies Used for Knowledge Sharing. International Journal Human- Computer Studies 43(5-6), 907-928. AALHE_05_31_201229