1. Presented by: Mary Catharine Lennon Senior Research Analyst Higher Education Quality Council of Ontario Presented to:State Higher Education Executive Authority August 9, 2012 Chicago, Illinois Ontario’s experience in AHELO’s Civil Engineering Strand Informing the Future of Higher Education

2. Canada is 1 of 9 countries participating in Engineering Australia Canada Columbia Egypt Japan Mexico Russia Slovakia United Arab Emirates 2 Informing the Future of Higher Education

3. HEQCO is implementing AHELO on behalf of MTCU 3 Informing the Future of Higher Education Council of Ministers of Education (CMEC) is the Canadian representative to OECD’s Education Branch Ministry of Training, Colleges and Universities (MTCU) is a member of CMEC HEQCO is an agency of MTCU

4. What is HEQCO? “The objective of the Council is to assist the Minister in improving all aspects of the postsecondary sector, including improving the quality of education provided in the sector, access to post-secondary education and accountability of post-secondary institutions.”

5. Ontario activities timeline 5 Informing the Future of Higher Education July 2011 HEQCO joined project October Recruited institutions November Hired lead scorer December Vetted and adapted the documents January 2012 IC training and ethics February Recruitment, IT preparations and TA training March Student testing April Faculty and institutional survey collection May Scoring

6. AHELO’s Civil Engineering strand Pilot managed by the Australian Council on Educational Research (ACER) Expected learning outcomes were developed through Tuning process Assessment tool developed by ACER, Japan’s National Institute for Educational Policy Research (NIER), the European and Global Engineering Education network (EUGENE), in collaboration with international Engineering education experts 6 Informing the Future of Higher Education

7. The Engineering Test framework Understanding and application of design methodologies to meet requirements Practical competencies required to solve problems, conducting investigations, and designing engineering devices and processes. Covers non-technical elements of civil engineering practice like professional ethics, responsibilities and the impact of engineering solutions in a global economic, societal and environmental context Using analytical methods to identify, formulate and solve problems Effective communication and awareness of the wider civil engineering context Knowledge and understanding of underlying scientific and mathematical principles – general sciences; materials and construction; structural engineering; geotechnical engineering; hydraulic engineering and urban and rural planning 7 Informing the Future of Higher Education Engineering design Engineering practice Engineering analysis Generic engineering skills Basic engineering skills

8. The test Students are provided 2 of 3 performance tasks, each requiring 30 minutes to complete Students are provided 20 of 40 multiple choice questions, together requiring 30 minutes to complete Students complete the context survey 8 Informing the Future of Higher Education

9. How it works Online tests and contextual questionnaires are completed by students submitted directly to ACER for evaluation Short written responses are graded by trained Ontario-based Civil Engineering faculty and submitted to ACER All analysis of Civil Engineering strand occurs at ACER ACER provides result of Engineering strand to OECD 9 Informing the Future of Higher Education

10. 9 of 10 Ontario universities participated 10 Informing the Future of Higher Education

11. Student recruitment Having specific and small population simplified recruitment Recruitment examples: emails, websites, classroom announcements, meetings of class and civil engineering student society representatives 11 Informing the Future of Higher Education

12. Student incentives Limited funds and lots of creativity! Incentive examples: class pictures, voucher for formal end of term dinner, donation to the engineering societies, individual gift cards, increased amount on gift card if certain percentage of participation achieved, food after exam, raffles for ipad3 12 Informing the Future of Higher Education

13. The results Ontario Institutions Students who wrote test/population Faculty who completed survey/population Institution 161/77 = 79.2%2/18 = 11.1% Institution 223/34 = 67.6%18/20 = 90% Institution 387/137 = 63.5%17/18 = 94.4% Institution 457/100 = 57%16/18 = 88.9% Institution 565/114 = 57%25/34 = 73.5% Institution 643/90 = 47.8%27/40 = 67.5% Institution 745/77 = 58.4%19/32 = 59.4% Institution 826/44 = 59.1%15/18 = 83.3% Institution 935/56 = 62.5% *16/17 = 94.1% Total442/729 = 60.6155/215 = 72% 13 Informing the Future of Higher Education *Note: Institution 9 had 62.5% of their students write the test, but only 5.4% were electronically captured, altering the final numbers for analysis to 56%*

14. AHELO Transmission of Results ‘Institution Report’: Institutions receive a report detailing the performance of the cohort of students from their individual institution ‘Jurisdictional report’: will outline the performance of Ontario’s Civil Engineering programs ‘Inter-jurisdictional report’: Australia and Canada have agreed to share data for comparative purposes ‘International Report’: The OECD will provide a final report reviewing the feasibility study 14 Informing the Future of Higher Education

15. Challenges and successes Challenges Successes 15 Informing the Future of Higher Education Institutional Research Ethics Board approval made it data linking impossible. Technical issues at one institution Considerable ‘buy-in’ from all levels Discipline-strand simplified sampling and recruitment Weekly Institutional Coordinator teleconferences were extremely useful Lessons Learned Get to institutions early in the academic year Allow time for Ethics approval Ensure adequate IT support

16. Find out more! Visit heqco.ca 16 Informing the Future of Higher Education