1. EVALUATING THE IMPLEMENTATION OF CDIO PROGRAMS AT SINGAPORE POLYTECHNIC: THE FIRST YEAR
Helene Leong-Wee Kwee Huay, Dennis Sale, Cheryl Wee Soon Peng
Department of Educational Development

2. Implementation of CDIO in SP
October 2006
13 engineering programmes revised and restructured
the School of Architecture and Built Environment,
the School of Chemical and Life Sciences,
the School of Electrical and Electronic Engineering, and
the School of Mechanical and Aeronautical Engineering.

3. Revisions The revised programmes were implemented in April 2008.
Revisions include:
Integration of the CDIO skills
Personal skills and attitudes
Communication
Teamwork
Introduction to Engineering (semester/year long)
Assessment schemes

4. Purpose of Evaluation
To provide a structured research driven approach to monitor and review the implementation of the CDIO Framework
“Evaluation is the process of delineating, obtaining and providing information useful for making decisions and judgements about educational programmes and curriculum.”
(Kemmis, 1989, p.117)

5. Research Questions
Were the learning outcomes, learning activities and assessments aligned?
Were the learning of the courses integrated in the Introduction to Engineering Module?
How has the integration of the CDIO skills into the syllabuses impacted the students?
What were the lecturers’ perception of the curriculum changes and their impact on students’ competence in the selected CDIO skills and interest in subject?

6. Data Collection Methods
Examination of a range of curriculum materials (e.g., course documents, module documents, learning plans, schemes of assessment, assessment items)
Student questionnaires
Student Blogs
Focus group interviews with students and lecturers teaching the CDIO programmes
Observation of selected lessons (e.g., those incorporating activities related to selected CDIO skills)

7. Examination of a Range Of Curriculum Materials
Examination of syllabuses, learning plans and activities, schemes of assessment and assessment items.
The aim was to ensure that the curriculum materials met the conditions of an aligned curriculum (e.g., Biggs, 1999) and were consistent with relevant CDIO standards (Crawley et al, 2007).

8. Student Questionnaire
Administered to all the students online at the end of each semester (Sept 08 & Feb 09).
45% response rate (549 participants in Sem 1 and 647 participants in Sem 2)
Some key differences in the questions posed between the two administrations (e.g., Sept 08 and Feb 09).
First semester, interest was in the infusion of the selected CDIO skills in the first questionnaire.
Second semester, the focus was more on establishing the extent of application of the skills.

9. Student Co-Participants and Blog
Student “co-participants” blogged their experiences of the lessons taught.
47 student co-participants were involved semester 1 and 55 for semester 2, with representation across schools
Briefed on the research purpose and their role and responsibilities

10. Responsibilities of Student Co-Participants
Chat to classmates and identify some broad experiences relating to learning the selected CDIO skills and the teaching approaches used
Make personal notes and/or blog their experiences
Meet with the researchers at least once a semester for group sharing

11. Student Co-Participants and Blog
Students were presented with specific questions relating to their experience of lessons taught, and asked to provide their responses with examples to illustrate where possible.
A useful and novel way to help build rapport with the students, encouraging more authentic and open communication.

12. Student Blogs
The semester will be ending in a few weeks time. List down the skills which you think you have developed through the Introduction to Engineering module in this semester. (E.g.: ability to troubleshoot problems, teamwork, etc)

13. What the students blogged…..
In Introduction to Engineering, I learned how things around me are manufactured. That’s why I appreciate every little tool and object around me. I also developed skills like troubleshooting problems, how the work piece is to be clamped such that the workpiece is supported to undergo cutting.
Besides that, teamwork is another skill I developed. I noticed the strength of every member that I can learn from. Without good teamwork, I could not possibly develop such a workpiece. One of the most valuable skill I gained is brainstorming for using the right tool for the right purpose and thinking of developing the final product using the simplest manufacturing method.
Year 1 Mechanical and Aeronautical Engineering student

14. What the students blogged …..
In my opinion, the best skill that I have developed through the module is teamwork and leadership skills. This module actually gave me an opportunity to train and develop my teamwork skills. As an introvert person, I did not really bother to be a leader before. However due to this module I have learnt how to gather the group member’s attention, assign work, manage conflicts and etc.. 
Other than the skills that I have mentioned above this module taught me some practical applications of the knowledge that I learnt during my lectures. The module helped me to see, integrate, design and implement the circuit to satisfy the required function.  
Year 1 Electrical and Electronic Engineering student

15. Focus Groups (Lecturers)
1-2 hours
Twenty-one lecturers, representing all schools, participated in 4 focus group sessions
What have you been involved in doing, in terms of CDIO implementation?
What have you specifically done and how?
What is your perception of its impact on student learning, based on your experience with student groups?

16. Focus Groups (Students)
1-2 hours
Asked about their experience relating to aspects of CDIO.
No ‘right’ or ‘better’ answers, only the best representation of their experience
The typical interview process for each area of interest involved:
A standard opening scenario is presented to the students
Encouragement to students to explain and provide examples
Clarification of meaning and checking understanding where appropriate

17. Observation of Lessons
To obtain
an ethnographic insight into what was actually occurring in the classroom
what might be useful for enhancing understanding of how students experience learning activities related to the selected CDIO skills.

18. Observation process
The classes to be observed are mutually agreed by lecturers and EDU staff
The lesson must incorporate a component relating to the teaching/assessment of a CDIO skill area
There is opportunity to ask student questions (at the end of the observation) pertaining to their experience of the learning activity and CDIO skill areas

19. Key Findings – Curriculum Documents and Materials
Some courses needed significant revision :
writing of learning outcomes generally (e.g., rationalization, performance focus, clarity of intent, etc)
Infusion of selected CDIO skills appropriately.
Designing of key learning tasks and assessment activities (including the scoring systems)

20. Key Findings – Student Questionnaire
Nearly 50% agreed and 25% strongly agreed, in favour of the implementation of CDIO.
Semester 1: mean scores ranged from 3.73 to 4.03, with an overall mean of 3.90 (SD = 0.90) .
Semester 2: mean scores ranged from 3.74 to 4.10, with an overall mean of 3.85 (SD = 0.93).
The Cronbach alphas showed high internal consistency in student responses.

21. Key Findings – Student blog and focus groups
Students perceived the importance of the CDIO skills as a valuable part of the curriculum
The explicit teaching of the selected skills seems to vary from module to module and across lecturers
Authentic learning opportunities for such skills to be fostered created
Specific real world tasks linked theory to practice and made the learning experience more meaningful and interesting

22. Key Findings – Lecturer Focus Groups
Lecturers saw the relevance of the CDIO framework and have experienced the positive impact of the changes made to the curriculum on student attention and interest.
More motivated students coped better with the more challenging integrated learning tasks, the less competent students required more help and time.
There was a range of practices concerning the explicit teaching of the selected CDIO skills.
There has been an increase in workload, resulting from the preparation and assessment involved.

23. Key Findings – Observation of Lessons
Some learning tasks provided opportunities for the development of types of thinking, teamworking and communication skills.
Students were generally positive about the learning activities, some needing more support than others.
The actual skills and what were involved in developing them were not seen to be taught explicitly in most observations.

24. Recommendations
Differentiate the learning tasks and teaching approaches to accommodate students’ different learning abilities
Continue to encourage the explicit development of CDIO skills
Address increase in lecturers’ workload
Provide necessary training to expand lecturers’ expertise

25. Conclusions
The eclectic methodology employed in the evaluation has provided a range of insights into aspects of teaching and learning relating to both CDIO implementation and the student experience of teaching and learning in SP
Selected CDIO skills implemented so far have been sufficiently well received by both students and lecturers
The next focus is on verifying and extending the range of pedagogic and assessment practices necessary for effectively meeting the range of CDIO standards.

26. Student Blogs
Samples

27. What the students blogged…..
Having gone through 2 semesters, how would you rate (1-5) your interest in your field of engineering? (1 being Not Interested and 5 being Very Interested) Briefly explain your rating.
Rate 5. What I love most in this module was the project. We were allowed to work as group and be independent. We have to complete the project in group. During the process of doing the project, we were like small engineers. Although our skills were far from the Professional engineers, we experienced many situation which allow us to improve and find solution. Problem-solving is what engineers should do.
Year 1 Architecture and Built Environment student

28. What the students blogged…..
Having gone through 2 semesters, how would you rate (1-5) your interest in your field of engineering? (1 being Not Interested and 5 being Very Interested) Briefly explain your rating.
I would rate it 4. I feel that it is very fun and exciting learning things that are related to products that can be seen in our everyday lives. Very few modules show the link between studying and the real engineering world. However, I would grade it 5, if the products to be manufactured are chosen by us.
Year 1 Mechanical and Aeronautical Engineering student

29. What the students blogged…..
Having gone through 2 semesters, how would you rate (1-5) your interest in your field of engineering? (1 being Not Interested and 5 being Very Interested) Briefly explain your rating.
I rate 5 for my interest in the field of engineering. Well before joining the school I was thinking of which course to join. I did not really have any specific course but since I like math and science, I decided to join the engineering course. At this moment, I am quite satisfied with my choice and gaining more interest in engineering although it’s hard to study and learn something new. Due to the knowledge that I have learnt, now I can see and understand some practical applications in our world which is fun.
Year 1 Electrical and Electronic Engineering student

30. Thank you