1. Course Development In Engineering Education
Doris R. Brodeur
Massachusetts Institute of Technology
November 2005

2. TODAY’S OBJECTIVES
Define learning outcomes that integrate CDIO into your courses
Select teaching and learning methods that promote understanding and lead to the attainment of course learning outcomes
Describe features of design-build experiences (project-based learning)
Select assessment methods that provide evidence of understanding and attainment of learning outcomes
Apply guidelines for feedback and grading
Determine methods to evaluate the course and plan for continuous improvement

3. OUTLINE Defining Content Defining Learning Outcomes
Selecting Teaching and Learning Methods
Designing Design-Build Experiences
Selecting Assessment Methods
Grading and Giving Feedback
Evaluating the Course

4. Learning A LEARNING FRAMEWORK Curriculum Teaching and Learning
Assessment
Learning
Environment

5. A LEARNING DESIGN (Wiggins & McTighe, 1998)

6. DEFINING CONTENT AND LEARNING OUTCOMES
Curriculum
Teaching
and Learning
Assessment
Learning
Environment

7. DEFINING CONTENT (adapted from Wiggins & McTighe, 1998)
What is worthy and requiring of understanding?
Worth being familiar with
Important to know and do
Enduring understanding

8. DEFINING CONTENT Design Considerations Design Criteria
Alumni and industry perspectives
Professional standards
Program strengths
Faculty expertise
Design Criteria
Enduring ideas
Discipline-based work
Uncoverage of misconceptions
Will engage students
Attainable and measurable
Attitudes
Concepts
Skills
Procedures

9. LEARNING OUTCOMES (Gronlund, 2000)
Provide a focus for instruction
Provide guidelines for learning
Provide targets for assessment
Communicate expectations to learners
Convey instructional intent to others
Provide for evaluation of instruction

10. EFFECTIVE LEARNING OUTCOMES...
Focus on skills and abilities central to the discipline and based on professional standards
Are general enough to capture important learning, but clear and specific enough to be measurable
Focus on aspects of learning that will develop and endure but that can be assessed in some form now
Are student-focused
Focus on the learning resulting from an activity, or course, or program
Reflect the institution’s mission and the values it represents
Are in alignment at course, academic program, and institutional levels
Focus on important, non-trivial aspects of learning that are credible to the public

11. PAIR-AND-SHARE
Use the 8 criteria on the previous slide to evaluate the learning outcomes for your course
Give an example of an outcome that meets most of the criteria
If you did not bring your own outcomes, use the samples on the slides that follow

12. SAMPLE LEARNING OUTCOMES
Calculate lift and drag for blimps and airfoils.
Use lift and drag calculations to evaluate aerodynamic vehicle performance.
Design an internal structural configuration for simple trusses, beams, columns, and shafts in order to meet specified leading and deformation criteria.
Explain at a level understandable by a non-technical person how jet propulsion works.
Create models of inviscid, steady fluid flow over simple profiles and shapes.
Explain the division of the resistance of a ship into its components.
Distinguish emissions from combustion characteristics.
Create interactive 3-D models of products and environments using VRML.
Analyze and evaluate different planning techniques.

13. SAMPLE LEARNING OUTCOMES
Draw conclusions about the solvability of a system of linear equations using determinant and rank of a matrix.
Solve geometric problems concerning lines and planes using vectors.
Choose a basis for the plane or the space suitable for a specific geometric problem.
Judge if proposals to modification or proposals to new uses are a) possible, b) suitable, and c) outstanding.
Conduct a heat balance over a conventional steam power plant.
Analyze the relationships among the properties, structures, heat treatment, and load for metals.
Be aware of typical properties and applications for common kinds of alloys.
Analyze the factors which cause metals to disintegrate in humid environments.

14. INTEGRATING CDIO LEARNING OUTCOMES
Identify and define a system, its behavior, and its elements. (CDIO 2.3.1) (Knowledge)
Explain the links between engineering theory and practice. (CDIO 2.5.4) (Comprehension)
Use prototypes and test articles in design development. (CDIO 4.4.1) (Application)
Analyze the strengths and weaknesses of the design team. (CDIO 3.1.1) (Analysis)
Formulate solutions to problems using creativity and good decision making skills. (CDIO 3.1.2) (Synthesis)
Appraise operational systems and recommend improvements. (CDIO 4.6.4) (Evaluation)

15. INTEGRATING CDIO LEARNING OUTCOMES
Recognize the ethical issues involved in using people in scientific experiments. (CDIO 2.2.3) (Affective)
Demonstrate the courage to act on principle despite adversity. (CDIO 2.5.1) (Affective)
Commit to a personal program of lifelong learning and professional development. (CDIO 2.4.6) (Affective)
Use appropriate nonverbal communications, e.g., gestures, eye contact, poise. (CDIO 3.2.6) (Psychomotor)
Create interactive 3-D models of products and environments using lightweight metals. (CDIO 1.3) (Psychomotor)
Determine the stress and deformation states of structures using the appropriate physical tools and measures (CDIO 1.3) (Psychomotor)

16. SELECTING TEACHING AND LEARNING METHODS
Curriculum
Teaching
and Learning
Assessment
Learning
Environment

17. TEACHING AND LEARNING METHODS
Which methods do you use? Add “balloons” for other methods not mentioned here.
Pre-Class Readings & Homework
Concept Questions
Learning Outcomes
Cooperative Learning
Design-Build
Experiences

18. CONCEPT QUESTIONS Design Considerations Procedure
Focus on a single concept
Are not solvable, in the time given, by relying solely on equations
Reveal common difficulties with the concepts
Have more than one plausible answer based on typical misunderstandings
Procedure
Pose the question
Ask students to indicate their answers, e.g., flash cards, show of hands, automated system
If most have the correct answer, give a brief explanation, then move on.
Else, clarify the concept.
Take another poll of students’ answers

19. REAL-TIME RESPONSE USING PRS
Responses from sophomores

20. PRE-CLASS READINGS AND HOMEWORK
Reading and homework assignments are due prior to in-class discussion of material
Classroom interactions can focus on concepts
Encourages self-directed learning
Same amount of work for students, but front-loaded
Improved feedback time

21. BENEFITS AND CHALLENGES OF DBE
Formula Student
Solar aircraft
DBEs develop
design skills
DBEs are fun for students and teachers
DBEs strengthen engineering
science learning
Nano-satellites
Autonomous robots
DBEs are time-consuming and costly
DBEs create cross-disciplinary links
DBEs pose
teaching & assessment challenges
DBEs require new learning environments

22. DESIGN-BUILD EXPERIENCES
Design Considerations
Identify problems that raise the concepts and principles relevant to the content domain
Design an authentic task, i.e., one in which the thinking required is consistent with the thinking in the environment for which the learner is preparing
Design the task and environment to reflect the complexity of the environment in which learners will later function
Encourage testing ideas against alternative views and alternative contexts
Set realistic and assessable parameters
Provide opportunities for reflection on both the content learned and the learning process

23. SEQUENCING DB EXPERIENCES
The learning sequence is not necessarily the same as the sequence of the process in the professional engineering environment
Sequence for levels of complexity in problem structure, type of solution, number of people required, length of time

24. LEVEL OF COMPLEXITY #1
Structured problem
Known solution
Individual or group solution
Same problem for all students
Short time frame

25. Same problem for all teams Short time frame
LEVEL OF COMPLEXITY #2
Structured problem
Known solution
Team solution
Same problem for all teams
Short time frame
Autonomous Robots at MIT

26. Solution can be known or unknown Team solution
LEVEL OF COMPLEXITY #3
Complex problem
Solution can be known or unknown
Team solution
Different problem for each team
Several weeks or months
Third-Year Electronics Project
at Linkoping University

27. Single problem solved by multiple sub-teams
LEVEL OF COMPLEXITY #4
Complex problem
Unknown solution
Team solution
Single problem solved by multiple sub-teams
More than one term long
ARGOS Project at MIT

28. PAIR-AND-SHARE
Discuss the levels of complexity and sequencing of design-build experiences in your engineering programs

29. SELECTING ASSESSMENT METHODS
Learning
Curriculum
Teaching
and Learning
Assessment
Learning
Environment

30. LEARNING ASSESSMENT (Huba & Freed, 2000)
“The process of gathering and discussing information from multiple and diverse sources in order to develop a deep understanding of what students know, understand, and can do with their knowledge as a result of their educational experiences”
“The process culminates when assessment results are used to improve subsequent learning.”

31. BASIC ASSESSMENT PRINCIPLES
1 Assessment requires attention to outcomes and equally to experiences that lead to those outcomes.
2 Different types of learning objectives require different methods of assessment.
3 Teaching and assessment are intertwined.
4 Any assessment is only a sample.
5 Assessment works best when it is ongoing, not episodic.
6 There are trade-offs between authenticity and efficiency, i.e., the closer the tasks are to real-world experiences, the more time and resources they require.

32. DISCUSSION
Divide into groups with each group taking one assessment principle.
In your own words, explain the meaning of the assessment principle as it relates to your courses. Give examples, if possible.
Choose someone to report to the whole group.

33. SELECTING ASSESSMENT METHODS
Which methods do you use? Add “balloons” for other methods not mentioned here.
Performance Assessment
Product Review
Learning Outcomes
Oral Questions and Interviews
Journals and Portfolios

34. ORAL QUESTIONS AND INTERVIEWS
Design Considerations
Spontaneous questions put to students by experts
Students must think on their feet, draw upon relevant facts, theories, and/or perspectives, and speak in a coherent, organized fashion
Improves likelihood of an accurate assessment by its dynamic nature
Opportunity for instructors to learn more about misconceptions
Procedure
Give students the question(s) 30 minutes prior to exam
Conduct oral exam for 30 minutes
Use a rating sheet that lists relevant concepts and students’ level of understanding of each

35. PERFORMANCE ASSESSMENT
Students prepare and present a performance of a valued activity
Examples:
Oral presentations and technical briefings
Problem-solving
Group work
Teamwork skills
CRITICAL DESIGN REVIEW

36. PRODUCT REVIEW Design Considerations
A project whose focus is on the development of a tangible product
The product itself, the process, and quality of reasoning are all assessed
Procedure
Use rubrics that address specific criteria relevant to the product, process and quality of reasoning and scales with 3 to 5 levels of mastery

37. JOURNALS AND PORTFOLIOS
Design Considerations
Demonstrate what students have learned
Develop a sense of themselves as learners
Understand more deeply what they have learned and not learned
See relationships among learning experiences
Become more invested in their own learning

38. MATCHING ASSESSMENT WITH OUTCOMES

39. SAMPLE ASSESSMENT PLAN
Product Review
Built to specification
Time
Team Collaboration
Written Documentation
Reflective Journal
Formula Student Project at
Chalmers Institute of Technology, Gothenberg

40. SAMPLE ASSESSMENT PLAN
Product Review
Built to specification
Course completion
Time
Number of trials
Team Collaboration
Articulation of robot logic

41. FEEDBACK FROM STUDENTS
“When we observe students while they are learning and collect frequent feedback from them, we can learn much about how they learn and, more specifically, about how they respond to particular teaching practices.” (Angelo & Cross, 1993)
Examples:
Muddiest point cards
Punctuated lecture
3-2-1 summary
Electronic mail feedback

42. FEEDBACK TO STUDENTS Guidelines
“Providing feedback on students’ work is one of the most expensive components of their education, but it is often not an effective investment simply because it happens too slowly.” (Gibbs, 1999)
Guidelines
Provide frequent feedback throughout the term
Be prompt in returning assignments and exams
Make the feedback as specific as possible so that students know how to improve
Provide progress reports one or two times during the term
Correct errors, but avoid sarcasm and condescending comments

43. GUIDELINES FOR GRADING
Communicate the grading criteria to students when homework and exams are assigned so they know what is expected
Apply the grading criteria consistently across students, across assignments, and across multiple graders
Use rating scales to increase objectivity in grading, e.g., when evaluating presentations, journals, and written reports
Clarify with students how the overall subject grade is determined
Be sure that weighting systems reflect the proportion of time spent on each requirement

44. Instructor reflective memos
EVALUATING THE COURSE
Student ratings
Items related to learning objectives
Items related to CDIO
Items about teaching, learning, and assessment methods
Instructor reflective memos
Achievement of learning objectives
Integration of CDIO
Effective teaching and learning methods
Effective assessment methods
Plans for continuous improvement

45. REFERENCES
Angelo, T. A., & Cross, P. K. (1993). Classroom assessment techniques: A handbook for college teachers, 2d ed. San Francisco, CA: Jossey-Bass.
Biggs, J. (2003). Teaching for quality learning at university, 2nd ed. Buckingham, England: The Society for Research into Higher Education and Open University Press.
Donald, J. G. (2002). Learning to think: Disciplinary perspectives. San Francisco, CA: Jossey-Bass.
Field-tested learning assessment guide. Available at
Gibbs, G. (1999). Using assessment strategically to change the way students learn. In Brown, S. & Glasner, A. (Ed.). Assessment matters in higher education. Buckingham, England: The Society for Research into Higher Education and Open University Press.
Gronlund, N. E. (2000). How to write and use instructional objectives, 6th ed. Upper Saddle River, NJ: Merrill.
Huba, M. E., & Freed, J. E. (2000). Learner-centered assessment on college campuses. Boston, MA: Allyn and Bacon.
Wiggins, G., & McTighe, J. (1998). Understanding by design. Alexandria, VA: Association for Supervision and Curriculum Development.

46. 3 important ideas I learned about course development
SUMMARY REFLECTION
3 important ideas I learned about course development
2 concerns I still have about course development
1 step that I will take related to course development