1. A Streamlined, Quantitative, Web-based, Course-Level Assessment Process Jörg Moßbrucker Glenn Wrate Mike O’Donnell Milwaukee School of Engineering

2. Dr.-Ing. Joerg Mossbrucker, MSOE 20062 Overview Do what you say Prove it Measure what you do Implement changes to improve Say what you do

3. Dr.-Ing. Joerg Mossbrucker, MSOE 20063 Say what you do, do what you say Detailed course descriptions Detailed course descriptions –Course outline –Course prerequisites by topic and course –Course outcomes –Course topics –Day-by-day descriptions of lecture and lab sessions

4. Dr.-Ing. Joerg Mossbrucker, MSOE 20064 Example: Circuits I Prerequisite by topics: Differentiation of algebraic and trigonometric functions Differentiation of algebraic and trigonometric functions Solution of a system of linear equations using a calculator and/or computer Solution of a system of linear equations using a calculator and/or computer

5. Dr.-Ing. Joerg Mossbrucker, MSOE 20065 Example cont: Course Outcomes Upon successful completion of this course, the student will: Demonstrate problem solving skills Demonstrate problem solving skills Demonstrate a standard of expertise in the understanding of circuit laws and in the analysis of electrical circuit Demonstrate a standard of expertise in the understanding of circuit laws and in the analysis of electrical circuit Write and solve KCL and KVL equations using standard methods of circuit analysis for CD circuits Write and solve KCL and KVL equations using standard methods of circuit analysis for CD circuits Simplify networks using source transformations and Thevenin's/Norton's theorems Simplify networks using source transformations and Thevenin's/Norton's theorems Use the superposition principle in circuit analysis Use the superposition principle in circuit analysis Demonstrate calculator skills to solve circuit equations Demonstrate calculator skills to solve circuit equations Demonstate the ability to analyze DC circuits using SPICE Demonstate the ability to analyze DC circuits using SPICE Demonstrate circuit laboratory skills and perform DC measurements Demonstrate circuit laboratory skills and perform DC measurements

6. Dr.-Ing. Joerg Mossbrucker, MSOE 20066 Example cont: Course Topics DC circuit laws and concepts (10 classes) DC circuit laws and concepts (10 classes) DC circuit analysis (9 classes) DC circuit analysis (9 classes) Op-Amps (3 classes) Op-Amps (3 classes) Inductors/Capacitors (3 classes) Inductors/Capacitors (3 classes) Review (3 classes) Review (3 classes) Tests and quizzes (2 classes) Tests and quizzes (2 classes)

7. Dr.-Ing. Joerg Mossbrucker, MSOE 20067 Example cont. All the above data plus day-by-day description of lecture sessions and suggested laboratory sessions are stored in a MS Access database All the above data plus day-by-day description of lecture sessions and suggested laboratory sessions are stored in a MS Access database Web description is automatically created from database Web description is automatically created from database Course coordinator is responsible for data Course coordinator is responsible for data

8. Dr.-Ing. Joerg Mossbrucker, MSOE 20068 Measure what you do Course assessment measures IN/OUT Course assessment measures IN/OUT IN: Assessment of student performance data of prerequisite topics IN: Assessment of student performance data of prerequisite topics –Prerequ-exam, homework etc. OUT: Assessment of student performance data of course outcomes OUT: Assessment of student performance data of course outcomes –Final exam

9. Dr.-Ing. Joerg Mossbrucker, MSOE 20069 Assessment Assessment of student performance data generated out of Prerequ. Exam and Final Exam Assessment of student performance data generated out of Prerequ. Exam and Final Exam Must generate meaningful data Must generate meaningful data Must generate only a marginal increase in workload for faculty Must generate only a marginal increase in workload for faculty

10. Dr.-Ing. Joerg Mossbrucker, MSOE 200610 Assessment cont. How can we generate assessment data out of already existing (exam) data? How can we generate assessment data out of already existing (exam) data? –Two-dimensional grading scheme How can we generate meaningful data (i.e. is there a difference between 80% and 83%)? How can we generate meaningful data (i.e. is there a difference between 80% and 83%)? –Reduction of input data range

11. Dr.-Ing. Joerg Mossbrucker, MSOE 200611 Assessment Data Data reduction by grouping student performance into three categories: Data reduction by grouping student performance into three categories: –Adequate understanding (excellent to satisfactory, 100% to app. 75%) –Marginal understanding (below satisfactory or poor) –Inadequate understanding (failure) GoodJust passedClueless

12. Dr.-Ing. Joerg Mossbrucker, MSOE 200612 Generation of data Student 1 Student 2 Problem 1Problem 2Problem 3Grade 90%75%80% 50%60%75% 82% 63% A M I AAM MII 1 0 1 0 1 1 1 1 0

13. Dr.-Ing. Joerg Mossbrucker, MSOE 200613 Generation of data cont. A M I 1 0 1 0 1 1 1 1 0 Outcome 1 Outcome 2 Outcome 3 Problem 1Problem 2Problem 3 * ** * AMI 101 121 110

14. Dr.-Ing. Joerg Mossbrucker, MSOE 200614 MS Excel Example

15. Dr.-Ing. Joerg Mossbrucker, MSOE 200615 Cont.

16. Dr.-Ing. Joerg Mossbrucker, MSOE 200616 Cont.

17. Dr.-Ing. Joerg Mossbrucker, MSOE 200617 Cont.

18. Dr.-Ing. Joerg Mossbrucker, MSOE 200618 Cont.

19. Dr.-Ing. Joerg Mossbrucker, MSOE 200619 Database Entry Web-based interface Web-based interface –Allows direct entry of assessment data –Enters data into MySQL database –Allows also entry of comments

20. Dr.-Ing. Joerg Mossbrucker, MSOE 200620 Data Flow Overview Access DB Workload Access DB Course Descr. Progress DB Online Course Descriptions MySQL DB for Courses/Year/ Quarters/Sections Web Data Input MySQL Database Access Database Adobe Acrobat Report in pdf Multiple Scripts In Access Reports/Queries

21. Dr.-Ing. Joerg Mossbrucker, MSOE 200621 Result Example

22. Dr.-Ing. Joerg Mossbrucker, MSOE 200622 Implement Changes to Improve What?How?When?Who? What? What? –Course outcomes with the lowest “Adequate” scores over one academic year –Collated and weighted results over all sections over one academic year reduces noise considerably

23. Dr.-Ing. Joerg Mossbrucker, MSOE 200623 Cont. How? How? –Change of course topics/focus –Change of lab sessions … –Removal of unnecessary course outcomes

24. Dr.-Ing. Joerg Mossbrucker, MSOE 200624 Cont. When and who? When and who? –At end of academic year assessment team assembles results and determines “critical” course outcomes requiring action –Assessment team focuses on “tracks” in EE, such as:  Digital, Analog, Circuits, Communications etc. –Course coordinators implement changes over summer

25. Dr.-Ing. Joerg Mossbrucker, MSOE 200625 Example Change of a 2-course sequence in circuits without lab to a 3-course sequence with lab

26. Dr.-Ing. Joerg Mossbrucker, MSOE 200626 Conclusions Assessment data generated directly from student performance data already available Assessment data generated directly from student performance data already available Data reduction by grouping into three categories Data reduction by grouping into three categories Instant graphical feedback to teaching faculty on performance of their section Instant graphical feedback to teaching faculty on performance of their section Simple, web-based interface to input assessment data Simple, web-based interface to input assessment data One-click calculation of assessment data of multiple sections/quarters/years One-click calculation of assessment data of multiple sections/quarters/years Assessment team determines annually action needed Assessment team determines annually action needed Course coordinators implement changes Course coordinators implement changes Data provides numerical feedback of changes Data provides numerical feedback of changes Program director has overview over course outcomes data for program outcomes Program director has overview over course outcomes data for program outcomes

27. Dr.-Ing. Joerg Mossbrucker, MSOE 200627 Issues (past and present) No. of prerequisites usually small -> noise exceptionally large No. of prerequisites usually small -> noise exceptionally large –Common prerequisite exam for all sections of a course determined by course coordinator Different sections behave very differently Different sections behave very differently –No common final exam –Instructor understanding of performance –Section/time/student etc. dependent –Adds to overall noise

28. Dr.-Ing. Joerg Mossbrucker, MSOE 200628 Cont. Reduction to a very small set of outcomes: Reduction to a very small set of outcomes: breadth vs. depth breadth vs. depth

29. ?

30. Dr.-Ing. Joerg Mossbrucker, MSOE 200630 Overview Things I need: Things I need: General process (do what you say, measure what you do, improve) General process (do what you say, measure what you do, improve) Do what you say Do what you say –Course descriptions –Course outcomes –Course topics –Role of the course coordinator –Example of course on-line –Where is the data stored Measure what you do Measure what you do –Different methods of assessment –In/Out measures –Advantages/disadvantages –Numerical/non-numerical data –Common exams yes/no –How can we use data already collected for a class? –Is there a difference between 85% and 80%? If so how much –-> reduction of in data (adequate/marginal/inadequate) –What do these mean? –How to get it out of final pre-requ? –Multiple sources for a single course outcome from different questions –Computed automatically in Excel (example) –How do we get the computed data and results –Databases used –Examples of results Improve Improve –Different possibilities how to improve –When to improve –Who decides when what –Assessment process Issues Issues –Honesty –Integrity of the course coordinator –Noise –Still too much data