A Streamlined, Quantitative, Web-based, Course-Level Assessment Process Jörg Moßbrucker Glenn Wrate Mike O’Donnell Milwaukee School of Engineering

Dr.-Ing. Joerg Mossbrucker, MSOE Overview Do what you say Prove it Measure what you do Implement changes to improve Say what you do

Dr.-Ing. Joerg Mossbrucker, MSOE Say what you do, do what you say Detailed course descriptions Detailed course descriptions –Course outline –Course prerequisites by topic and course –Course outcomes –Course topics –Day-by-day descriptions of lecture and lab sessions

Dr.-Ing. Joerg Mossbrucker, MSOE Example: Circuits I Prerequisite by topics: Differentiation of algebraic and trigonometric functions Differentiation of algebraic and trigonometric functions Solution of a system of linear equations using a calculator and/or computer Solution of a system of linear equations using a calculator and/or computer

Dr.-Ing. Joerg Mossbrucker, MSOE Example cont: Course Outcomes Upon successful completion of this course, the student will: Demonstrate problem solving skills Demonstrate problem solving skills Demonstrate a standard of expertise in the understanding of circuit laws and in the analysis of electrical circuit Demonstrate a standard of expertise in the understanding of circuit laws and in the analysis of electrical circuit Write and solve KCL and KVL equations using standard methods of circuit analysis for CD circuits Write and solve KCL and KVL equations using standard methods of circuit analysis for CD circuits Simplify networks using source transformations and Thevenin's/Norton's theorems Simplify networks using source transformations and Thevenin's/Norton's theorems Use the superposition principle in circuit analysis Use the superposition principle in circuit analysis Demonstrate calculator skills to solve circuit equations Demonstrate calculator skills to solve circuit equations Demonstate the ability to analyze DC circuits using SPICE Demonstate the ability to analyze DC circuits using SPICE Demonstrate circuit laboratory skills and perform DC measurements Demonstrate circuit laboratory skills and perform DC measurements

Dr.-Ing. Joerg Mossbrucker, MSOE Example cont: Course Topics DC circuit laws and concepts (10 classes) DC circuit laws and concepts (10 classes) DC circuit analysis (9 classes) DC circuit analysis (9 classes) Op-Amps (3 classes) Op-Amps (3 classes) Inductors/Capacitors (3 classes) Inductors/Capacitors (3 classes) Review (3 classes) Review (3 classes) Tests and quizzes (2 classes) Tests and quizzes (2 classes)

Dr.-Ing. Joerg Mossbrucker, MSOE Example cont. All the above data plus day-by-day description of lecture sessions and suggested laboratory sessions are stored in a MS Access database All the above data plus day-by-day description of lecture sessions and suggested laboratory sessions are stored in a MS Access database Web description is automatically created from database Web description is automatically created from database Course coordinator is responsible for data Course coordinator is responsible for data

Dr.-Ing. Joerg Mossbrucker, MSOE Measure what you do Course assessment measures IN/OUT Course assessment measures IN/OUT IN: Assessment of student performance data of prerequisite topics IN: Assessment of student performance data of prerequisite topics –Prerequ-exam, homework etc. OUT: Assessment of student performance data of course outcomes OUT: Assessment of student performance data of course outcomes –Final exam

Dr.-Ing. Joerg Mossbrucker, MSOE Assessment Assessment of student performance data generated out of Prerequ. Exam and Final Exam Assessment of student performance data generated out of Prerequ. Exam and Final Exam Must generate meaningful data Must generate meaningful data Must generate only a marginal increase in workload for faculty Must generate only a marginal increase in workload for faculty

Dr.-Ing. Joerg Mossbrucker, MSOE Assessment cont. How can we generate assessment data out of already existing (exam) data? How can we generate assessment data out of already existing (exam) data? –Two-dimensional grading scheme How can we generate meaningful data (i.e. is there a difference between 80% and 83%)? How can we generate meaningful data (i.e. is there a difference between 80% and 83%)? –Reduction of input data range

Dr.-Ing. Joerg Mossbrucker, MSOE Assessment Data Data reduction by grouping student performance into three categories: Data reduction by grouping student performance into three categories: –Adequate understanding (excellent to satisfactory, 100% to app. 75%) –Marginal understanding (below satisfactory or poor) –Inadequate understanding (failure) GoodJust passedClueless

Dr.-Ing. Joerg Mossbrucker, MSOE Generation of data Student 1 Student 2 Problem 1Problem 2Problem 3Grade 90%75%80% 50%60%75% 82% 63% A M I AAM MII

Dr.-Ing. Joerg Mossbrucker, MSOE Generation of data cont. A M I Outcome 1 Outcome 2 Outcome 3 Problem 1Problem 2Problem 3 * ** * AMI

Dr.-Ing. Joerg Mossbrucker, MSOE MS Excel Example

Dr.-Ing. Joerg Mossbrucker, MSOE Cont.

Dr.-Ing. Joerg Mossbrucker, MSOE Cont.

Dr.-Ing. Joerg Mossbrucker, MSOE Cont.

Dr.-Ing. Joerg Mossbrucker, MSOE Cont.

Dr.-Ing. Joerg Mossbrucker, MSOE Database Entry Web-based interface Web-based interface –Allows direct entry of assessment data –Enters data into MySQL database –Allows also entry of comments

Dr.-Ing. Joerg Mossbrucker, MSOE Data Flow Overview Access DB Workload Access DB Course Descr. Progress DB Online Course Descriptions MySQL DB for Courses/Year/ Quarters/Sections Web Data Input MySQL Database Access Database Adobe Acrobat Report in pdf Multiple Scripts In Access Reports/Queries

Dr.-Ing. Joerg Mossbrucker, MSOE Result Example

Dr.-Ing. Joerg Mossbrucker, MSOE Implement Changes to Improve What?How?When?Who? What? What? –Course outcomes with the lowest “Adequate” scores over one academic year –Collated and weighted results over all sections over one academic year reduces noise considerably

Dr.-Ing. Joerg Mossbrucker, MSOE Cont. How? How? –Change of course topics/focus –Change of lab sessions … –Removal of unnecessary course outcomes

Dr.-Ing. Joerg Mossbrucker, MSOE Cont. When and who? When and who? –At end of academic year assessment team assembles results and determines “critical” course outcomes requiring action –Assessment team focuses on “tracks” in EE, such as:  Digital, Analog, Circuits, Communications etc. –Course coordinators implement changes over summer

Dr.-Ing. Joerg Mossbrucker, MSOE Example Change of a 2-course sequence in circuits without lab to a 3-course sequence with lab

Dr.-Ing. Joerg Mossbrucker, MSOE Conclusions Assessment data generated directly from student performance data already available Assessment data generated directly from student performance data already available Data reduction by grouping into three categories Data reduction by grouping into three categories Instant graphical feedback to teaching faculty on performance of their section Instant graphical feedback to teaching faculty on performance of their section Simple, web-based interface to input assessment data Simple, web-based interface to input assessment data One-click calculation of assessment data of multiple sections/quarters/years One-click calculation of assessment data of multiple sections/quarters/years Assessment team determines annually action needed Assessment team determines annually action needed Course coordinators implement changes Course coordinators implement changes Data provides numerical feedback of changes Data provides numerical feedback of changes Program director has overview over course outcomes data for program outcomes Program director has overview over course outcomes data for program outcomes

Dr.-Ing. Joerg Mossbrucker, MSOE Issues (past and present) No. of prerequisites usually small -> noise exceptionally large No. of prerequisites usually small -> noise exceptionally large –Common prerequisite exam for all sections of a course determined by course coordinator Different sections behave very differently Different sections behave very differently –No common final exam –Instructor understanding of performance –Section/time/student etc. dependent –Adds to overall noise

Dr.-Ing. Joerg Mossbrucker, MSOE Cont. Reduction to a very small set of outcomes: Reduction to a very small set of outcomes: breadth vs. depth breadth vs. depth

?

Dr.-Ing. Joerg Mossbrucker, MSOE Overview Things I need: Things I need: General process (do what you say, measure what you do, improve) General process (do what you say, measure what you do, improve) Do what you say Do what you say –Course descriptions –Course outcomes –Course topics –Role of the course coordinator –Example of course on-line –Where is the data stored Measure what you do Measure what you do –Different methods of assessment –In/Out measures –Advantages/disadvantages –Numerical/non-numerical data –Common exams yes/no –How can we use data already collected for a class? –Is there a difference between 85% and 80%? If so how much –-> reduction of in data (adequate/marginal/inadequate) –What do these mean? –How to get it out of final pre-requ? –Multiple sources for a single course outcome from different questions –Computed automatically in Excel (example) –How do we get the computed data and results –Databases used –Examples of results Improve Improve –Different possibilities how to improve –When to improve –Who decides when what –Assessment process Issues Issues –Honesty –Integrity of the course coordinator –Noise –Still too much data