Interpreting Data & “Closing the Loop” Margaret Kasimatis, PhD VP for Academic Planning & Effectiveness

ARTICULATE MISSION/ GOALS IDENTIFY SPECIFIC OUTCOMES DETERMINE PRACTICES USED TO ACHIEVE OUTCOMES GATHER EVIDENCE REVIEW & INTERPRET RESULTS RECOMMEND ACTIONS Location in Assessment Cycle

Step #1: Organize Your Data Assemble the following in one place: –All data or possible sources of data –List of learning outcomes/research questions –Curriculum, experiences, practices Map data sources to outcomes –Outcomes x Measures Map Map results to outcomes –Present in “user-friendly” way

LEARNING OUTCOMES ASSESSMENT MEASURES Alumni Survey National Survey of Student Engagement (NSSE) Evaluation of Clinic Presentations Evaluation of First-Year Presentations Rubric Evaluation of Upper-level Coursework Ability to apply knowledge in math, science, and engineering XX Ability to design a system, component, or process to meet desired needs X Ability to identify, formulate, and solve engineering problems XXX Ability to communicate effectively XXXXX Ability to use techniques, skills, and modern engineering tools necessary for engineering practice XX Ability to function on multidisciplinary teams XXX Recognition of need for and ability to engage in lifelong learning XX Understanding of professional and ethical responsibility XX Broad education necessary to understand the impact of engineering solutions in a global & societal context XX Mapping Data Sources to Outcomes: Engineering Example

It’s best to present results one outcome at a time –It’s not recommended to present results primarily by measure Make sure to present the data in a user- friendly way (i.e., in a way that faculty in your department are comfortable with) Presenting Results

ACADEMIC CHALLENGE: LMUJesuit Master’s NSSE Total How often did you work harder than you thought you could to meet an instructor's standards or expectations? (Percent of students who said ‘Often’ or ‘Very Often) FY 62%55%54%52% SR 54%59%60%56% How much time did you spend preparing for class (e.g. studying, reading, writing, doing homework, rehearsing, other academic activities) ? (Percent of students who reported spending 11 or more hours per week) FY 65%68%49%56% SR 40%58%52%54% How often did you come to class without completing readings or assignments? (Percent of students who said ‘Often’ or ‘Very Often’) FY 21%17% 19% SR 27%23%20%23% Describe the extent in the current year to which exams challenged you to do your best work. (mean response on a 7-point scale) FY SR

Step #2: Interpret the Data What type of criterion? What is “significant?” Are the findings reliable/valid?

What type of criterion? Norm-referenced –Need appropriate comparison group Avoid percentile rankings –Good for more nebulous findings Absolute standard –Usually more appropriate for performance-based measures

What is significant? Important to test when making comparisons –Numbers that look different may not really be different However, just because a difference is significant doesn’t mean it’s important

Are findings reliable/valid? Can we believe student self-reports? Are standardized measures more valid? Single measure vs. multiple measures

Step #3: Make Recommendations Start by considering –Where in the curriculum are outcomes addressed? And at what level? Refer to (or create) curriculum map, or Inventory for particular outcome –How? What practices/techniques are used? –Where are the gaps? –What can be changed? In the short-term? In the long-term?

Step #3: Make Recommendations Possible types of recommendations: –Changes to pedagogy –Changes to curriculum/programming –Allocation of resources Keep in mind that you can’t fix everything at once – so start small It’s usually better to modify than add Be as specific/concrete as possible

PROPOSED ACTIONIMPLEMENTATION Document (with grades) the importance of presentation clarity Essay questions added to exams Engineering Curriculum Committee developed writing rubric to be used to evaluate project report Make common writing style guidelines available to faculty and students Department selected Hacker, A Writer’s Reference Hacker, A Writer’s Reference required in E4 Hold Clinic team members accountable for peer review of team reports Clinic advisors distributed peer review sheets to Clinic team members. Students completed the peer review sheets when they reviewed fellow team members’ portions of Clinic report. Step #4: Implement & Document Recommend Actions Example

Example of Assessment Loop Step #1: Articulate Goals/Outcomes –Goal: Graduates will possess strong communication skills Outcome: Graduates will be able to write clearly and effectively Step #2: Determine practices used to achieve outcome –Survey of required writing experiences in Core and academic programs

Sem #BIOCHEMCSENGMATHPHYSH/SS One4 papers TwoReport in E4Tech report in Phys 28 4 papers Three1-2 papers in Bio lab reports Lab notebook & paper in Chem 53 Tech report in Phys 53 Variable Amount Of Writing Four4 lab reports 1-2 papers in Bio 108 Essays/paper in Chem 52 Lab notebook Exp abstracts Design doc, Report, & User guide in CS 121 Tech report in Phys 54 Term paper in Phys 52 Five1-2 papers in Bio lab reports 3 reports + paper in 109 Exp abstracts Abstracts in CS 110 Proposal & mid-year report OR Six1-2 papers in Bio 113 Seminar paper Project paper in Chem 114 Essay in CS 131 Clinic reportTech report in Phys 134 IE Paper SevenProposal 3 Lab reports Letter of Intro Prog report Proposal + midyear report Proposal + midyear report Seminar Researc h Paper EightThesis Clinic report Thesis or Clin. report Thesis or Clin. report

Example of Assessment Loop, cont’d Step #3: Conduct measurements –Higher Education Research Institute (HERI) College Student Survey (CSS) –National Survey of Student Engagement –HMC Alumni Survey

Step #4: Review & interpret results –Faculty committees Assessment Curriculum Teaching & Learning –Full faculty - workshop Example of Assessment Loop, cont’d

Writing Skills Frequency data from HERI, NSSE & Alumni Survey HMC Seniors 4-Yr Non- Sectarian Cohort Cohort HMC Seniors Liberal Arts AITU Seniors

Seniors’ Perceptions of Writing Skills by Major (from NSSE)

Alumni Perceptions of Writing Emphasis by Major (alumni graduating in last 10 years)

Sem #BIOCHEMCSENGMATHPHYSH/SS One4 papers TwoReport in E4Tech report in Phys 28 4 papers Three1-2 papers in Bio lab reports Lab notebook & paper in Chem 53 Tech report in Phys 53 Variable Amount Of Writing Four4 lab reports 1-2 papers in Bio 108 Essays/paper in Chem 52 Lab notebook Exp abstracts Design doc, Report, & User guide in CS 121 Tech report in Phys 54 Term paper in Phys 52 Five1-2 papers in Bio lab reports 3 reports + paper in 109 Exp abstracts Abstracts in CS 110 Proposal & mid-year report OR Six1-2 papers in Bio 113 Seminar paper Project paper in Chem 114 Essay in CS 131 Clinic reportTech report in Phys 134 IE Paper SevenProposal 3 Lab reports Letter of Intro Prog report Proposal + midyear report Proposal + midyear report Seminar Researc h Paper EightThesis Clinic report Thesis or Clin. report Thesis or Clin. report

Step #5: Recommend actions –Universal writing handbook –Faculty workshop on writing instruction –Revision to Core Curriculum Example of Assessment Loop, cont’d

MULTIPLE CURRICULAR OBJECTIVES IN HORIZONTALLY INTEGRATED CORE CURRICULUM Content disciplinary knowledge discipline-related techniques Skills oral and written communication critical thinking teamwork and collaboration project management leadership Context relationship of science or technology with contemporary society individual and comparative explorations of cultural identity

QUESTIONS?