Going Beyond Grades In Assessing Student Learning Slides: www.zogotech.com/innovations Dr. Blaine Bennett Richard (Dick) Whipple Michael Taft

Outline About SWTJC Review Traditional Analysis Learning Outcomes Completion, Grades, Placement Scores, etc. Learning Outcomes Find It Assess It Improve It - Close the Loop Questions Hector: Welcome and Explain who we are, our recent growth, our SACS issues and ATD. BB

About SWTJC Large service area HSI ~ 6,000 Enrollment 11 Counties (16,769 square miles) Rural HSI ~ 6,000 Enrollment 70+% FTIC need remediation Achieving the Dream Developmental Education Gateway Courses Culture of Evidence Hector: Welcome and Explain who we are, our recent growth, our SACS issues and ATD. BB

Background: SWTJC Student Data Warehouse Blaine Presents: Lots of data. Give opportunity to discuss other data elements (quantitative + qualitative) BB -> DW

Traditional Learning Analysis Grades + GPA 2.0 = LEARNING “Those were the days!” DWhipple: Discuss the old way of determining if students are learning; and new state mandates (access to success) DW

Previous Work Course Completion DW

Previous Work Gateway Course Completion Above: Shows the correlation between gateway course completion and graduation. Within 4 terms of completing College Level math (in green) 30% of students have graduated (almost double that of the reading and writing gateway courses!) DW

Previous Work Test scores Looking at class and test history, impossible to see where student is in dev ed sequence DW 8

Previous Work Test Scores + Grades = Subject Levels Incorporate test scores + grades into single “level” Easy to understand, analyze Test History Class History Bottom Line è DW-BB

Learning Indicators Direct Indirect (affects learning) Grades (GPA) Test Data Placement Completion (dev ed / gateway courses, core, certificate/degree) Subject Levels (normalized placement) Academic Status (Probation, Suspension, PTK, Near graduation, LHF) Outcome Assessment Data Indirect (affects learning) Demographic & Personal Data Attendance (99% of success is showing up) Student Activities/Support Services (Engagement- Advisement, Tutoring) Enrollment patterns External factors (Financial, job, family, etc) Blaine Presents: Lots of data. Give opportunity to discuss other data elements (quantitative + qualitative) DW 10

Find It, Assess It, Improve It SACS Expectations 3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: 3.3.1.1 educational programs, to include student learning outcomes Find It, Assess It, Improve It Close the Loop Dick Starts here DW

Outline Find It - identify expected outcomes Assess It – integrate multiple assessment methods Improve It/Close the loop – use detailed student information to improve learning outcomes DW

Identify Outcomes (Find it) (C.S. 3.3.1.1) Identifies expected outcomes Academic Departments developed Program Outcomes – e.g. General Studies – “Reason quantitatively as well as verbally.” Identify course outcomes as they relate to Program Outcomes Dick Presents DW

Outcomes – College Algebra Define and interpret functions including: linear, quadratic, higher degree polynomial, absolute value, radical, exponential and logarithmic Use the graph and other methods to determine the domain and range of a function Solve equations/inequalities of various types with particular emphasis on linear and quadratic polynomials. Perform basic operations on functions including arithmetic combinations, composition/inverse and the transformations of their various graphs Use properties of logarithmic/exponential expressions in graphing and solving Apply exponential functions to model exponential growth/decay Use graphing calculator features to determine algebraic and graphical results Use the fundamental theorem of algebra Use algebraic concepts to solve real-world applications Note: Outcomes developed by faculty DW

Assess Outcomes (Assess it) Assesses expected outcomes Acquired Prosper (Scantron Test Management System) to collect and report outcome assessment data Developed custom forms using objective and subjective answer responses Used forms to create pre and post assessments for use in dev-ed and gateway courses Created assessment-based “grades” to better reflect true student learning DW

Develop Outcome Assessment Develop questions for each outcome (e.g. 4 questions per outcome) Enter questions and answers into Prosper Link questions to outcomes Create assessments from question pool Establish outcomes mastery level (e.g. 75% 3 of 4 questions for related outcome answered correctly) Score assessments and store results on central Prosper database for immediate access Upload results to Estudias (student data warehouse) More often than not, grades and assessment do not line up. DW-MT

Key points: Automatic downloads, no additional data entry Data Flow More often than not, grades and assessment do not line up. Key points: Automatic downloads, no additional data entry MT

Prosper Answer Form Objective and subjective (i.e. writing samples) responses captured MT

Assess It – Captured Image Writing for Academic Success Objective Image CGarabedian discusses the Prosper form and WAS. Subjective Image

Outcome Analysis (Demo) MATH 1314 (College Algebra) outcomes MT

Outcome Analysis (Demo) looking at one outcome across faculty members We can see great differences in how students did on one particular outcome across faculty members MT

Improved Programs & Learning (Institution) College Algebra Final Grade vs. Post-test Assessment (All sections) Ideally peaks should line up. DW

Improved Programs & Learning (Faculty) College Algebra Final Grade vs. Post-test Assessment (Single Section) Realistic result: Assessment lags grade. Better alignment between grades and assessment DW

Identify Outliers Once Assessment scores are in the data warehouse we can compare how the students did on the post-assessment test to what letter grade (Final Grade above) they received in the class. One would expect them to be similar, but … In theory, most students should fall on the two diagonals shown Students off the diagonals (outliers) offer possibilities for interventions 21% of the students who got a C on the post-test, received an A in the class Students who work hard, but are poor test takers? 4% of the students who got an A on the assessment, got a C in the class Students who know the material, but are bored with the class? DW

So what does all of this have to do with Aaron & Susan? Aaron Abel Suzanna Alvarez BB

Student Dashboard (Improve It / Close the Loop) Blaine Presents: Lots of data. Give opportunity to discuss other data elements (quantitative + qualitative) BB

Demo Get groups, flip through students Good group: Close to graduation Bad: Not attending Good outlier, bad outlier Groups based on assessment

Conclusion Don’t forget about individual students in institutional analysis (student information should drive institutional information) Need multiple data sources to analyze beyond just grades (i.e. outcomes assessment ,attendance, pre/post testing, NSC) A data warehouse (i.e. ZogoTech) can help Blaine Presents: Lots of data. Give opportunity to discuss other data elements (quantitative + qualitative)

Questions Southwest Texas Junior College ZogoTech Blaine Bennett blaine.bennett@swtjc.edu (830)591-7275 Dick Whipple rbwhipple@swtjc.edu (830)591-7326 Hector Gonzales ZogoTech Michael Taft mtaft@zogotech.com (214) 774-4780 x801