Download presentation
Presentation is loading. Please wait.
Published bySuzanna Patrick Modified over 9 years ago
1
Using IDEA for Assessment Howard Community College August 17, 2012 Shelley A. Chapman, PhD
2
Plan for this Session Program Evaluation & Assessment of Student Learning Group Summary Reports Accreditation Guides Benchmarking Reports Aggregate Data File
3
What makes IDEA unique? 1.Focus on Student Learning 2.Focus on Instructor’s Purpose 3.Adjustments for Extraneous Influences 4.Validity and Reliability 5.Comparison Data 6.Flexibility
4
Student Learning Model: 2 Assumptions Assumption 1: Types of learning must reflect the instructor’s purpose.
5
Student Diagnostic Form Assumption 2: Effectiveness determined by students’ progress on objectives stressed by instructor
6
Diagnostic Report Overview Page 1 – Big Picture How did I do? Page 3 – Diagnostic What can I do differently? Page 2 – Learning Details What did students learn? Page 4 – Statistical Detail Any additional insights?
7
Your Average (5-point Scale) RawAdj. A.Progress on Relevant Objectives 1 Four objectives were selected as relevant (Important or Essential—see page 2) 4.14.3 1 If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average. The Big Picture
8
Progress On Relevant Objectives 4 4.3 + 4.3 4.1 4.2 3.6 5
9
Summary Evaluation: Five-Point Scale Report Page 1 Your Average Score (5-point scale) RawAdj. A.Progress on Relevant Objectives Four objectives were selected as relevant (Important or Essential—see page 2) 4.14.3 Overall Ratings B. Excellent Teacher4.74.9 C. Excellent Course4.14.4 D. Average of B & C4.44.7 Summary Evaluation (Average of A & D)4.34.5 50% 25%
10
Individual Reports to Group Reports
11
The Group Summary Report How did we do? How might we improve?
12
Defining Group Summary Reports (GSRs) Institutional Departmental Service/Introductory Courses Major Field Courses General Education Program
13
Adding Questions Up to 20 Questions can be added Institutional Departmental Course-based All of the above
14
Local Code Use this section of the FIF to code types of data.
15
Defining Group Summary Reports Local Code 8 possible fields Example: Column one – Delivery Format 1=Self-paced 2=Lecture 3=Studio 4=Lab 5=Seminar 6=Online Example from Benedictine University
16
Example Using Local code Assign Local Code 1=Day, Tenured 2=Evening, Tenured 3=Day, Tenure Track 4=Evening, Tenure Track 5=Day, Adjunct 6=Evening, Adjunct Request Reports All Day Classes Local Code=1, 3, & 5 All Evening Classes Local Code=2, 4, & 6 Courses Taught by Adjuncts Local Code=5 & 6
17
Description of Courses Included in this Report Number of Classes Included Diagnostic From 42 Short Form 27 Total 69 Number of Excluded Classes 0 Response Rate Classes below 65% Response Rate 2 Average Response Rate 85% Class Size Average Class Size 20 Page 1 of GSR
18
Assessment of Learning What are our faculty emphasizing? How do students rate their learning? How do our classes compare with others? How do our students compare with others (self- rated characteristics)? What efforts can we make for improvement? (How can we “close the loop”?)
19
What learning are we targeting? HCC Gen Ed Courses123456789101112 English 121 CompositionXXXX Biology 101 General BiologyXXXX Math 108 Business MathXXX Art History 104 Art History 1XX History 121 The Ancient World XXXX Psychology 101 General Psychology XXX Anthropology 110 Global Archeology XXX IDEA Learning Objectives
20
What learning are we targeting? HCC Gen Ed Courses123456789101112 English 121 CompositionXXXX Biology 101 General BiologyXXXX Math 108 Business MathXXX Art History 104 Art History 1XX History 121 The Ancient World XXXX Psychology 101 General Psychology XXX Anthropology 110 Global Archeology XXX IDEA Learning Objectives
21
What are We Emphasizing? Percent of Classes Selecting Obj. as Important or Essential This GroupInstitutionIDEA System Objective 116%70%78% Objective 213%59%75% Objective 341%58%75% Objective 432%35%55% Objective 523%19%32% Objective 632%14%25% Objective 722%27% Objective 878%43%47% Objective 919%23%41% Objective 107%11%23% Objective 1168%42%49% Objective 1220%23%41% Average # of Obj. Selected3.74.25.7 Page 2
22
What are We Emphasizing? Page 9 Section B Number Rating Percent indicating amount required None or Little SomeMuch Writing662%17%82% Oral Communication666%42%52% Computer Application6650%44%6% Group Work6627%59%14% Mathematics/Quantitative Work 6597%3%0% Critical Thinking660%30%70% Creative/Artistic/Design6661%33%6%
23
How do students rate their learning? Page 3 Part 1: Distribution of Converted Scores Compared to the IDEA Database
24
Overall Progress Ratings (Courses) Page 3 Percent of Classes at or Above the IDEA database Average
30
Instructors’ Reports on Course Emphases: Selected Pairings-Writing and Oral Communication
31
Instructors’ Reports on Course Emphases: Selected Pairings-Critical Thinking & Writing
32
Highlights for ABCU Remarkably similar profiles across terms Overall response rates ranged from 75% to 81% 1 st term in which administration was primarily online achieved a 75% response rate Transition from paper to online (fall 2010 to fall 2011) does not show major differences in profiles ABCU faculty focus on 3-4 outcomes as essential/important IDEA scores for ABCU faculty are in the similar range on each of the 4 primary indices of teaching effectiveness Over the last 3 terms, a significant increase on several objectives has been observed: application of course material, oral and written communication skills, & analysis and critical thinking skills (objectives 3, 8, & 11, respectively)
33
Overall Progress Ratings (Courses) Part 3: Percent of Classes at or Above This Institution’s Average Page 4
34
Do Students’ report of learning meet our expectations? Pages 5 and 6 Raw Average Adj. Average # of Classes This Report 3.9 11 Institution 4.2 3,963 IDEA System 4.0 31,991 Objective 1: Gaining factual knowledge (terminology, classifications, methods, trends )
35
Which teaching methods might we use to improve learning? Page 7 Teaching Methods and Styles Stimulating Student Interest# ClassesAv.s.d. 15. Inspired students to set and achieve goals which really challenged them 423.80.5
36
Relationship of Learning Objectives to Teaching Methods
37
How do students view course work demands? Page 8B Student Ratings of Course Characteristics Diagnostic Form Item # & ItemAverage% Classes Below 3.0 % Classes 4.0 or Above 33. Amount of Reading This Report3.421%24% Institution3.331%19% IDEA System3.233%15% 34. Amount of work in other (non-reading) assignments This Report3.324%10% Institution3.423%20% IDEA System3.421%18% 35. Difficulty of subject matter This Report3.219%0% Institution3.513%19% IDEA System3.420%18%
38
Program Evaluation GSRs, Accreditation Guides, Benchmarking Reports, Aggregate Data Files, Raw Data Files
39
GSRs Help Address Questions Longitudinal Contextual Curricular Pedagogical Student Learning- focused
40
Aggregate Data File Allows you to Use Excel Spreadsheet Use with SAS or SPSS Ask other types of questions Display data in different ways
41
Benchmarking Institutional and Discipline Reports
42
Benchmarking Reports Comparison to 6-10 Peers Same Carnegie Classification IDEA database
43
Benchmarking Reports The student, rather than the class, is the unit of analysis Percentage of positive ratings is given rather than averages
44
Comparison Groups ABCU ---------- ---------- Peer* ------------------------------ Carnegie ---------- ---------- National ---------- ---------- * Peer group is based on 10 institutions identified by the Provost staff and Deans
45
Response Rates ABCU student participation is similar to that of each comparison group ABCU=79% Peer=77% Carnegie=79% National=75%
46
Students’ Perceptions
47
Trends: Student Perceptions Small, steady positive trend over last 5 years ABCU clusters with the other comparison groups Although trends for ABCU are higher than for other groups, these are not meaningful differences Of the 5 contexts identified on the FIF, the strongest positive perceptions at ABCU were for progress on objectives in the graduate education context; this represents a narrow, yet important, segment of ABCU programs (see next slide)
48
Graduate Students
49
Instructional Objectives Selected by Instructors Instructors’ Intentions/ focus Students’ Self-Reported Progress on Learning
50
IDEA Objective 3 Learning to apply course material (to improve thinking, problem solving, and decisions)
51
IDEA Objective 8 Developing skill in expressing oneself orally or in writing
52
IDEA Objective 11 Learning to analyze and critically evaluate ideas, arguments, and points of view
53
Teaching Methods and Styles Reported by Students (Diagnostic Forms Only) Fostering Student Collaboration Encouraging Student Involvement
54
Assessment of Learning in HCC’s General Education Howard Community College General Education Possible IDEA Learning Objectives 1 English 121: Composition8, 9 Biology 101: General Biology1, 2 Math 108: Business Math1, 2, 5 Art History 104: Art History 16, 4, 1, 2 History 121: The Ancient World1, 11, 8 Psychology 101: General Psychology 1, 2, 3, Anthropology 110: Global Archeology 7, 10,12 1 Based partly on study of percent of classes selecting objectives by discipline
55
How Did Students Rate their Learning in General Education?
56
Benchmarking Look at “General Education”
58
Using Aggregate Data for Assessment Howard Community College Goals Program Learning Outcomes Group Summary Report Group Summary Report, Include Extra Questions Benchmarking: One Year or 3-5 Year Trend Report Benchmarking: Discipline Report Course Learning Outcomes Course Learning Outcomes Course Learning Outcomes Course Learning Outcomes
59
HCC General Education Goals and IDEA Howard Community College General Education Goals IDEA Learning GoalsIDEA Teaching Methods Associated with Progress on Learning Goal Written Communication 8 Developing skill in expressing myself orally or in writing 5, 4, 14 (real life), 15, 18 Oral Communication 7 Gaining a broader understanding and appreciation of intellectual/cultural activity 7, 8, 13, 19 Scientific and Quantitative Reasoning 2 Learning fundamental principles, generalizations, or theories 11 Learning to analyze and critically evaluate ideas, arguments, and points of view 2, 7, 8, 13, 15, 16, 18, 19 Critical Thinking 9 Learning how to find and use resources for answering questions or solving problems 2, 7, 8, 9, 13, 14, 15, 18, 19 Global Competency 7. Gaining a broader understanding of intellectual/cultural activity Extra Question 7, 8, 13,19
60
Example: Pre-Pharmacy Pre-Pharmacy Goals & OutcomesIDEA data Mastery of fundamental biological, chemical, and physical principles underlying the living systems Learning Objective 2, Teaching Methods 2, 4, 6, 8, 10, 12, 15, 16 Apply knowledge of scientific principles and concepts to clinical/scientific problems Learning Objective 3 Teaching Methods 1, 2, 3, 4, 6, 8, 10, 11, 15 Prepare effective written reports and oral presentations to communicate science to professionals and lay persons Learning Objective 8 Teaching Methods 7, 15, 16, 18, 19 Use mathematical concepts to analyze dataLearning Objectives 2 & 3 Teaching Methods 1, 2, 3, 4, 6, 8, 10, 15
61
Questions? www.theideacenter.org
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.