RESEARCH TO PRACTICE IN HIGHER EDUCATION: USE THE RESULTS OF RECENT RESEARCH TO INFORM HOW YOU DOCUMENT AND EVALUATE STUDENT LEARNING Wendy Machalicek & Melinda Leko UW-Madison 2010 Teaching and Learning Symposium

MELINDA LEKO, Ph.D. Assistant Professor Department of Rehabilitation Psychology and Special Education WENDY MACHALICEK, Ph.D. Assistant Professor Department of Rehabilitation Psychology and Special Education

Agenda ① Background ② Case studies ③ Discussion

Teaching is scholarly work Acts of discovery, application, integration, and teaching are all "scholarships” Boyer (1990) Teaching and research share questioning, exploring, testing, and professing Bender & Gray (1999)

Benefits of approaching teaching in scholarly way Accountability Documentation of our commitment to excellence in teaching Collegiality Brings the usually private, lonely work of teaching into the public sphere Preparing students as researchers Potential peer-reviewed publication

Best practices for assessing student learning Special Education Teacher Education Literature Attitudes, Dispositions, & Perceptions Marable, Leavitt-Noble, & Grande (2010) Evaluated how the use of nonfiction book talks affected students’ attitudes regarding disability Qualitative thematic analyses of reflection assignment and open-ended survey Carroll, Petroff, & Blumberg (2009) Evaluated students’ perceptions of an undergraduate inclusive program Qualitative modified analytic induction of post program interviews with 12 students

Best practices for assessing student learning, cont. Special Education Teacher Education Literature Knowledge Miller et al. (2009)Used concept maps to assess growth in student knowledge Qualitative analyses on the depth and breadth of concepts included on concept maps Quantitative analyses on the number of concepts and interrelationships Dieker et al. (2009)Evaluated student learning of various content area instructional techniques Pre/post written descriptions Pre/post questionnaires Pre/post teacher made assessment

Best practices for assessing student learning, cont. Applied Behavior Analysis Literature Knowledge Malanga & Sweeney (2008) Compared effects of daily assessment & response cards on weekly quiz scores Single-subject alternating treatments design Saville et al. (2006)Compared interteaching and lecture on quiz performance Single-subject alternating treatments design Pretest Social validity questionnaire

Best practices for assessing student learning, cont. Applied Behavior Analysis Education Literature Knowledge Neef, McCord, & Ferreri (2006) Compared guided lecture notes to completed lecture notes on quiz performance Single-subject alternating treatments design Saville & Zinn (2009)Evaluated effects of “quality” points on exam scores Single-subject alternating treatments design

Purpose of today’s presentation Present strategies that we have used in our own classes to evaluate the effects of instructional activities on student learning/growth Provide opportunity for you to brainstorm potential strategies to evaluate student learning in your own courses

EVALUATING STUDENT LEARNING IN THE CONTEXT OF TEACHER PREPARATION

An Example: RPSE 479 Language & Reading Instruction for Students with Learning & Behavioral Disabilities Desired Outcomes: 1.Knowledge of content knowledge and pedagogical content knowledge for teaching reading 2.Beliefs and Attitudes to Support evidence-based reading practices 3. Motivation, Interest, & Engagement

Desired Outcome: Knowledge 1.Pre/post concept maps 2.Online quizzes 3.In class summative assessment 4.Culminating research paper Higher levels of Bloom’s Lower levels of Bloom’s

Desired Outcome: Beliefs & Attitudes Online journaling -students respond to 6 prompts over the course of the semester - prompts are designed to elicit students’ beliefs and attitudes about various instructional practices in reading

Desired Outcome: Beliefs & Attitudes Online journaling First Prompt: Describe your philosophy for teaching reading to students with disabilities. Last Prompt: Revisit your original philosophy for teaching reading. What revisions would you make?

Desired Outcome: Motivation, Interest, & Engagement Informal student evaluations -Administered at least twice a semester -”Describe 2 things that are going well in the course and 2 suggestions you have for improving the course.”

RPSE 330 Behavior analysis: Applications for persons w/disabilities In this course you will… –Become familiar with the basic principles of human behavior as they apply to children with exceptional learning and behavior characteristics –Apply these principles to develop, implement, and evaluate programs of behavior change in educational settings

Desired Outcomes Knowledge, understand, apply, analyze, evaluate, and create 1.Weekly quizzes 2.Dyadic discussions 3.Video clip of key concept 4.Annotated bibliography of intervention literature 5.In class application activities Higher levels of Bloom’s Lower levels of Bloom’s

Instructional design Adapted inter-teaching method –reciprocal peer tutoring –problem-based learning –cooperative learning –Instructor reinforcement of student engagement Boyce & Hineline, (2002); Saville et al., (2005) Saville, Zinn, Neef, Van Norman, & Ferrari (2006)

Review and student assessment Each review session based on student feedback concerning areas of interest and/or concepts and skills that require clarification Following review session, students complete a short quiz of the previous week’s readings and activities

Dyadic discussion Students discuss assigned readings with one or more peers. Weekly discussion guides created by each student used to prompt and guide discussion. The professor and teaching assistant move around the room taking notes on the topic and quality of discussions, engage students in dialogue, and help students to clarify any questions, concerns, or areas of interest that arise. Dyads randomly called on to summarize key points of readings

Discussion guide Describe the 3 main points of this week’s readings Describe your own experiences in regard to the topics covered Describe 3 ideas, concepts, or strategies you would like to discuss further Describe any concerns or difficulties you have with this week’s assigned readings Any questions?

Example from discussion guide Own experience “I was homeschooled by my mother up until the 5 th grade and I noticed how similar her teaching style was to modeling….When my sister finally learned how to read, I was even more inclined to learn myself because I wanted to follow their behavior pattern. The same thing happening when my mom showed enthusiasm towards various tasks and projects—instead of being against her, my sister and I seemed to be enthusiastic and had a good time learning with her.”

Brief lecture and application activity Brief lecture based on and expanding upon concepts covered in the readings One or more problem-based small group application activities One or more groups randomly called upon to present response to activity to entire class.

Student self-evaluation and feedback At end of each class, students asked to provide feedback on the quality of their discussions/completion of the application activities and to describe concepts or skills that require further clarification or practice Professor uses feedback guides to structure a review for the following week

Student self-evaluation Instructions: Circle the one number (1- 6) that best describes the quality of your discussion today. Provide a brief explanation for your rating. “I rated the discussion I had with 2 of my peers a 5, because I felt I understood the concepts better when I got to hear the answers they came up with and compare them to what I got.”

Student feedback Instructor Guide Instructions: List and/or describe any concepts covered today that require further clarification. “Could you go over positive and negative reinforcement and punishment more in detail?” “Discrete versus chained target behaviors, ABC for everyday functional use (besides severe behavior)”

Adapted inter-teaching process Review of previous week’s material Dyadic discussion Brief lecture on current week’s materials Application activity Student self- evaluation & feedback

RESULTS DISCUSSION

Some key questions What student outcomes are of most interest to you? What strategies or research methodologies for evaluating student learning might you want to try? How will you document student outcomes?

MELINDA LEKO, Ph.D. Assistant Professor Department of Rehabilitation Psychology and Special Education WENDY MACHALICEK, Ph.D. Assistant Professor Department of Rehabilitation Psychology and Special Education