1. Assessing course reform in calc-based physics at OSU Dedra Demaree  Assistant Professor of Physics Sissi Li  PhD student in SMED

2. PER research shows us: Students out of traditional introductory physics lecture courses have:  Minimal conceptual understanding  “Plug-and-chug” problem solving skills that are lost  Attitudes that are less expert-like than when they registered Interactive-engagement is more effective for student learning Sophisticated epistemologies are encouraged when students are metacognitive about how they gain knowledge A focus on higher-order learning goals gains us more

3. Our Approach to reform: Build on existing Paradigms expertise  Department-wide, committee based reform Borrow and adapt existing materials  Major influences from Rutgers and NC State Build our goals into all aspects of the intro course  Different homework, labs, learning-activities, EVEN learning spaces!! In spring, change to:  2 h lecture, 2 h activity-based learning, 2 h lab

4. Investigative Science Learning Environment (ISLE): Systematic learning cycle Constructivist approach: students know how they know what they know Focus on teaching scientific abilities Strong on experimental design Transfer of knowledge Student attitudes

6. Scientific Abilities: Representing information Conducting experiments Thinking divergently Collecting and analyzing data Constructing, modifying and applying relationships and explanations Being able to coordinate these abilities

7. Types of ISLE activities: Observation, testing, and application experiments (design-based labs, videos and in-class activities) Anomalous data experiments Multiple Representation practice Video problems Evaluation tasks Jeopardy tasks (Activities online and in Active Learning Guide – X.1 = qualitative concept building and testing, X.2 = conceptual reasoning, X.3 = quantitative concept building and testing, X.4 = quantitative reasoning)

9. ETKINA paper DISCUSSION: What would be meta-goals for your particular field? Are they worth the time needed to spend on them in the classroom? What expense might there be in implementing them? What would be the preparatory need for the instructor? How might you assess the meta-goals?

10. Assessing goal implementation Heavily ISLE-based with use of Peer Instruction Researcher observes back to back lecture classes  Records audio/video data of instructor and a subset of student groups  Takes notes on ‘observation rubric’ Researcher and Instructor both journaling after class Post-class discussions  Modeled after Paradigms

11. Observer in the classroom notices: Can hear student dialogue in the back and tell when students are on/off task: clearer idea of student engagement Has different perspective on why students find some things hard Can see more going on in the classroom than an instructor can attend to while teaching Sees subtle differences in instruction Provides a second opinion on how instructor ‘comes across’ to the students

12. Engagement: it’s mutual (or: ‘it’s always the handler's fault’) We’re engaged in a MUTUAL activity: neither of us could be successful without the other Must pay close attention to each other and surroundings Easy with a highly focused 1 min one-on-one playful interaction Challenging with 200 students for 1 hr – the key is MUTUALITY Frustrations are more often instructor ‘fault’ than I thought

13. Goals for the day (12/03/2008): In later part of work/energy chapter Non-’content’ goals:  Help students by solving an example with integration  Emphasize the importance of carefully setting up a problem by setting up but not solving a second example  Have students work through an example, breaking it down into small steps to make it accessible Class did not go as planned – demo equipment failed, instructor ran out of time, student engagement seemed low in the first class, students seemed confused

14. Video Clip Video Clip (12/03/2008) The instructor wasn’t happy (and was hungry!!) 1:30 – 1:58 Observer can clearly state what the problem was with the students – they didn’t understand the task – I was confused/surprised 1:58 – 2:33 Observer noticed the difference between my instruction in the two classes that made an improvement 2:33 – 2:47 Observer can tell clearly when students are on/off topic 2:47 – 3:06 Observer can explain student behavior

15. Journals from that day: Researcher/observer: The last problem was challenging… heard many groups trying to make sense of the question and not jump straight into problem solving and physics Instructor: …both classes completely floundered on the problem I gave them... I have not made any progress at helping them see a new/unknown situation and trust the process to help them move forward even if they don't know what the answer will be.

16. More from journals that day: Researcher/observer: In the 3pm class she worked through the first answer with the students and then it seems like the students moved along better. Instructor: On a positive note, I did feel they were more responsive today - the 3pm more so than the 2pm Notice different tone and what the observer could point out that the instructor did not notice The instructor frustration at not achieving goals would otherwise make it hard to make progress

17. Impact on instruction ‘Telling them what to do’ was NOT mutual engagement – I presented a broken down list for the global problem then left them to it  I would never have guessed that this could be confusing Attending to their confusion helps them engage more, where (duh!) chastising them adds distance  Hearing about the group that didn’t know where to start made me realize that off-topic is my failure in the engagement, not theirs! Observer sees more ‘progress’ than I do – and can give feedback to promote future success

18. ELBY paper DISCUSSION: What role does epistemological beliefs play in the classroom? How does a teacher’s orientation toward teaching (teacher orientation) influence how activities are implemented in the classroom? How does this tie into effectively implementing meta-goals?

19. ISLE as Teaching framework and impact of discourse analysis ISLE provides framework from which to analyze what is happening in the classroom  provides the instructor with the ability to adjust and adapt in real-time Having explicit goals changes the types of classroom discussions and level of student engagement with activities Studying the subtleties of the interactions informs how we improve our implementation Documenting PCK and how students engage with activities will inform other teachers’ instruction

20. Observations Normalized FCI gain of 0.4 with 200 student lecture, one teacher, no reform (yet) to lab or recitation Students dialogue about multiple explanations (in groups and with full-class discussions) Students initiate in-depth dialogue about context and assumptions of standard clicker questions Students produce multiple representations during class and use them to start example problems Students interject and answer each other, creating spontaneous student-lead full-class discussions

21. Ongoing challenges Implementing “partial” ISLE cycles Students seem to be irregularly responsive Base set of students who dominate conversations while others are not comfortable participating Students struggle to use new ideas without heavy scaffolding Sometimes teacher falls back on “old-ways” – especially when short on time

22. Teacher adjustments: Learning the PCK around specific activities Gave a “standard” voting question about momentum conservation Students in the earlier lecture prompted a discussion about what difference a crucial assumption would make and how to treat the scenario with and without that assumption When the later lecture students did not bring up the question, the instructor was able to use knowledge from the earlier class to enhance the discussion

23. Teacher adjustments: PCK for How to effectively ramp up to full ISLE goals In an ISLE cycle, student in later class offered an unconventional explanation for an observation  Enabled full use of ISLE cycle including falsification  Provided a more convincing justification for the physics concepts that hold up under testing  Discussion helped those students buy-in to the activity  Took much longer to ramp earlier class up to same level of dialogue during ISLE cycles – may have been helped by prompting a similar discussion

24. In-lecture group work

25. Teacher adjustments: Learning how to ask questions – eliciting interest  Voting question had two possible correct answers  In both classes most students quickly saw one of the correct answers  In first class asked “vote again, and see if you have any different ideas” – students did not change votes and were not engaged in follow-up discussion on other correct answer  In second class asked “vote again, and see if you can change your neighbor’s mind” – students still did not change their votes, but a student chimed up to explain what happened in the group discussion and the whole class was engaged (nods, audible agreements…)

26. Teacher adjustments: Supporting spontaneous full-class dialogues Learning to step back and attend to class needs A student interrupts “can you hold on a minute? Can you explain that more?” Teacher asks: “what is your understanding so far?” Student explains their reasoning, instructor chimes in with a clarification, a new student understanding the view and having recently made sense of it interrupts and explains it in a way that makes sense to the confused student – and a fourth person chimes in with a funny physical implication leading to laughter in the classroom

27. ASSESSMENT FROM A STUDENT ANGLE: SISSI’S THESIS WORK

28. Understanding the role of students’ perceived identity in their ability to participate and take advantage of learning opportunities Sissi Li Oregon State University 17 October 2009 OR AAPT, Eugene OR

29. Amelia always sat in the first two rows of class. During small group discussions in class, she frequently grouped and talked with the same students yet she did not often feel confident asserting her answer. On the occasions she spoke up in class, she more often asked questions rather than make statements she justified. Amelia got an A in PH211.

30. Community of Practice (CoP) Community of practice model (Wenger, 1998)  Learning ≈ participation  Trajectory  Identity Students’ perceived identity strongly brings to bear upon their participation trajectory as members and learners in our class CoP  Newcomer  Central  participant

31. Goals Goal: promote productive learning environment  Help S be participants in classroom practices Communicate ideas, design and evaluate experiments to refine understanding, use multiple representations to solve problems  Encourage negotiation of meaning between members of the CoP  Help students develop ownership of knowledge negotiated

32. Research Question  What are the student identities with respect to the physics class CoP as they enter the community?  Expectations of student and teacher roles in the community  Self efficacy (SE) towards doing physics  Perceived self image  Attitude towards social learning  How do the identities change as students engage in the practices? (i.e. What are their trajectories?)  What are the student identities with respect to the physics class CoP as they leave the community?  How do the students perceive their identities in other CoP where they experience physics?

33. Method and Data: Entry ID  Large enrollment calculus-based introductory physics  Over 400 students in two lecture sections  Large array of identities requires organization  Voluntary online survey probing the four identity dimensions using Likert scale items and multiple choice questions  N = 145 (approx 35% of enrolled students)  SE questions modified from a biology self efficacy instrument  Remaining items created by researcher

34. Analysis: identity grouping  Exploratory factor analysis as a guide to group survey items  Desired responses independently determined by two researchers and then discussed to agreement  Emergent identity groups based on survey items determined independently by two researchers and then combined in discussion  Student characteristics derived from description of identity groups

35. Student: Alfred (LH_) Alfred prefers to work on his own rather than with group which is consistent with his being resistant to other forms of social learning. He tends to expect the teacher to tell him what is right and how to do problems. He did well in his calculus class and has average GPA. Alfred things of himself as a good student, science geek and a quiet loner.

36. Student Characteristics Not desiredCharacteristicDesired Prefers to work individually, teaching others does not help me learn Value social learning Prefers to learn in groups, like to engage in meaning making discussions, teaching helps learning I am supposed to listen quietly and accept what I am told. Student roleI am supposed to try to make sense of concepts in class and ask when I don’t understand. Physics is hard and I can’t get it. It doesn’t matter how hard I try. Physics self efficacy Confident in ability to do physics and to learn new physics. Average to poor student, slow learner, lazy student Self imageGood student, others look to me as role model No one notices/knows me, I can’t affect change, I don’t matter. AgencyImportant member of community, I can convince others, I expect others to listen to what I have to say. Teacher is supposed to tell me what to do/say/know, I can’t learn w/o the teacher, learning means I can find the right answer for the teacher. Responsibility for learning I make sure I understand what I’m learning; the teacher is a resource to help me learn.

37. Identity Groups ISLE: ISLE alignment  Average of student role, agency, value of social learning, responsibility for learning. SE: Physics self efficacy  Average of all questions in survey about physics self efficacy AA: Academic achievement  Accounts for self image from survey and GPA prior to PH211

38. Student: Ben (HHL) Ben prefers to listen quietly while working in a group. He is inclined to work with others to learn and is fairly confident in his ability to understand science. He has a high GPA and thinks of himself as an average student who is not good at math. Ben thinks others see him as slow learner.

39. Trajectory & Factors Student attitude on group work (in class observations)  Most are comfortable talking to their peers  More enthusiastic about real life connections  Learning styles differ  Open/close dialog space  Expectations changing over time, matching with student development

40. Student Characteristics  Value of social learning  Role of social learner  Want to be told answer, single approach  ISLE orientation  SE to tackle problem  SE to test exp, justify  SE to convey, explain to others  Anticipated success in course  GPA  Self-image – good student  Student agency, empowerment  Self-image – important member  Attitude about science  Responsibility for learning

41. Discussion Consider these two students and their potential trajectory  Optimal trajectory for success in the course  Likely trajectory  Negative trajectory Pick one type of trajectory for one of the students and come up with factors that may influence the student to follow that trajectory.  What data would you take to observe and examine that trajectory?

42. Trajectory & Factors Student AStudent B Personal perception of learner ability Being in an open supportive group

43. Method and Data: Exit ID  Identity development  Simpler: Guided by desired classroom practices  More complex: Students know more and use physics outside of class CoP  Personal meaning map (PMM)  Six students selected who were observed to frequently participate in practices of small group talk or whole class discussions  Individual interviews  10 minutes of drawing PMM with prompt  20 minutes of discussion about PMM and related tangents

47. Open questions How do students adjust to interactive environment? How does teacher learn to teach with these methods and meta-goals? How do we get local teachers ready to take over the course? What needs to be documented for external dissemination?

49. NOTES: -start with goal discussion and motivation for reform, understanding student adjustment/learning w/i reformed class, building PCK, learning "what works” -D's talk (summer AAPT 2009): learing from post class disc, developing PCK, -C's talk (ORAAPT ID talk): present PMM as method, motivate ID survey as selection process to view complex entry identity with adequate fine grain, connect pre/post assessment of identity -discussion questions based on the papers 1. what meta-level (higher order) goals might you set for learners in your choice of setting (K12/College/FC)? (You should define meta-goal for the class.) 2. what skills/K does teacher/facilitator need? 3. teacher/facilitator orientation discussion? (I think this would be an awesome discussion but it might also take a long time to argue) 4. how do we change teacher orientation at the dept level? 5. how would you assess a meta-goal?

50. How we’re doing it: people Curriculum development teams including faculty and instructors  3 teams, each taking on a few-week ‘bite’ of curriculum  2 members are on all three teams for coherence  Members range from experienced to new faculty that ‘specialize’ in both upper and lower division instruction One ‘service course’ committee to bind it  Surveying other departments, choosing baseline assessments, choosing textbook change, studying what other universities have accomplished…

51. How we’re doing it: structure Original:  3 1-hour lectures per week, 1 3-hour lab, 1 1-hour optional recitation  One instructor/faculty responsible for all lectures and supervising a TA team to do labs/recitation New structure:  2 1-hour lectures per week, 1 2-hour SCALE-UP, 1 2-hour lab  Possible ‘7 th ’ hour for people with extra interest (computational focus), or with extra need (remedial math)  Head instructor (and coordinator) with rotating lecturers, Head TA to help lead SCALE-UP sessions, other TAs and LAs

52. Structural Reform: new SCALE-UP room, lecture hall remodel and course restructuring: less lecture, more group work Mixed-methods: FCI, CLASS, Classroom observations, instructor feedback, surveys, student interviews Site visits to learn from successful models, borrow materials and ideas: ISLE, Peer Instruction, M&I, Knight Department-wide involvement in 'bite- sized' chunks, head committee with broad membership Set explicit content and higher-order goals Build in Active Engagement Structural and course Reform Mixed- methods Assessment

53. Sustainability: Need faculty buy-in for the big picture  Department hired PER-faculty to lead this  Team-based, department-wide involvement Commitment to and understanding of choices  Bite-size curricular committees for ‘all’ faculty  Choices driven by concrete information (surveys, research at other institutions…) Need intro courses more ‘approachable’ for faculty  Can lecture a ‘bite-sized’ chunk without the administrative hassle of 500+ students

54.  Conceptual  Understanding  Problem  Solving  Epistemologies/  Attitudes  Scaffolding/  Math  Appreciation/  Curiosity  Contributes to  success in  Leads back to  More productive  participation  Quality choices help improve  (new nanolab)  Success contributes to productive  Model driving reform choices:

55.  Conceptual  Understanding  Problem  Solving  Epistemologies/  Attitudes  Scaffolding/  Math  Appreciation/  Curiosity  FCI, CSEM…  Productive attitudes enable more participation and engagement in the community of practice  Student feedback, engagement  CLASS, qualitative studies  Model driving assessment choices:  ISLE Rubrics  Committee choices, teacher implementation, student engagement