1. Making Sense of Online Learning: Frames, Rubrics, Tools & Coding Systems for Analyzing Asynchronous Online Discourse Theresa Flynn Pepperdine University tflynn@pepperdine.edu Linda Polin Pepperdine University lpolin@pepperdine.edu

2. Co-construction of Knowledge & Asynchronous Online Discourse ASSUMPTIONS :  collaborative learning is happening online  Asynchronous discourse is a rich source of data for analysis

3. Research Questions Can we capture evidence of learning in asynchronous online settings? If so, how? Are there indicators that “deep learning” is occurring in asynchronous learning settings?

4. Study Context Pepperdine University’s 90% Online Masters in Educational Technology Program—2002/2003 5th year of the program 3 trimesters July 2002—July 2003 23 students—across the USA and abroad Only 3 face-to-face meetings in the 13 month program

5. Rubric Development Audience –To whom are messages being addressed in newsgroups? Discourse Function –What roles do different “types” of posts serve in the learning process?

6. Audience Rubric Self Specific Person(s) Instructor Group Other

7. Discourse Functions Rubric Rumination Storytelling Disagreement/Argumentation Social Interaction Procedural/Logistical

8. Discourse Functions Rubric Acknowledgement Reference/Resource Inquire Other

9. Thread Selection 2 threads from 1 st trimester 2 threads from 2 nd trimester Length of threads—looked for active posting Topic: 2 threads about course readings (chapters) 2 threads about class process (asynchronous & synchronous classes)

10. Results of Coding: Audience

11. Chapter Threads

12. Process-Oriented Threads

13. Results of Coding: Discourse Functions

14. Instructor: A Trimester 1 # of messages: 29 Thread Length: 20 days # of Unique Participants: 15 Total # of class members: 23 Chapter 4: Workplace Settings

15. Instructor: B Trimester 2 # of messages: 28 Thread Length: 26 days # of Unique Participants: 17 Total # of class members: 23 Chapter 3: Schooling

16. Instructor: A Trimester 1 # of messages: 16 Thread Length: 8 days # of Unique Participants: 11 Total # of class members: 23 Synchronous Chats

17. Instructor: B Trimester: 2 # of messages: 29 Thread Length: 11 days # of Unique Participants: 22 Total # of class members: 23 Posting Protocol

18. Validity: Interviews 8 students interviewed All agreed with the categories presented in both rubrics “The rubric covers it—You’ve done a fantastic job framing all of this. I never thought it had structure”

19. Validity: Interviews “I think [the self category] is right on track—these are the posts that usually trigger the best discussions—but also sometimes the hardest to share ” “Very rarely do I feel that I am writing for the benefit of the professor” “When I ruminate it’s because I’ve reflected on something. When I’m storytelling it’s because I’m reflecting on a previous experience and how it’s linked to the current learning experience. When I’m disagreeing it’s because I’m reflecting on how and what you’re saying jives or doesn’t jive with a previous belief I have. In other words, everything that I’m saying in newgroups is a form of reflection”

20. Reliability 2 coders Inter-rater agreement 89% agreement in Discourse Functions category 94.5% agreement in Audience category

21. Disagreement/Argumentation No messages coded as disagreement/argumentation in any thread Closer-examination revealed we had missed several instances of disagreement Dissent was subtle Only apparent when examining the entire context of the newsgroup thread

22. Disagreement/Argumentation: A Closer Look at the Text Tension between “Old-timers” & “Newcomers” Sometimes “newcomers” leave or “jump ship” “Do students have the option of jumping ship?” MLK reference “But what do you do as the professional stagnating in a place unwilling to change?”

23. Conclusion Don’t sacrifice validity for reliability Large degree of inference involved in coding messages—makes coding more difficult The message alone as a unit of analysis is insufficient Must consider the context of the discourse

24. Student Insight The most important part of the newsgroup process is not the reading or the posting— it’s the part in between where I’ve read it, and then I reflect on it before I post a reply. It’s that little, in that processing place”.

25. Context AND Content