1. Student Self-Review: Impacts on Future Class Discussion
Jim Flowers Professor & Director of Online Education Ball State University
Samuel Cotton Assistant Professor Ball State University
A full report on this research is currently being prepared for publication.

2. Goal
Improving Learning in Online Education
Cognitive Dialog

3. Feedback from the Instructor
Meets instructor’s criteria
Timely
May be
Private
Public
Content-specific
Organizational/other

4. Costs of Instructor Feedback
Demand for instructor’s time
Dependence on instructor rather than self-reliance

5. Question:
Can some meaningful “feedback” one how a student is doing be provided without instructor time?
Early in a class
Regarding engaging in a critical cognitive dialog

6. Possible New Solution
Student self-review

7. Does Student Self-Review Impact Cognitive Dialog
Cognitive discourse in threaded discussions
Quantity of messages posted
Quality of messages posted
Reports from participants

8. Measures of the Quality of Cognitive Dialog
Function
Skill
Level
(Adapted from Henri, 1992; Henri & Rigault, 1996; Rose, 2002)

9. Functions
Cognitive
Metacognitive
Organizational
Social

10. Cognitive Skills Inference Analysis or Elaboration Judgment
Cognitive Strategy
Elementary Clarification

11. Level
High (complex, long)
Low

12. Dependent Variables Quantity Quality of cognitive dialog
Total number of messages posted
Quality of cognitive dialog
Percent cognitive messages
Percent analysis
Percent inference
Percent complex

13. Sample Single, online, graduate class (n=20) Students are instructors
5-weeks, mid-May to mid-June, 2004

14. Method Week 1 discussions Week 2 discussions
Week 3 discussions and students categorize their Week 1 contributions
Week 4 discussions
Week 5 discussions and questionnaire

15. Method Week 1 discussions: Pre-Treatment
Week 3 discussions and students categorize their Week 1 contributions
Week 4 discussions: Post-Treatment
Week 5 discussions and questionnaire

16. Week 3 treatment: Self-categorization (analysis)
each student identifies their
Total number of messages
Social messages
Other off-topic messages
On-topic messages, according to the following:

17. High, fair, poor quality examples of on-topic dialog:
Posing a relevant question
Offering unsolicited observation, opinions, advice related to the topic
Brief response to another’s question or statement
Lengthy response to another’s question or statement
(Multiple counts per message were possible.)

18. Qualitative Results: Students’ Analyses & Questionnaires
Mixed, but showing some perceived benefit to this activity.
Future research may attempt to investigate this related to learning preferences, self-concept, open-mindedness, and language skills, to name a few.

19. “After reading my postings, I realized that I did not go as in depth as I would have liked to.”

20. “…I feel my thought process is changing and my responses are improving in quality.”

21. “I have a tendency to offer unsolicited observations…
“I have a tendency to offer unsolicited observations…. Apparently I tend to tie almost everything into past experiences and I have become very opinionated… Hopefully this will change because of this reflection.”

22. “In some ways, I find the rubric limiting in that I feel I must consciously pay attention to how much I am posting and of what type I am posting. I think I would post more if it were more free-flowing and not proscribed.”

23. Quantitative Results: Discussion Analysis by Researchers

24. Unitizing Students looked at a message as the unit.
Researchers looked at the sentence.
This may have decreased the percentage of High Level units in both the pre- and post-treatment data.

25. Coding Two coders Third as a tie-breakers
Discussions to resolve 3-way disputes

26. Intercoder Reliability
Function: 88%
Skill: 62%
Level: 92%

27. What do you think? Effect of self-categorization on
Total number of units
Percent cognitive
Percent inference
Percent analysis / elaboration
Percent high level

28. Quantitative Results : Discussion Analysis by Researchers

29. Number of units Decreased 48.7 to 35.6 per person
(Average unit length decreased from 17.3 words to 13.2 words.)

30. Function Cognitive: 90% Pre, 83% Post (p=.018)
Metacognitve: 7% Pre, 5% Post (p=.143)
Organizational: 2% Pre, 5% Post (p=.003)
Social: 2% Pre, 7% Post (p=.001)

31. Skill Inference: 10% Pre, 6% Post (p=.049)
Analysis: 18% Pre, 7% Post (p<.001)
Judgment: 16% Pre, 29% Post (p<.001)
Strategy: 1% Pre, 5% Post (p=.024)
Elem. Clarification: 55% Pre, 54% Post (p=.417)

32. Level
High: 13% Pre, 8% Post (p=.007)
Low: 87% Pre, 92% Post

33. Percent cognitive

34. Conclusions
This self-categorization reduced the quantity and quality of cognitive dialog, likely due to an increased hesitancy brought on by an increased awareness that each message would be scrutinized.
This activity should not be used as a means to improve cognitive dialog.

35. Conclusions
Some participants reportedly became aware of a need to change that did not manifest itself in their discussion.

36. Recommendations
Attempts to improve cognitive dialog should focus on interventions that do not emphasize student performance assessment.

37. Recommendations for Future Studies
Control group
Longer course than 5 weeks
Allowing acclimation and additional discussion experience prior to treatment
Practice, successive feedback

38. References
Rose, M. A Cognitive dialogue, interaction patterns, and perceptions of graduate students in an online conferencing environment under collaborative and cooperative structures. Ed.D. diss., Indiana University, Bloomington.
Henri, F Computer conferencing and content analysis. In Collaborative learning through computer conferencing: The Najaden papers ed. A. R. Kaye, New York: Springer-Verlag.
Henri, F., and C. R. Rigault Collaborative distance learning and computer conferencing. In Advanced educational technology: Research issues and future potential. Vol. 128, Computer and Systems Sciences ed. C. O’Malley, New York: Springer-Verlag.