Student Self-Review: Impacts on Future Class Discussion Jim Flowers Professor & Director of Online Education Ball State University Samuel Cotton Assistant Professor Ball State University A full report on this research is currently being prepared for publication.

Concerns: Limited-quality discussion Need for more feedback Insufficient instructor time to give individual feedback on how to discuss Alternate strategies

Research Question What is the impact of student self-review on future cognitive dialog?

Weekly Blackboard discussions Week 1 discussions: Pre-Treatment Week 2 discussions: Pre-Treatment Week 3 discussions and students categorize their Week 1 contributions Week 4 discussions: Post-Treatment Week 5 discussions and questionnaire

Self review Survey Investigators analyzed discussions

Week 3 treatment: Self-categorization (analysis) each student identifies their 1.Total number of messages 2.Social messages 3.Other off-topic messages 4.On-topic messages, according to the instructor’s criteria:

Instructor’s criteria for high, fair, poor quality examples of on-topic dialog 1.Posing a relevant question 2.Offering unsolicited observation, opinions, advice related to the topic 3.Brief response to another’s question or statement 4.Lengthy response to another’s question or statement (Multiple counts per message were possible.)

Measures of the Quality of Cognitive Dialog Function Skill Level (Adapted from Henri, 1992; Henri & Rigault, 1996; Rose, 2002)

Functions 1.Cognitive 2.Metacognitive 3.Organizational 4.Social

Cognitive Skills 1.Inference 2.Analysis or Elaboration 3.Judgment 4.Cognitive Strategy 5.Elementary Clarification

Level 1.High (complex, long) 2.Low

Dependent Variables Quantity Total number of messages posted Quality of cognitive dialog Percent cognitive messages Percent analysis Percent inference Percent complex

Sample Single, online, graduate class (n=20) Students are practicing teachers/instructors 5-weeks, mid-May to mid-June, 2004

Quantitative Results: Discussion Analysis by Researchers

Unitizing Students looked at a message as the unit. Researchers looked at the sentence. This may have decreased the percentage of High Level units in both the pre- and post- treatment data. (Students had looked at a full message as the unit.)

Coding by Function, Skill & Level Two researchers as coders Third as a tie-breaker Discussions to resolve 3-way disputes

Intercoder Reliability Function: 88% Skill: 62% Level: 92%

What do you think? Effect of self-categorization on Total number of units Percent cognitive Percent inference Percent analysis / elaboration Percent high level

Results : Discussion Analysis by Researchers

Number of units Decreased 48.7 to 35.6 per person (Average unit length decreased from 17.3 words to 13.2 words.)

Function Cognitive: 90% Pre, 83% Post (p=.018*) Metacognitve: 7% Pre, 5% Post (p=.143) Organizational: 2% Pre, 5% Post (p=.003*) Social: 2% Pre, 7% Post (p=.001*)

Skill Inference: 10% Pre, 6% Post (p=.049*) Analysis: 18% Pre, 7% Post (p<.001*) Judgment: 16% Pre, 29% Post (p<.001*) Strategy: 1% Pre, 5% Post (p=.024*) Elem. Clarification: 55% Pre, 54% Post (p=.417)

Level High: 13% Pre, 8% Post (p=.007) Low: 87% Pre, 92% Post

Summary: Decreases in Quantity Total number of messages posted Quality of cognitive dialog Percent cognitive messages Percent analysis Percent inference Percent complex

Results: Students’ Analyses & Questionnaires Mixed, but showing some perceived benefit to this activity.

Student Comment “After reading my postings, I realized that I did not go as in depth as I would have liked to.”

Student Comment “…I feel my thought process is changing and my responses are improving in quality.”

Student Comment “I have a tendency to offer unsolicited observations…. Apparently I tend to tie almost everything into past experiences and I have become very opinionated… Hopefully this will change because of this reflection.”

Student Comment “In some ways, I find the rubric limiting in that I feel I must consciously pay attention to how much I am posting and of what type I am posting. I think I would post more if it were more free-flowing and not proscribed.”

Conclusions This self-categorization reduced the quantity and quality of cognitive dialog, likely due to an increased hesitancy brought on by an increased awareness that each message would be scrutinized.

Conclusions This activity should not be used to: improve cognitive dialog; or reduce instructor time for feedback without reducing quality of discussions.

Conclusions Some participants reportedly became aware of a need to change that did not manifest itself in their discussion.

Recommendations Attempts to improve cognitive dialog should focus on interventions that do not focus students’ attention on the assessment of their cognitive dialog.

Recommendations for Future Studies Future research may attempt to investigate this related to learning preferences, self- concept, open-mindedness, and language skills, to name a few.

References Rose, M. A Cognitive dialogue, interaction patterns, and perceptions of graduate students in an online conferencing environment under collaborative and cooperative structures. Ed.D. diss., Indiana University, Bloomington. Henri, F Computer conferencing and content analysis. In Collaborative learning through computer conferencing: The Najaden papers ed. A. R. Kaye, New York: Springer-Verlag. Henri, F., and C. R. Rigault Collaborative distance learning and computer conferencing. In Advanced educational technology: Research issues and future potential. Vol. 128, Computer and Systems Sciences ed. C. O’Malley, New York: Springer-Verlag.

Student Self-Review: Impacts on Future Class Discussion Jim Flowers Professor & Director of Online Education Ball State University Samuel Cotton Assistant Professor Ball State University A full report on this research is currently being prepared for publication.