Student Self-Review: Impacts on Future Class Discussion Jim Flowers Professor & Director of Online Education Ball State University Samuel Cotton Assistant Professor Ball State University A full report on this research is currently being prepared for publication.

Goal Improving Learning in Online Education Cognitive Dialog

Feedback from the Instructor Meets instructor’s criteria Timely May be Private Public Content-specific Organizational/other

Costs of Instructor Feedback Demand for instructor’s time Dependence on instructor rather than self-reliance

Question: Can some meaningful “feedback” one how a student is doing be provided without instructor time? Early in a class Regarding engaging in a critical cognitive dialog

Possible New Solution Student self-review

Does Student Self-Review Impact Cognitive Dialog Cognitive discourse in threaded discussions Quantity of messages posted Quality of messages posted Reports from participants

Measures of the Quality of Cognitive Dialog Function Skill Level (Adapted from Henri, 1992; Henri & Rigault, 1996; Rose, 2002)

Functions Cognitive Metacognitive Organizational Social

Cognitive Skills Inference Analysis or Elaboration Judgment Cognitive Strategy Elementary Clarification

Level High (complex, long) Low

Dependent Variables Quantity Quality of cognitive dialog Total number of messages posted Quality of cognitive dialog Percent cognitive messages Percent analysis Percent inference Percent complex

Sample Single, online, graduate class (n=20) Students are instructors 5-weeks, mid-May to mid-June, 2004

Method Week 1 discussions Week 2 discussions Week 3 discussions and students categorize their Week 1 contributions Week 4 discussions Week 5 discussions and questionnaire

Method Week 1 discussions: Pre-Treatment Week 3 discussions and students categorize their Week 1 contributions Week 4 discussions: Post-Treatment Week 5 discussions and questionnaire

Week 3 treatment: Self-categorization (analysis) each student identifies their Total number of messages Social messages Other off-topic messages On-topic messages, according to the following:

High, fair, poor quality examples of on-topic dialog: Posing a relevant question Offering unsolicited observation, opinions, advice related to the topic Brief response to another’s question or statement Lengthy response to another’s question or statement (Multiple counts per message were possible.)

Qualitative Results: Students’ Analyses & Questionnaires Mixed, but showing some perceived benefit to this activity. Future research may attempt to investigate this related to learning preferences, self-concept, open-mindedness, and language skills, to name a few.

“After reading my postings, I realized that I did not go as in depth as I would have liked to.”

“…I feel my thought process is changing and my responses are improving in quality.”

“I have a tendency to offer unsolicited observations… “I have a tendency to offer unsolicited observations…. Apparently I tend to tie almost everything into past experiences and I have become very opinionated… Hopefully this will change because of this reflection.”

“In some ways, I find the rubric limiting in that I feel I must consciously pay attention to how much I am posting and of what type I am posting. I think I would post more if it were more free-flowing and not proscribed.”

Quantitative Results: Discussion Analysis by Researchers

Unitizing Students looked at a message as the unit. Researchers looked at the sentence. This may have decreased the percentage of High Level units in both the pre- and post-treatment data.

Coding Two coders Third as a tie-breakers Discussions to resolve 3-way disputes

Intercoder Reliability Function: 88% Skill: 62% Level: 92%

What do you think? Effect of self-categorization on Total number of units Percent cognitive Percent inference Percent analysis / elaboration Percent high level

Quantitative Results : Discussion Analysis by Researchers

Number of units Decreased 48.7 to 35.6 per person (Average unit length decreased from 17.3 words to 13.2 words.)

Function Cognitive: 90% Pre, 83% Post (p=.018) Metacognitve: 7% Pre, 5% Post (p=.143) Organizational: 2% Pre, 5% Post (p=.003) Social: 2% Pre, 7% Post (p=.001)

Skill Inference: 10% Pre, 6% Post (p=.049) Analysis: 18% Pre, 7% Post (p<.001) Judgment: 16% Pre, 29% Post (p<.001) Strategy: 1% Pre, 5% Post (p=.024) Elem. Clarification: 55% Pre, 54% Post (p=.417)

Level High: 13% Pre, 8% Post (p=.007) Low: 87% Pre, 92% Post

Percent cognitive

Conclusions This self-categorization reduced the quantity and quality of cognitive dialog, likely due to an increased hesitancy brought on by an increased awareness that each message would be scrutinized. This activity should not be used as a means to improve cognitive dialog.

Conclusions Some participants reportedly became aware of a need to change that did not manifest itself in their discussion.

Recommendations Attempts to improve cognitive dialog should focus on interventions that do not emphasize student performance assessment.

Recommendations for Future Studies Control group Longer course than 5 weeks Allowing acclimation and additional discussion experience prior to treatment Practice, successive feedback

References Rose, M. A. 2002. Cognitive dialogue, interaction patterns, and perceptions of graduate students in an online conferencing environment under collaborative and cooperative structures. Ed.D. diss., Indiana University, Bloomington. Henri, F. 1992. Computer conferencing and content analysis. In Collaborative learning through computer conferencing: The Najaden papers ed. A. R. Kaye, 117-36. New York: Springer-Verlag. Henri, F., and C. R. Rigault. 1996. Collaborative distance learning and computer conferencing. In Advanced educational technology: Research issues and future potential. Vol. 128, Computer and Systems Sciences ed. C. O’Malley, 146-61. New York: Springer-Verlag.