 When Certainty is Uncertain A presentation of “Linguistic Certainty, Task Certainty and Academic Success,” a research paper written for Cleveland State University’s COM 633. Julie A Cajigas

Rationale  Why CATA?  Short et al. (2010) Validity Piece  How has LIWC been validated? Is that enough?

What is Certainty?  Certainty has been examined throughout the Communication, Social Science and Mathematical literatures.  Types of uncertainty/certainty:  Relational Uncertainty/Certainty  Task Uncertainty/Certainty  Cognitive Certainty  Mathematical Certainty  Linguistic Certainty?

Certainty  In examining COM 101 Journals, we chose Task Certainty.  Task uncertainty refers to the difficulty and variability of the work undertaken by an organizational unit. (Van de Ven, 1976)  Rationale for choosing task certainty over relational or other types of certainty.  Van de Ven’s Task Certainty Scale

Certainty  Certainty: The ability to predict outcomes in a given context.  What do we think of this definition?

Certainty in Academic Writing  Previously used in several research studies:  Hyland and Milton (1997)  Kennison (2003)  Rubin (2010)  Schmader et al. (2010)  Carroll (2007)  Also looked at LIWC Linguistic Variables

Certainty and LIWC  Kennison 2003  Found that LIWC certainty had a negative relationship with the subscales of the California Critical Thinking Skills Test.  Carroll 2007  Saw an insignificant decrease in LIWC certainty over the course of an academic semester in student writing.  This begs the question: What is LIWC measuring exactly?

LIWC Dictionary

LIWC Certainty  Words included in the Pennebaker internal dictionary for certainty are included below.  Unlike other categories, which contain from words in LIWC 2001, the certainty variable contains relatively few words at 30, though it is nearly impossible to think of words that should be added to this list, giving the dictionary good face validity. Certainty words account for an average of 1.1% (SD.9) of words across 43 studies as reported in the LIWC 2001 manual (Pennebaker et al., 2001).  For the purposes of the present study, it is most telling to look at the average for control writing, which is conceptually the closest to the student’s journal writing. The average for control writing over the 43 studies analyzed was.70. The mean for the student journals here is 1.25% (SD =.96), which is very close to the grand mean for certainty words, and close to the mean for emotional writing, which is 1.4%.

LIWC related Hypotheses/Research Questions  H 1 Student task certainty for an assignment will have a positive relationship with written certainty, as measured in that assignment.  H 2 Student task certainty for an assignment will have a negative relationship with written tentativeness and discrepancy, as measured in that assignment.

Research Questions 1-2 Because of the following points from the literature, research questions emerged. Certainty has been shown to decrease with familiarity, critical thinking and language proficiency, which is counterintuitive, and there are a relatively small number of studies with this effect (Hyland & Milton, 1997; Kennison, 2003; Carroll, 2007)  RQ 1a-b How is written/linguistic certainty (as measured by LIWC) related to a. assignment grade expectations and b. course grade expectations?  RQ 2 How is written/linguistic certainty related to the actual assignment grades?

Research Question 3 Several studies examine Tentativeness, Discrepancy and Certainty as measured by LIWC. Some add them together under CogMech and some use the individually. These seem like concepts which might have an inverse relationship.  RQ 3 How are linguistic tentativeness and discrepancy related to linguistic certainty?

Research Question 4 Carroll (2007) showed that Linguistic Variables including Word Count, Words Per Sentence, Big Words etc., had a relationship with Student Achievement.  RQ 4a-c How are student scores on LIWC linguistic variables related to student a. assignment grade expectations, b. course grade expectations and c. actual assignment grade?

Research Question 5 Van De Venn (1976) utilizes task certainty to predict task outcomes in organizational settings. Does Task Certainty relate to academic outcomes?  RQ 5a-b How is student task certainty related to a. assignment grade expectations, b. course grade expectations and c. actual assignment grade?

Variables  Written/Linguistic Certainty  Conceptually: The use of sure language in written communication.  Operationally: The LIWC Measure of Certainty  Task Certainty  Conceptually: Task uncertainty refers to the difficulty and variability of the work undertaken by an organizational unit. (Van de Ven, 1976)  Operationally: Van De Ven’s (1976) Task Uncertainty Index (utilized measures 1, 4, 5; with Cronbach’s Alpha of.71)  Academic Outcomes  Conceptually: The outcome of an academic effort.  Operationally: The student’s assignment grades, predicted and actual, the student’s predicted course grade.

Sampling (The Easy Way)  Population: COM 101 Sections at Cleveland State University during Fall  Sampling Method: Census of Students in above population.

Sampling (Actual)  Actual Population: College students in courses which require academic writing participation around the nation.  Actual Sampling Method: a convenience sample of COM 101 students at Cleveland State University during the Fall semester students completed the questionnaire and survey responses.  The unit of measurement for the content analysis was the journal, while the unit for the overall study was the student.

Sampling (Ideal)  Population: Students in colleges and universities nationally.  Sampling Method: Systematic Random Sampling  Generate a list of all active college & university addresses, randomize them with a random number generator, and select them at intervals.

Procedure  Students in COM 101 were asked to submit an electronic copy of their week 9 journals, and to complete a task certainty instrument, which included the Van De Ven Task Uncertainty Index, two of Ashill& Jobber’s measures of Environmental Certainty, and a number of questions about the academic outcomes. 47 students completed the survey and turned in their journals.  Journals were cleaned up for spelling errors and run through the LIWC2001 CATA content analysis program.

Methods  Van De Ven’s (1976) measure of Task Uncertainty was coded and reverse coded so that certainty would be represented by a higher score. Measures from this scale were converted into Z scores and measured for scale reliability. The combination of questions with the highest reliability (Surety of finding info, surety of outcome and time it takes to be sure on the outcome) with a cronbach’s alpha of.71, were summed and became the new variable “TaskCert”

Methods (cont)  Q9, which is the student’s predicted grade is used in the analysis, along with Q17 which is the student’s predicted semester grade.  Student grades on these journals have not yet been received, and so hypotheses and RQ’s that require them are left out of the following results section.

Results: H1 H 1 Student task certainty for an assignment will have a positive relationship with written certainty, as measured in that assignment. NOT SUPPORTED

Results H2  H 2 Student task certainty for an assignment will have a negative relationship with written tentativeness and discrepancy, as measured in that assignment. NOT SUPPORTED

RQ 1  RQ 1a-b How is written/linguistic certainty (as measured by LIWC) related to a. assignment grade expectations and b. course grade expectations? (No Significant Relationships)

RQ3  RQ 3 How are linguistic tentativeness and discrepancy related to linguistic certainty? (No Significant Relationship)

RQ 4 RQ 4a-c How are student scores on LIWC linguistic variables related to student a. assignment grade expectations, b. course grade expectations and c. actual assignment grade?

RQ 5  RQ 5a-b How is student task certainty related to a. assignment grade expectations, b. course grade expectations and c. actual assignment grade?

Validation This study is an attempt to validate a current measure (LIWC certainty) using criterion validity or predictive validity: Short et al. (2010) define predictive validity as “the extent to which the operationalization predicts other constructs consistent with theoretically derived a priori expectations.” Neuendorf (2002, 115) defines criterion validity as “the extent to which a measure taps an established standard or important behavior that is external to the measure.”

External Validity  The current study, with an N of 47 gathered via non- random sampling method has a low external validity.  We should not attempt to generalize this study to a larger population, but simply to repeat the study with funding and time allowing in a more externally valid way.

Face Validity  The face validity of this study is high, but has some challenges.  Face validity is high because “on the face of things” both the task uncertainty measure and the certainty measure in LIWC seem to be measuring the same thing.  One threat to face validity is the knowledge that LIWC was validated using emotional narratives rather than academic papers. This could reduce the validity of our overall argument.

Content Validity  “The extent to which the measure reflects the full domain of the concept being measured.”  While task certainty is a validated measure, LIWC certainty may only measure a small part of what we would consider “certainty.” While task certainty is very specific in what it applies to, LIWC certainty is not.

Construct Validity  As we saw in the results, Task Certainty has strong Construct Validity, with significant relationships to the student’s predicted grade, whereas LIWC certainty has little construct validity based on several of the research questions.

Limitations  Lack of true conceptual definition for LIWC Certainty.  Idea for future study of LIWC Certainty  Small number of respondents/sampling technique.  Limited variance in certainty measures because it was the end of the semester, task has been repeated many times.  Additional limitations?

Discussion  Still working on this part – let’s discuss!

 Fin Thank you all for your help and friendship this semester!