1. The Impact of Lexical Complexity on the Public’s Understanding of
Governmental Communication
Patrick J. Lyons, III · PAD Capstone Project, May 2018· Project Advisor: Dr. Atta Ceesay
Abstract
Complexity of language and specific jargon can make documents challenging to comprehend.
Government has lagged in implementing these recommended guidelines.
This research aims to determine if modified documents: a) are easier to read; b) are easier to comprehend; and c) require less assistance to understand than the original, unmodified documents
Three focus groups were conducted and participants read modified and unmodified documents
Data analysis included descriptive statistics, sign tests, effect sizes, paired sample t-tests, and qualitative explanations to determine outcomes.
Results indicated that all documents were significantly more readable with moderate to large effect sizes found. Significant differences were not found, however, for improved comprehension and less required assistance to understand.
Future steps should be taken to determine more feasible and parsimonious ways to improve readability and perhaps more importantly, comprehension.
Data Collection Methods
Focus groups read unmodified and modified documents from Social Security Administration
Completed a survey asking for demographic information, impressions of readability on a likert scale, and responses to comprehension questions
Groups qualitatively discussed their perceptions of the modified versus unmodified documents
Sign Test Analyses of Employee Assistance
count p
Focus Group 1 (SSI) 4
0.69
Focus Group 2 (RIB) 9
0.25
Focus Group 3 (MED) 8
0.36
Results
Sign Test Analyses of Friend/Family Assistance
Participant Demographics
count p
Focus Group 1 (SSI) 5
0.19
Focus Group 2 (RIB) 8
0.14
Focus Group 3 (MED) 6
0.34
Variable
Percentage of Focus Group 1
Percentage of Focus Group 2
Percentage of Focus Group 3
Education level
Some high school
High School
Some College
Associate’s degree
Bachelor’s degree
Post graduate degree
Gender
Male
Female
Age range
Under 19
20-25
25-30
30-40
40-50
50-60
Race
White
Black
Hispanic
Asian
Non-latino
Native American
Native Alaskan
Mixed race
Missing 0%
14%
43%
21% 7%
79%
36%
83%
17%
42%
58%
50% 8%
100%
25%
19% 6%
13%
Purpose
The Purpose of the study is to determine if recommended practices from the Plain Writing Act are implemented in unmodified documents from Social Security. Specifically, the study seek to answer:
What does the public think about the complexity of the documents?
Are modified documents significantly more readable than unmodified documents?
Are participants significantly less likely to seek assistance from Social Security employees or family members/friends in order to understand the modified than the unmodified document?
Does reading the modified document significantly improve participants’ comprehension?
Sign Test Analyses of Readability
count p
Cohen’s d
Focus Group 1 (SSI) 8
.035
0.73
Focus Group 2 (RIB) 11
.0005
2.16
Focus Group 3 (MED) 12
.02
0.61
Review of Literature
Average Percentage of Correct Responses for Comprehension
The research has helped to pave the way for more accessible documents.
Nonetheless, gaps in the literature are still apparent.
One limitation is that many articles focus on the opinions of the researchers rather than the specific readability guidelines and formulas that have been set forth. Using a standardized way to improve readability and comprehension is necessary in order to reduce objectivity.
Second, much of the research that has been done to modify documents has not considered whether or not the document still meets necessary legal standards. If this is not accomplished, the document may no longer be valid.
Unmodified
Modified n M
Focus Group 1 (SSI) 15
57%
60%
Focus Group 2 (RIB) 14
66%
Focus Group 3 (MED) 16
49%
51%
Study Design
Measured the complexity of Retirement, Medicare, and Supplemental Security Income documents using the Flesch-Kincaid tool (Flesch, 1948)
Rewrote the documents using guidelines from the Plain Writing Act
Had each document reviewed by a judge to determine if it met the same legal standards as the original
Ran three focus groups (one for each document) to determine qualitative perceptions
Assessed for readability using a likert scale
Assessed for comprehension using 6-8 questions for each document
Paired Samples t-test Analyses for comprehension
Conclusions / Future Research
Flesch-Kincaid tool is efficient and cost effective to determine readability
Modifying documents is time consuming, and a cost/benefit analysis needs to be conducted
The study found that readability was significantly improved with modified documents, but comprehension was not
Limitations include small sample size, omission of cloze comprehension assessment, reading similar documents twice
Future research should determine effective ways to improve readability and the impact this has on employee efficiency
Should also consider high-stakes documents and stress and the influence that could have on comprehension in addition to readability
Unmodified
Modified n M SD t df p
Focus Group 1 (SSI) 13
0.56
0.05
0.59
0.06
0.88 12
0.40
Focus Group 2 (RIB)
0.03
0.63
0.96
0.36
Focus Group 3 (MED) 15
0.51
0.09
0.52
0.07
0.22 14
0.83
Sample Selection
Convenience samples
2 focus groups at SUNY Buffalo State
1 focus group at Canisius High School