1. Improving Student Engagement Through Audience Response Systems
November 20, 2016
Improving Student Engagement Through Audience Response Systems
David P. Stevens, Ph.D.
Associate Professor, Department of Management
University of Louisiana at Lafayette
This page is an opening page option.

2. Previous Research Summary of Findings
DSI 2016
Previous Research Summary of Findings
Effectiveness of clickers is common theme
No research found quantifying perceptions of students regarding course and instructor
Previous clicker research:
Help students engage
Improves attention during lectures
Improves class attendance
Increases class participation
Positively impacts student “learning experience”, although grades were not higher than in equivalent courses without clickers
Allows instructors to obtain immediate feedback
Facilitates introduction of more material during class
In some studies, students actually participated less and relied more on non-verbal communication via the clicker 1
Research for a reason.

3. Turning Point’s Audience Response Device a.k.a. “clicker”
DSI 2016
Turning Point’s Audience Response Device a.k.a. “clicker”
Hardware was loaned to the college of business 2
Research for a reason.

4. Method Background Info
DSI 2016
Method Background Info
Medium sized, 4-year university in Lafayette, LA
Student enrollment of approximately 18,000
Courses included:
Principles of marketing: 64 students,
Leadership (management): 40 students,
Operations management: 74 students, and
Introductory business statistics: 2 sections of 41 and 60 students (taught by 2 separate instructors)
Very small numbers of students in 3 courses resulted in those student responses being dropped from the study, resulting in a total of 279 students in 5 courses
Survey consisting of 3 sets of questions 3
Research for a reason.

5. Method Survey Questions
DSI 2016
Method Survey Questions
Three sets of questions:
Subset of questions from university’s standardized student evaluation of instruction (SEI), to gauge student perceptions of course and instructor
Set of questions regarding usefulness and effectiveness of clickers
Demographic questions
SEI questions answered using 5-point Likert scale with 1 = strongly agree, 2 = agree, 3 = neutral, 4 = disagree, and 5 = strongly disagree
Clicker effectiveness questions (e.g. ease of use and impact on the classroom) answered using 5-point Likert scale with 1 = excellent, 2 = very good, 3 = good, 4 = fair, and 5 = poor
Clicker effectiveness is treated as an independent variable
Research sought to determine the impact of clicker usage on the SEI measures 4
Research for a reason.

6. Method Effect Size Index, f
DSI 2016
Method Effect Size Index, f
f = square root(eta-squared/(1 – eta-squared))
Numerator is proportion of variation explained by independent variable,
Denominator is proportion of variation not explained
As explained variation increases, f increases
As explained variation decreases, f approaches its minimum value of 0
Cohen (1988) suggested the following guidelines for interpreting f:
f = 0.10 is a “small” effect, analogous to the differences in heights between 15 and 16 year-old girls
f = .25 is a “medium” effect, analogous to the differences in heights of 14 and 18 year-old girls
f = .40 is a “large” effect, analogous to the differences in ages of 13 and 18 year-old girls 5
Research for a reason.

7. Results Descriptive Statistics
DSI 2016
Results Descriptive Statistics
Smaller scores indicate stronger positive responses (these are desirable)
Larger scores indicate stronger negative responses 6
Research for a reason.

8. Results Descriptive Statistics, continued
DSI 2016
Results Descriptive Statistics, continued 7
Research for a reason.

9. Results What the Descriptive Statistics Say
DSI 2016
Results What the Descriptive Statistics Say
All questions have means on the positive end (closer to 1) of the Likert scale
Students have very favorable opinions of all clicker questions; more specifically:
Usability of clickers
Overall opinions of clickers in the classroom
Specific ways clickers help the student
Averages on SEI questions are better than for the same instructor in the same course but without clickers 8
Research for a reason.

10. Results Correlations with SEI Questions
DSI 2016
Results Correlations with SEI Questions
Five SEI survey questions to be assessed:
Instructor has good written communication skills
Instructor has good verbal communication skills
Teaching methods in this class are effective
I would recommend this instructor
I would recommend this course
These 5 questions are the core of “performance evaluation” in terms of teaching quality
Correlations between the 13 clicker questions and SEI responses were calculated 9
Research for a reason.

11. Results Correlation Values
DSI 2016
Results Correlation Values 10
Research for a reason.

12. Results What the Correlations Mean
DSI 2016
Results What the Correlations Mean
Almost all correlations are statistically significant
Values of the correlations range from weak (around .25) to moderate (around .50), with 2 exceptions:
“Clickers make me more likely to attend class” has no statistically significant correlations with any of the SEI measures
“Clicker use helps me focus” has sig. values greater than 0, but less than .10 for the first two and last two SEI questions
None of the correlations are negative
Correlation measures linear relationship
Eta-squared measures curvilinear relationship 11
Research for a reason.

13. Results Eta-squared and f Values: Questions 1 through 6
DSI 2016
Results Eta-squared and f Values: Questions 1 through 6 12
Research for a reason.

14. Results Eta-squared and f Values: Questions 7 through 13
DSI 2016
Results Eta-squared and f Values: Questions 7 through 13 13
Research for a reason.

15. Results Interpreting the Effect Sizes
DSI 2016
Results Interpreting the Effect Sizes
Significant chi-square results indicate the two questions are related and not independent
Student perceptions about overall ease of use of clickers has a medium effect on the students opinions of the teaching methods and their recommendation of the course
Student opinions of overall usefulness of clickers has a large effect on their opinions about instructor’s verbal communication skills, the effectiveness of teaching methods, and recommendation of both course and instructor
Overall, small/medium/large effect sizes can be thought of as analogous to weak/moderate/strong correlations 14
Research for a reason.

16. Results Differences by Gender
DSI 2016
Results Differences by Gender
Responses from 125 females and 153 males
Females gave smaller and statistically significantly different (less than alpha = .10) scores (meaning a stronger agreement with the statement) for each of the following:
More likely to attend class
Clickers help me focus
Use of clickers makes me read textbook
Other 10 questions regarding clickers had 0 statistically significant differences between females and males 15
Research for a reason.

17. Discussion Summary Points
DSI 2016
Discussion Summary Points
Clicker usage provides significant positive benefits as indicated by questions rating:
1) usefulness of clickers,
2) reading the textbook,
3) performing better on exams, and
4) helping students learn
All clicker questions have averages on the low end of the scale, indicating very favorable opinions of clickers, the instructor, and the course; this result was consistent with correlation results
“Clicker usefulness” has a large effect on every one of the five SEI questions: instructor has good written and verbal communication skills, teaching methods are effective, and would recommend course and instructor
Indicates instructors can improve student evaluations by use of clickers 16
Research for a reason.

18. Discussion Summary Points
DSI 2016
Discussion Summary Points
Impact of clickers is especially impressive given the limited use of clickers in the classroom
One instructor used only to review material at the end of a chapter/unit
One instructor used only for quizzes, also given at end of chapter/unit
Total time utilized was less than 20 minutes of class time per week
Quantitative and non-quantitative courses included 17
Research for a reason.

19. Limitations and Future Research
DSI 2016
Limitations and Future Research
Study at only one university, and only from the college of business
Most courses at most colleges do not use clickers, so results may be due to use of a novel technology
Many statistically significant results not reported because there were an insufficient number of responses in certain categories, invaliding the chi-square test; having more data could allow those results to be included
Future research could:
compare clicker effectiveness across schools and/or colleges
follow students for more than one semester
resulting effect size index measurements comparable across studies 18
Research for a reason.

20. Thank You for Attending Today’s Presentation
David P. Stevens, Ph.D. 2