1. Innovation through Research and Development
Kristin McNary Product Team Manager
IT | Programming | Web Development | MIS
Cengage Learning
Dr. Kyla A. McMullen Assistant Professor Computer & Information Science & Engineering University of Florida

2. Synergistic Interests
Intro to Programming course at UF
(500 Students)
Scalable
Interactive
Usable
Adaptable

3. The CS Education Landscape is Changing
Undergraduate CS enrollment is soaring
Technology-based instruction is increasingly commonplace
CS education pedagogy is evolving
active learning
paired-programming
[1] CRA Taulbee Survey (2015)

4. Overarching Question: What drives learning efficacy?
“How can new educational technology can facilitate active learning and paired- programming methodologies in the classroom?”
Best practices
What works
What doesn’t
User studies with members of the target population

5. Specific Questions
How usable is this educational technology interface?
System Suability Scale (SUS) [2]
What features are desirable in an educational technology interface?
What features are undesirable in an educational technology interface?
How did performance compare to the traditional means?

6. Experiment Design (Method)
3 Leading Educational Technology Suppliers
Balanced, within-subject design
3 Sessions
28 Participants from Introduction to Programming Course
A to E students
Paired by ability [3]
$50 or Extra Course Credit Compensation
Unexplored Topics
Exception Handling, Recursion, Generic Methods

7. Experiment Design (Procedure)
Students were paired during session 1
Participated in active learning using the given interface
20 minutes of learning the content
Followed by paired-programming
Driver and navigator
Switching every 15 minutes
Students submitted solutions
Completed the SUS Evaluation

8. Experiment Design (Data Collection & Evaluation Metrics)
Video, voice, solution, survey
Collected – Facial sentiment, Speech Sentiment, code analysis, impressions of the syste
Really, the solution and survey turned out to be the most informative measures

9. Best Practices – Usability
Students agreed that the educational technology interface was:
was easier to use
was well integrated
had more useful built-in features
could be learned quickly
gave students more confidence
was a better choice for first-time programmers
had more support for active learning
was more “attention-grabbing” during tasks
did not require students to have exceptional programming abilities in order to use the interface.
As compared to the traditional means

10. Best Practices – Desirable Features
Integrated textbook and coding environment
Error identification
Active feedback
A simple, and easy to use interface
Good coding examples in the book

11. Best Practices – Undesirable Features
Uninformative error messages
Confusing interface design choices
Buttons, Submitting work
Slow compile time
Lack of informative figures and examples
Correct/Conventional auto-indentation
Compiler errors separated from code
Difficult-to-navigate

12. Best Practices / Recommendations
Closely follow IDE conventions
Ctrl-F for finding content
Auto indentation
Syntax highlighting
Auto complete
Brackets, parentheses, method names, variable names
Improved error explanation
Results console before submitting
Button explanation
Ability to adjust difficulty level
Better examples
Hints
Largely in line with dislikes

13. Future Work Replication with more students Direct platform comparison
Longer/more extensive coding problems to solve

14. References
[1] Stuart Zweben. Computing degree and enrollment trends (from the taulbee survey). Computing Degree and Enrollment. Accessed September, 15, 2016.
[2] Brooke, J. (1996). SUS: A “Quick and Dirty” Usability Scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability Evaluation in Industry, 189–194. Bristol, PA: Taylor & Francis.
[3] Bevan, Jennifer, Linda Werner, and Charlie McDowell. "Guidelines for the use of pair programming in a freshman programming class." Software Engineering Education and Training, 2002.(CSEE&T 2002). Proceedings. 15th Conference on. IEEE, 2002.

15. Kristin McNary Product Team Manager
IT | Programming | Web Development | MIS
Cengage Learning
Dr. Kyla A. McMullen Assistant Professor Computer & Information Science & Engineering University of Florida