1. Test-Driven Learning in Early Programming Courses David Janzen California Polytechnic State University San Luis Obispo, CA Hossein Saiedian University of Kansas Lawrence, KS SIGCSE’08

2. I Have a Dream That all students will some day… –learn to design and test as they learn to program “... we have as many testers as we have developers. And testers spend all their time testing, and developers spend half their time testing.” Bill Gates in Information Week, May 2002 SIGCSE’08

3. How to Accomplish the Dream? Test-Driven Development TDD is a design (and testing) approach involving short, rapid iterations of RefactorUnit TestCode SIGCSE’08 Forces programmer to consider use of a method before implementation of the method

4. TDD in Action class RectangleTest { @Test public void testArea() { Rectangle r = new Rectangle(5,8); assertEquals(40, r.getArea()); } class Rectangle { private length, width; public Rectangle(int l,int w) { length = l; width = w; } public int getArea() { return length * width; } @Test public void testPerim(){ Rectangle r = new Rectangle(5,8); assertEquals(26,r.getPerim()); } public int getPerim() { return (length * 2) + (width * 2); } SIGCSE’08

5. “beyond the visionary phase and into the early mainstream” IEEE Software 2007 Adoption Evolution Objects Research Labs Dahl & Nygaard Industry Smalltalk, C++ Grad & Advanced UG First Year CS NASA Project Mercury Grad & Advanced UG First Year CS TDD Industry Any SW Process Industry XP Predicting the Future SIGCSE’08

6. Questions We Want To Answer 1.Can we teach automated unit testing in first year programming courses without sacrificing other first year topics? 2.Is it better to teach test-first (TDD) or test- last programming to first year students? 3.What do students think about TDD? SIGCSE’08

7. How Do We Teach TDD? Test-Driven Learning involves teaching testing by example System.out.println(“Result: ” + myObject.calculateResult()); assertEquals(25, myObject.calculateResult()); XXXXXXXXXXXXX SIGCSE’08

8. Experiment Design Proposed TDD/TDL integration into CS1 and CS2 courses at University of Kansas in 2006 as part of PhD research We don’t always get what we ask for –Given permission to teach two labs and assign two projects Weeks 6 and 7 in a 16-week CS1 course Weeks 1 and 3 in a 16-week CS2 course SIGCSE’08

9. TF/TL Instruction & Survey Project 1 Project 2 CS1 CS2 Test-First Test-Last Test-First Test-LastTest-First Survey Experiment Design 40 66 40 66 6 30 Project 1 solution with tests provided Project 1 solution with tests provided CS1 Students Assigned by ID CS2 Students Self-selected SIGCSE’08

10. Experiment Context CS1 and CS2 both used C++, vi/emacs, and g++ Project Topics1 st Project2 nd Project CS1Arrays, functions, text I/O Classes, text I/O CS2Classes, array- based list, text I/O Classes, pointer- based list, text I/O SIGCSE’08

11. Goal: Keep Unit Testing Simple #include class Exams {... }; int main() { run_tests(); } void run_tests() { { //test 1 Minimum of empty list is 0 Exams exam1; assert(exam1.getMin() == 0); } //test 1 { //test 2 Exams exam1; exam1.addExam(90); assert(exam1.getMin() == 90); } //test 2 } SIGCSE’08

12. CS1 Results: Grades & Effort Project grades were virtually the same Time spent on project was ~10% higher for test-first programmers Time (min)1 st Project2 nd Project TF mean285.29343.48 TL mean260.20308.40 % diff10%11% p-value.60.47 SIGCSE’08

13. CS1 Results: Unit Tests Test-first programmers wrote more tests on the first project, but fewer on second Remember that students switched approaches # asserts1 st Project2 nd Project TF mean5.851.89 TL mean3.853.10 % diff52%-39% p-value.11 SIGCSE’08

14. CS1 Results: Confidence No difference in confidence on first project On second project, test-first programmers were more confident of their solutions (P1 tests) 2 nd ProjectQualityChangesReuse TF mean3.983.903.69 TL mean3.253.062.88 % diff22%27%28% p-value.05.02 SIGCSE’08

15. CS1 Results: Student Opinions SIGCSE’08

16. CS2 Results: Grades Project grades were higher for test-first No significant difference in previous experience or gpa Score1 st Project2 nd Project TF mean88.8390.17 TL mean79.4772.83 % diff12%24% p-value.09.04 SIGCSE’08

17. CS2 Results: Effort Time spent on project was lower for test- first programmers Score1 st Project2 nd Project TF mean12.4216.00 TL mean14.7018.06 % diff-16%-11% p-value.30.42 SIGCSE’08

18. CS2 Results: Unit Tests Test-first programmers wrote more tests Not significant, high standard deviation Students not graded on tests # asserts1 st Project2 nd Project TF mean53.6747.67 TL mean13.4325.03 % diff300%90% p-value.08.39 SIGCSE’08

19. CS2 Results: Confidence No significant difference in confidence on either project Both ProjectsQualityChangesReuse TF mean3.834.003.67 TL mean3.213.893.71 % diff16%3%-1% p-value.14.87.74 SIGCSE’08

20. CS2 Results: Student Opinions SIGCSE’08

21. Reluctance to adopt test-first despite perceived benefits 11% vs 63% would choose test-first SIGCSE’08

22. Validity Concerns Study too small to draw any conclusions Lack of randomization in CS2 Confounding factors: –New concepts introduced with each project –Solutions with tests provided after 1 st project –Simple assert-based unit tests, no framework –C++ However, many of the results line up with results from industry and more advanced courses SIGCSE’08

23. Conclusions Unit testing can be integrated into CS1/2 courses without taking time away from other topics Test-first programmers tend to write more tests Test-first programmers may do as well (CS1) or better (CS2) on projects Test-first programmers may spend more time (CS1) or perhaps less time (CS2) on projects Early programmers are reluctant to use test-first, even after positive experiences with it SIGCSE’08

24. Future Work Currently conducting a study in CS1.5 –Java and JUnit –TDD from beginning –TDD throughout 10-week quarter Early Results –TDD takes no extra instruction time –Existing course materials can be rewritten w/TDD –TDD could influence course refactoring e.g. delay coverage of I/O SIGCSE’08

25. Questions Resources at http://www.simexusa.com/tdl/ If you are an early adopter of TDD in CS1/2, I’d like to know, email me at djanzen (at) calpoly.edu Acknowledgements –SIGCSE Special Projects Grant – seed funding –Lockheed Martin – current (future work) study SIGCSE’08