21-25 Feb 2001SIGCSE 2001 Integrating Testing into the Curriculum – Arsenic in Small Doses Edward L. Jones CIS Department Florida A&M University Tallahassee, FL USA

21-25 Feb 2001SIGCSE 2001 Lessons from ACE2000 Practice (10 slides) SPRAE framework, carefully explained Arsenic pills: –7 things to do –Automated grading –Student mentoring Tester certification in the experience factory On-going work

21-25 Feb 2001SIGCSE 2001 Motivation Industry need Testing experience adds value Avoid isolationist view – confine to course Salt courses with test related experiences Provide opportunity for advanced study Impact our curriculum

21-25 Feb 2001SIGCSE 2001 Possible Approaches Teach a course in software testing Expose student work to rigorous testing Train selected students Opportunistic insertion of testing into existing courses Formal training environment

21-25 Feb 2001SIGCSE 2001 The Holistic Approach Software Test Lab Core Curriculum Elective Testing Course Testing In Action (Automated Grading) SPRAE Testing Framework

21-25 Feb 2001SIGCSE 2001 What is Meant by Holistic Unifying framework – what’s important Experience-based Experiences aligned to framework Multiple experiences in different contexts Multiple means of delivery

21-25 Feb 2001SIGCSE 2001 The SPRAE Framework S Specification – the basis for testing P Premeditation – a systematic process R Repeatability – tester independence A Accountability – documented process, results E Economy – cost-effective practices

21-25 Feb 2001SIGCSE 2001 Test Life Cycle Analysis Design Implementation Execution Evaluation Specification Test Plan Test Script, Data, Driver Defect Data Problem Reports Test Results, Log Test Cases

21-25 Feb 2001SIGCSE 2001 The Software Testing Course 80% practice, 20% theory 2/3 fine-grained testing (functions) 1/3 object and application testing Test cases, drivers, and scripts Decision tables the "formalism" of choice Functional, boundary, white-box testing Evaluation via coverage & error seeding

21-25 Feb 2001SIGCSE 2001 Course -- Lessons Learned Advantages –In-depth, continuous concept treatment –Complement to other software skills –Basis for future advanced work Deficiencies –Why Johnny can’t test? Programming skills –Not a mainstream course available to all –Students compartmentalize course content

21-25 Feb 2001SIGCSE 2001 Testing in Action – Automated Program Grading Prepare AssignmentImplement GraderGrade Programs Write Program Grading Log Grade Report Student Program Assignment Specification Assignment Specification Test plan Test cases Test driver Checker script Automated Grader Overhead

21-25 Feb 2001SIGCSE 2001 Results -- Automated Program Grading Student shock & outrage at exactness Behavior modification – tester mindset Extra work on teacher –Specification must be better than average! –Practice what you preach (test process) –Cost amortized via similar assignment styles Is grader too strict for CS1/CS2? Selling colleagues on the idea!

21-25 Feb 2001SIGCSE 2001 Opportunistic Insertion into Existing Courses Risk of diluting course content Hard to transfer to colleagues (comfort zone with subject matter) Value-Added approach –Testing strengthens other skills –Testing brings objectivity to student Some examples

21-25 Feb 2001SIGCSE 2001 Simple Things to Try Grade another student’s program and justify grade -- in writing. (Group) Develop test cases before writing program. Treasure Hunt. Find seeded errors, document process followed, give evidence of fix. Develop and sell certified components to be used in subsequent assignments. Blind testing. Write specification from executable.

21-25 Feb 2001SIGCSE 2001 Mentoring Students - Experiences Hand picked students -- paid Series of testing projects Skills taught as needed Student access to examples from courses, other students MANAGEMENT NIGHTHMARE! Structured environment and process needed!

21-25 Feb 2001SIGCSE 2001 The Software TestLab Environment for discovery & learning An evolving laboratory Tools & Tutorials Staff (students, faculty) Test problem/solution test bed Students participate in the evolution Feedback on lab resources Create/Refine resources Technology insertion into classroom Vison

21-25 Feb 2001SIGCSE 2001 TestLab -- The Big Picture Marketing Interns New Hires Support Curriculum Students Software TestLab Research Publications Corporate Sponsors

21-25 Feb 2001SIGCSE 2001 Student Mentorship Model Manage skill development Set clear achievement goals Key Practices x Competency Levels Certify levels of progression Enable student-student mentoring Establish recognition program Mentorship

21-25 Feb 2001SIGCSE 2001 Key Practices Practitioner - performs defined test. Builder - constructs test “machinery” Designer - designs test cases. Analyst - determines test needs, strategy. Inspector - verifies correct process, results. Environmentalist - establishes & maintains the test environment. Specialist - performs entire test life cycle. Mentorship

21-25 Feb 2001SIGCSE 2001 Competency Levels Practitioner Test Designer Test Analyst Test Inspector Test Environmentalist Test SPECIALIST 1 Test Builder Test Practitioner Mentorship

21-25 Feb 2001SIGCSE 2001 Test Specialist I Practitioner I - Run function test script & document test results. Builder I - Develop function test driver. Designer I - Develop functional and boundary test cases from decision table. Analyst I - Develop decision table from functional specification. Inspector I - Verify adherence to function test process and standards. Mentorship

21-25 Feb 2001SIGCSE 2001 Certification Testbed Repository of testing problems Repository of students’ test artifacts Best in class promoted to solutions testbed Deficient solutions used for future tester certification Infrastructure

21-25 Feb 2001SIGCSE 2001 Conclusions Testing must compete for air time with existing subject matter Opportunities to insert testing exist Testing can bring added value to course Need rigorous study of value-added hypothesis Biggest job may be selling colleagues

21-25 Feb 2001SIGCSE 2001 On-Going & Future Work Evolve TestLab Mentorship Model –Experience Ladder & Certification –Evolving Problem/Solution Artifacts Careful study of value-added hypothesis Exploit automated grading – student mindset Disseminate Results

21-25 Feb 2001SIGCSE 2001 Questions?

21-25 Feb 2001SIGCSE 2001 Thank You

21-25 Feb 2001SIGCSE 2001 Training Sequence (1) TestLab Environment Basic Training Unix basics C++/language refresher Encapsulation of function under test Repositories Infrastructure

21-25 Feb 2001SIGCSE 2001 Training Sequence (2) Black-Box Function (unit) Testing Specification Stimulus-response test cases Function Test driver (5 styles) Test driver input/results files Test script (set-up + perform) Test log Decision tables Functional (partition) test case design Boundary test case design Infrastructure

21-25 Feb 2001SIGCSE 2001 Training Sequence (3) Black-Box Object Testing Specification of object’s methods Analysis of method’s stimulus-response Test planning Test cases = method + stimulus + response Object Test driver Object Test driver input/results files Operational test scenarios Infrastructure

21-25 Feb 2001SIGCSE 2001 Training Sequence (4) White-Box Function Testing Control flow graph basics Coverage criteria/measures Instrumentation for data collection Use of in-house coverage tools Coverage analysis White-box coverage during black-box test Supplemental white-box test cases Infrastructure

21-25 Feb 2001SIGCSE 2001 Training Sequence (5) Clear-Box Object Testing Goal is to overcome information hiding Test windows into internal object state set_state ( … ) get_state ( … ) Test case Stimulus: precondition + method-stimulus Response:method-response + postcondition Clear-box object test driver Clear-box test oracles Infrastructure

21-25 Feb 2001SIGCSE 2001 Training Sequence TestLab Environment Basic Training Black-Box Function (unit) Testing Black-Box Object Testing White-Box Function Testing Clear-Box Object Testing Infrastructure

21-25 Feb 2001SIGCSE 2001 TestLab Infrastructure SPRAE Framework / Lifecycle Software Testing Course Training Sequence Standards & Techniques Student Mentorship Model Problem & Solution Testbeds Status

21-25 Feb 2001SIGCSE 2001 Standard Products Specification (narrative, semiformal) Decision Table Test Plan Test Script Test Driver Test Driver Input File Test Results File Test Log Infrastructure

21-25 Feb 2001SIGCSE 2001 Techniques Functional Equivalence Partitioning Boundary Value Analysis Function Encapsulation Control Flow Analysis Error-seeding for tester certification Infrastructure

21-25 Feb 2001SIGCSE 2001 What Makes It Holistic? Testing an integral part of curriculum Goal is multiple test experiences At least one experience in each course Repetition and reinforcement Accumulation of experiences Coverage of test lifecycle

21-25 Feb 2001SIGCSE 2001 Course Outline 1 Course Overview 2 Software Quality Assurance 3 The Practice of Testing 4 Specification-Driven Testing 5 Boundary Testing 6 Measuring Test Effectiveness 7 Testing Object-Oriented Software 8 Application Testing 9 Course Review & Wrap-Up

21-25 Feb 2001SIGCSE 2001 Automation Issues Does the teacher have the time? Is grader too strict for CS1/CS2? Additional automation to lower cost. The trap: “just a little more automation” Selling colleagues on the idea!

21-25 Feb 2001SIGCSE 2001 Outline The Holistic Approach The SPRAE Testing Framework The Software Testing Course Automated Program Grading Opportunistic Insertion The Software Test Lab Conclusions / Future Work

21-25 Feb 2001SIGCSE 2001 Example - Pay (S) Specification: Compute pay for an employee, given Hours worked and hourly pay Rate; overtime is 1.5 times hourly Rate, for Hours above 40. Hours Rate Pay Compute Pay

21-25 Feb 2001SIGCSE 2001 Principle P Premeditation: Use a systematic process to devise test cases based on the specification. Our Technique: Decision analysis -- identify behaviors One test case per behavior Determine expected result

21-25 Feb 2001SIGCSE 2001 Example - Pay Test Case Design: Decision Table. Columns identify behaviors to test.

21-25 Feb 2001SIGCSE 2001 Principle R Repeatability: Processes for test case creation and test execution must yield equivalent results, independently of the tester.

21-25 Feb 2001SIGCSE 2001 Principle A Accountability: Keep records that document test process and artifacts. Documentation answers: What tests were planned? Which tests were conducted? Who did what testing, and when? What were the results? How were the results interpreted?

21-25 Feb 2001SIGCSE 2001 Example - Pay Repeatability/Accountability:

21-25 Feb 2001SIGCSE 2001 Principle E Economy: Test activities must not require excessive time or effort. Automation Test drivers (classical tool) Simplified processes for Test case generation Data collection