Florida A&M University Software Testing at Florida A&M University: Perspective, Direction and Opportunities Dr. Edward L. Jones CIS Department Florida A&M University Tallahassee, FL

Part I INTRODUCTION 3/19/2002 Testing at FAMU

My Institution 12,000 students total 600+ CIS majors 15 Faculty Heavy teaching Graduate program (20 students) Increasing research 3/19/2002 Testing at FAMU

My Experience 13 years Harris Corporation Software engineering technology $200M+ NASA project methodology expert software inspections, testing process training 20+ years university teaching 3 years focusing on testing 3/19/2002 Testing at FAMU

The Vision Quality Awareness in Curriculum Solid Framework Enhanced Student Skill Set Expanded Student Opportunities Industry Partnerships Research & Dissemination 3/19/2002 Testing at FAMU

Software TestLab - Context CS course Technology Transfer Testing Enthusiast Donated/public test artifact Artifact or tutorial Graduate Testing Research TestLab Artifacts artifact Test Arcade artifact artifact Certification / Tutorial Arcade Problems (+Meta-Data) TestLab Student Training 3/19/2002 Testing at FAMU

My Perspective Testing is not just for testers! In ideal world, fewer testers required Verification vs testing More skilled developers No silver bullet … just bricks Simple things provide leverage Testing in-the-small -- classroom the lab Technology transfer crucial 3/19/2002 Testing at FAMU

SPRAE Testing Framework S pecification – basis for testing P remeditation – follow a process R epeatability – tester independence A ccountability – documentation E conomy – cost effectiveness 3/19/2002 Testing at FAMU

Why SPRAE? A value system for testing practice Explains the “what” and “why” Sets goals for test experiences Each experience reinforces values A framework for life-long learning. 3/19/2002 Testing at FAMU

A Test Life Cycle Analysis Design Implementation Execution Evaluation Specification (Y/N) Test Plan/Strategy Test Script, Data, Driver Defect Data Problem Reports Test Results, Log Test Cases 3/19/2002 Testing at FAMU

(What can Academics Do Besides Teaching A Course?) Part II EDUCATION MISSION (What can Academics Do Besides Teaching A Course?) 3/19/2002 Testing at FAMU

Caught and Taught Caught Taught Strategy must contain both elements Attitudes Respect for consequence of not testing Taught Techniques for specific kinds of testing Basic analytical and programming skills Strategy must contain both elements 3/19/2002 Testing at FAMU

A Holistic Approach SPRAE Testing Framework Software Test Lab Core Course Experiences Elective Testing Course Testing In Action Experiences SPRAE Testing Framework 3/19/2002 Testing at FAMU

What Is Meant By “Holistic”? Testing an integral part of curriculum Multiple test experiences Experiences in each course Repetition and reinforcement Accumulation of experiences Experiences cover test lifecycle Holistic, NOT Exhaustive! 3/19/2002 Testing at FAMU

A Software Testing Course 80% practice, 20% theory 2/3 fine-grained testing (functions) 1/3 object and application testing Test cases, drivers, and scripts Decision tables the "formalism" Function, boundary, white-box testing Effectiveness: Coverage & error seeding 3/19/2002 Testing at FAMU

Course -- Lessons Learned Advantages A ”taste" of how hard testing really is Preparation for advanced study Rounds out analytical & programming skills Deficiencies Course not available to all students Students compartmentalize knowledge 3/19/2002 Testing at FAMU

Technology Transfer -- Program Grading Services Provided by TestLab students Experience designing automated tests Shell programming Instructor plays active part Refines specification Creates grading plan Benefits Faculty more inclined to incorporate testing “module” into course 3/19/2002 Testing at FAMU

Technology Transfer -- Plagiarism Deterrent Mutation analysis/error seeding applied to suspicious student program Student must debug modified program TestLab project – tunable error seeding Benefits Tool for teaching students to debug Less plagiarism … or … more skilled programmers 3/19/2002 Testing at FAMU

Core Course Test Experiences Grade another student's program Blind test -- develop specification by testing a working program. Treasure Hunt – find/fix seeded errors Write test cases before coding Develop, certify and reuse components 3/19/2002 Testing at FAMU

The Software TestLab Environment for discovery & learning Basic testing skills Mentoring / Competency based training Evolving Laboratory Tools & tutorials Staffed by students and faculty Problems/solutions test bed Dissemination Strategy 3/19/2002 Testing at FAMU

Software TestLab - Context CS course Technology Transfer Testing Enthusiast Donated/public test artifact Artifact or tutorial Graduate Testing Research TestLab Artifacts artifact Test Arcade artifact artifact Certification / Tutorial Arcade Problems (+Meta-Data) TestLab Student Training 3/19/2002 Testing at FAMU

Student Mentorship Model Managed skill development Clear achievement goals Key Practices x Competency Levels Progress certification Student-Student mentoring Recognition Program 3/19/2002 Testing at FAMU

Key Practices Practitioner -- performs defined test Builder -- constructs test “machinery” Designer -- designs test cases Analyst -- sets test goals, strategy Inspector -- verifies process/results Environmentalist -- maintains test tools & environment Specialist -- performs test life cycle. 3/19/2002 Testing at FAMU

Specialist I - Competencies 1 2 3 4 5 ... Test Practitioner Practitioner 1 2 3 4 5 ... Test Builder 1 2 3 4 5 ... Test Designer 1 2 3 4 5 ... Test Analyst 1 2 3 4 5 ... Test Inspector 1 2 3 4 5 ... Test Environmentalist 1 2 3 4 5 ... Test SPECIALIST 3/19/2002 Testing at FAMU

Test Specialist I Practitioner I - Run function test script & document test results. Builder I - Develop function test driver. Designer II - Develop functional and boundary test cases from decision table. Analyst I - Develop decision table from functional specification. Inspector I - Verify adherence to function test process and standards. 3/19/2002 Testing at FAMU

Training Topics TestLab Environment Basic Training Black-Box Function (unit) Testing Black-Box Object Testing White-Box Function Testing Clear-Box Object Testing . . . 3/19/2002 Testing at FAMU

Problems/Solutions Testbed Repository of testing problems Repository of student test artifacts Best in class  solutions testbed Deficient solutions  “almost” testbed Testbed used for tester certification 3/19/2002 Testing at FAMU

The Test Arcade Fun & Games approach Players compete solving testing problem Scored on time and accuracy Ranked list of players by score HELP facility provides a "teaching" mode Supports testing contests, certification NSF Funding requested 3/19/2002 Testing at FAMU

Part III RESEARCH ACTIVITIES 3/19/2002 Testing at FAMU

Research Obstacles Search for the silver bullet All eggs in one basket, i.e., testing only In-the-large empirical studies Access to project data Experimental design Graduate/research student pool Critical mass of faculty 3/19/2002 Testing at FAMU

Research Funding $85K Corporate support Task on $1.5M NSF MII grant $50K from Dell Star Program (TestLab) $35K for student / travel Task on $1.5M NSF MII grant $500K NSF ITR proposal submitted (2x) 3/19/2002 Testing at FAMU

Seminal Projects Decision-based test methods Reliability testing training simulator Test arcade Testing via design mutation Test patterns/verification agents Other Ideas 3/19/2002 Testing at FAMU

Decision Based Testing (DBT) Decision Table Logic model, lightweight formality Simple to teach Systematic test condition generation Column => behavioral equivalence class Conditions => basis for boundary analysis Complication DT variables computed from inputs 3/19/2002 Testing at FAMU

Simple Example DBT & SPRAE Compute pay with overtime calculation based on employee classification Example illustrates SPRAE principles Test case design Opportunities for automation 3/19/2002 Testing at FAMU

Principle: Specification Specification: Compute pay for an employee, given Hours worked and hourly pay Rate; overtime is 1.5 times hourly Rate, for Hours above 40. Salaried employees earn over $20/hour, and are always paid for 40 hours. Hours Rate Pay Compute 3/19/2002 Testing at FAMU

Principle: Premeditation Use a systematic process to devise test cases based on the specification. Functional testing: Decision rule (column) => behaviors One test case per behavior Determine expected result. 3/19/2002 Testing at FAMU

Decision Analysis Behavior 3/19/2002 Testing at FAMU

Functional Test Cases 1 2 3/19/2002 Testing at FAMU

Boundary Test Cases 3/19/2002 Testing at FAMU

Principle: Repeatability Processes for test case creation and test execution must yield equivalent results, independently of the tester. Systematic processes for deriving Functional test cases Boundary test cases Processes can be automated 3/19/2002 Testing at FAMU

Repeatability -- Test Scripts Functional Test Cases Boundary Test Cases 3/19/2002 Testing at FAMU

Principle: Accountability Keep records that document test process and artifacts. Documentation answers: What tests were planned? Which tests were conducted? Who did what testing,and when? What were the results? How were the results interpreted? 3/19/2002 Testing at FAMU

Accountability -- Test Log Test Results Summary Failure Description 3/19/2002 Testing at FAMU

Principle: Economy Economy: Test activities must not require excessive time or effort. Systematic processes Risk-based Automation Test drivers/harnesses (classical) Test case generation/assistance Test results analysis (oracle, database). 3/19/2002 Testing at FAMU

DBT -- Related Work State-based testing Published methods State models a form of decision table State explosion problem -- infeasibility of exhaustive testing Published methods Parnas’ Software Cost Reduction (SCR) methodology Object-oriented testing 3/19/2002 Testing at FAMU

Commercial/Research Decision Table Tool Prologa (PROcedural Logic Analyzer) Interactive, rule-based Table construction and manipulation Facilitates tasks such as checking, adding or reordering conditions. Reference Robbin, F., Vanthienen, J., “Developing Legal Knowledge Based Systems Using Decision Tables,” Communications of the ACM, 1993. 3/19/2002 Testing at FAMU

TestLab Decision-Based Test Tools DT-Editor Testdata DT-2-logicspec DT Test Data Specifications TD-verify DT-Reverse BTD-Generate Testset Adequacy Source Code Boundary Testdata 3/19/2002 Testing at FAMU

Custom Decision Table Tools Support software specification and testing TD-verify determines functional test set adequacy based on DT rules Post facto verification of whitebox coverage (when DT-Reverse used) Execution-free adequacy test Formalism to support test arcade 3/19/2002 Testing at FAMU

Seminal Projects Decision-based test methods Reliability testing training simulator Test arcade Testing via design mutation Test patterns / verification agents Other Ideas 3/19/2002 Testing at FAMU

Test Arcade Problem: Testbed of problem templates + problem/answer generator + presentation + response manager + score manager. Approach: Identify problem templates and develop knowledge base for representation and generation. Target problems to specific skill levels. Research: Refine competency levels x test instances. Accumulate problem set per (level, instance) pairs. Tool: database + web access. 3/19/2002 Testing at FAMU

Seminal Projects Decision-based test methods Reliability testing training simulator Test arcade Testing via design mutation Test patterns / verification agents Other Ideas 3/19/2002 Testing at FAMU

Reliability Test Simulator Prototype from Graduate IV&V Course Published in ISSRE 2001 Proceedings Excerpts from Presentation 3/19/2002 Testing at FAMU

Reliability Simulation -- TBDs Simulation for experimentation Imperfect defect removal Operational testing Thesis topic(s) Model validation vs. published/industry results Management decision-support via simulation 3/19/2002 Testing at FAMU

Seminal Projects Decision-based test methods Reliability testing training simulator Test arcade Testing via design mutation Test patterns / verification agents Other Ideas 3/19/2002 Testing at FAMU

Design Mutation Reflects design misinterpretations Mutant killed when Failk is nonempty D1 Gen Test Sets Tset1 Fail1 Run Tests Design Fail2 D2 Tset2 Mutate D3 Tset3 Fail3 ... ... ... Tsetn Failn Dn 3/19/2002 Testing at FAMU

Mutation of Decision Table Mutant Behavior Mutant Behavior 3/19/2002 Testing at FAMU

Impact of Mutations on Functional Test Cases Bad test case Bad test case 3/19/2002 Testing at FAMU

Design Mutation -- Prospectus Application Requirements/design models Guided inspection -- ensure mutants not being created Testing: creation of BAD test cases Practical if automated Facilitated by formal representation 3/19/2002 Testing at FAMU

Seminal Projects Decision-based test methods Reliability testing training simulator Test arcade Testing via design mutation Test patterns / verification agents Other Ideas 3/19/2002 Testing at FAMU

Test Patterns (Example) GUI input controls Properties determine required testing Derive black-box test patterns for GUI input controls Propose automated test agents knowledgeable about specific test pattern 3/19/2002 Testing at FAMU

Test Patterns (Continued) Test patterns for design patterns Empirical test patterns reflecting organizational practices Reverse engineer practices into patterns Forward engineer patterns into practice 3/19/2002 Testing at FAMU

Test Agents Properties of Agents Scope of Agents Responsibility Knowledge Autonomy Scope of Agents Functional Unit (F) Architectural Component (A) System (S) 3/19/2002 Testing at FAMU

Test Agents -- Prospectus Intelligence Generator of checklist / procedure for human tester Watch-dog and status reporter Reminder to complete test action Performer of test action Coordinator of other test agents 3/19/2002 Testing at FAMU

Seminal Projects Decision-based test methods Reliability testing training simulator Test arcade Testing via design mutation Test patterns / verification agents Other Ideas 3/19/2002 Testing at FAMU

Testing: A State of Practice Study Problem: Develop a method for characterizing the state of practice in software testing. Approach: Use frameworks like the Test Maturity Model and SPRAE to characterize/assess state of practice. Research: Relate attributes of testing practice to qualitative and quantitative indicators of effectiveness and satisfaction. Devise easy to use evaluation system that identifies areas needing improvement, and which provide insightful measures of performance. 3/19/2002 Testing at FAMU

Adaptive Operational Testing Problem: Perfecting software by OT is biased by expected feature usage. Even when errors are flushed out for one feature, the test emphasis remains the same. OT leads to slow rate of error discovery in other features. Approach: Given features A,B,C with probabilities pA, pB, pC, and MTBF of tA, tB, tC. Shift feature probability as feature reliability increases. Research: Determine criteria for shifting p’s so that feature starvation does not occur. Tool: Reliability simulator to prototype solutions. 3/19/2002 Testing at FAMU

Testing and Reverse Engineering Testing answers the questions “What do we have here?” (exploratory) “Is this what we wanted?” (acceptance) “Is this what we expected?” (scripted) Testing is the last chance to document the as-built system Exploratory testing -- can it be sooner? 3/19/2002 Testing at FAMU

Testability via Self-Diagnostics in Objects Problem: Lack of observability [Voas] in O-O software complicates testability. Encapsulation prevents error propagation to outside. Approach: Design in self diagnostics along with means of propagation to outside. Research: Research observability problem in object-based testing. Apply and extend frameworks/methods like JUnit to implement self-diagnostics. 3/19/2002 Testing at FAMU

Part IV CONCLUSION 3/19/2002 Testing at FAMU

Support Acknowledgment Lucent Technologies Dell Computer 3M Telcordia Students NSF (EIA-9906590) Lockheed Martin Abbott Laboratories Eli Lilly and Company Software Research, Inc. 3/19/2002 Testing at FAMU

Future Evolve TestLab Mentorship Model Transfer to Selected Courses TestLab students = transfer agents Disseminate Results Web site, newsletter, conferences Procure Funding for Research Find Research Collaborators 3/19/2002 Testing at FAMU

Opportunities To P A R T N E 3/19/2002 Testing at FAMU

Questions? Questions? Questions? 3/19/2002 Testing at FAMU