1. Florida A&M University
Software Testing at Florida A&M University: Perspective, Direction and Opportunities
Dr. Edward L. Jones
CIS Department
Florida A&M University
Tallahassee, FL

2. Part I
INTRODUCTION
3/19/2002
Testing at FAMU

3. My Institution 12,000 students total 600+ CIS majors 15 Faculty
Heavy teaching
Graduate program (20 students)
Increasing research
3/19/2002
Testing at FAMU

4. My Experience 13 years Harris Corporation
Software engineering technology
$200M+ NASA project
methodology expert
software inspections, testing
process training
20+ years university teaching
3 years focusing on testing
3/19/2002
Testing at FAMU

5. The Vision Quality Awareness in Curriculum Solid Framework
Enhanced Student Skill Set
Expanded Student Opportunities
Industry Partnerships
Research & Dissemination
3/19/2002
Testing at FAMU

6. Software TestLab - Context
CS course
Technology
Transfer
Testing
Enthusiast
Donated/public test artifact
Artifact or
tutorial
Graduate Testing Research
TestLab Artifacts
artifact
Test
Arcade
artifact
artifact
Certification / Tutorial
Arcade Problems (+Meta-Data)
TestLab Student Training
3/19/2002
Testing at FAMU

7. My Perspective Testing is not just for testers!
In ideal world, fewer testers required
Verification vs testing
More skilled developers
No silver bullet … just bricks
Simple things provide leverage
Testing in-the-small -- classroom the lab
Technology transfer crucial
3/19/2002
Testing at FAMU

8. SPRAE Testing Framework
S pecification – basis for testing
P remeditation – follow a process
R epeatability – tester independence
A ccountability – documentation
E conomy – cost effectiveness
3/19/2002
Testing at FAMU

9. Why SPRAE? A value system for testing practice
Explains the “what” and “why”
Sets goals for test experiences
Each experience reinforces values
A framework for life-long learning.
3/19/2002
Testing at FAMU

10. A Test Life Cycle Analysis Design Implementation Execution Evaluation
Specification (Y/N)
Test Plan/Strategy
Test Script, Data,
Driver
Defect Data
Problem Reports
Test Results,
Log
Test Cases
3/19/2002
Testing at FAMU

11. (What can Academics Do Besides Teaching A Course?)
Part II
EDUCATION MISSION
(What can Academics Do Besides Teaching A Course?)
3/19/2002
Testing at FAMU

12. Caught and Taught Caught Taught Strategy must contain both elements
Attitudes
Respect for consequence of not testing
Taught
Techniques for specific kinds of testing
Basic analytical and programming skills
Strategy must contain both elements
3/19/2002
Testing at FAMU

13. A Holistic Approach SPRAE Testing Framework Software Test Lab
Core Course Experiences
Elective Testing Course
Testing In Action Experiences
SPRAE Testing Framework
3/19/2002
Testing at FAMU

14. What Is Meant By “Holistic”?
Testing an integral part of curriculum
Multiple test experiences
Experiences in each course
Repetition and reinforcement
Accumulation of experiences
Experiences cover test lifecycle
Holistic, NOT Exhaustive!
3/19/2002
Testing at FAMU

15. A Software Testing Course
80% practice, 20% theory
2/3 fine-grained testing (functions)
1/3 object and application testing
Test cases, drivers, and scripts
Decision tables the "formalism"
Function, boundary, white-box testing
Effectiveness: Coverage & error seeding
3/19/2002
Testing at FAMU

16. Course -- Lessons Learned
Advantages
A ”taste" of how hard testing really is
Preparation for advanced study
Rounds out analytical & programming skills
Deficiencies
Course not available to all students
Students compartmentalize knowledge
3/19/2002
Testing at FAMU

17. Technology Transfer -- Program Grading Services
Provided by TestLab students
Experience designing automated tests
Shell programming
Instructor plays active part
Refines specification
Creates grading plan
Benefits
Faculty more inclined to incorporate testing “module” into course
3/19/2002
Testing at FAMU

18. Technology Transfer -- Plagiarism Deterrent
Mutation analysis/error seeding applied to suspicious student program
Student must debug modified program
TestLab project – tunable error seeding
Benefits
Tool for teaching students to debug
Less plagiarism … or … more skilled programmers
3/19/2002
Testing at FAMU

19. Core Course Test Experiences
Grade another student's program
Blind test -- develop specification by testing a working program.
Treasure Hunt – find/fix seeded errors
Write test cases before coding
Develop, certify and reuse components
3/19/2002
Testing at FAMU

20. The Software TestLab Environment for discovery & learning
Basic testing skills
Mentoring / Competency based training
Evolving Laboratory
Tools & tutorials
Staffed by students and faculty
Problems/solutions test bed
Dissemination Strategy
3/19/2002
Testing at FAMU

21. Software TestLab - Context
CS course
Technology
Transfer
Testing
Enthusiast
Donated/public test artifact
Artifact or
tutorial
Graduate Testing Research
TestLab Artifacts
artifact
Test
Arcade
artifact
artifact
Certification / Tutorial
Arcade Problems (+Meta-Data)
TestLab Student Training
3/19/2002
Testing at FAMU

22. Student Mentorship Model
Managed skill development
Clear achievement goals
Key Practices x Competency Levels
Progress certification
Student-Student mentoring
Recognition Program
3/19/2002
Testing at FAMU

23. Key Practices Practitioner -- performs defined test
Builder -- constructs test “machinery”
Designer -- designs test cases
Analyst -- sets test goals, strategy
Inspector -- verifies process/results
Environmentalist -- maintains test tools & environment
Specialist -- performs test life cycle.
3/19/2002
Testing at FAMU

24. Specialist I - Competencies1 2 3 4 5
...
Test Practitioner
Practitioner 1 2 3 4 5
...
Test Builder 1 2 3 4 5
...
Test Designer 1 2 3 4 5
...
Test Analyst 1 2 3 4 5
...
Test Inspector 1 2 3 4 5
...
Test Environmentalist 1 2 3 4 5
...
Test SPECIALIST
3/19/2002
Testing at FAMU

25. Test Specialist I
Practitioner I - Run function test script & document test results.
Builder I - Develop function test driver.
Designer II - Develop functional and boundary test cases from decision table.
Analyst I - Develop decision table from functional specification.
Inspector I - Verify adherence to function test process and standards.
3/19/2002
Testing at FAMU

26. Training Topics TestLab Environment Basic Training
Black-Box Function (unit) Testing
Black-Box Object Testing
White-Box Function Testing
Clear-Box Object Testing
. . .
3/19/2002
Testing at FAMU

27. Problems/Solutions Testbed
Repository of testing problems
Repository of student test artifacts
Best in class  solutions testbed
Deficient solutions  “almost” testbed
Testbed used for tester certification
3/19/2002
Testing at FAMU

28. The Test Arcade Fun & Games approach
Players compete solving testing problem
Scored on time and accuracy
Ranked list of players by score
HELP facility provides a "teaching" mode
Supports testing contests, certification
NSF Funding requested
3/19/2002
Testing at FAMU

29. Part III RESEARCH ACTIVITIES
3/19/2002
Testing at FAMU

30. Research Obstacles Search for the silver bullet
All eggs in one basket, i.e., testing only
In-the-large empirical studies
Access to project data
Experimental design
Graduate/research student pool
Critical mass of faculty
3/19/2002
Testing at FAMU

31. Research Funding $85K Corporate support Task on $1.5M NSF MII grant
$50K from Dell Star Program (TestLab)
$35K for student / travel
Task on $1.5M NSF MII grant
$500K NSF ITR proposal submitted (2x)
3/19/2002
Testing at FAMU

32. Seminal Projects Decision-based test methods
Reliability testing training simulator
Test arcade
Testing via design mutation
Test patterns/verification agents
Other Ideas
3/19/2002
Testing at FAMU

33. Decision Based Testing (DBT)
Decision Table
Logic model, lightweight formality
Simple to teach
Systematic test condition generation
Column => behavioral equivalence class
Conditions => basis for boundary analysis
Complication
DT variables computed from inputs
3/19/2002
Testing at FAMU

34. Simple Example DBT & SPRAE
Compute pay with overtime calculation based on employee classification
Example illustrates
SPRAE principles
Test case design
Opportunities for automation
3/19/2002
Testing at FAMU

35. Principle: Specification
Specification: Compute pay for an employee, given Hours worked and hourly pay Rate; overtime is 1.5 times hourly Rate, for Hours above 40. Salaried employees earn over $20/hour, and are always paid for 40 hours.
Hours
Rate
Pay
Compute
3/19/2002
Testing at FAMU

36. Principle: Premeditation
Use a systematic process to devise test cases based on the specification.
Functional testing:
Decision rule (column) => behaviors
One test case per behavior
Determine expected result.
3/19/2002
Testing at FAMU

37. Decision Analysis
Behavior
3/19/2002
Testing at FAMU

38. Functional Test Cases 1 2
3/19/2002
Testing at FAMU

39. Boundary Test Cases
3/19/2002
Testing at FAMU

40. Principle: Repeatability
Processes for test case creation and test execution must yield equivalent results, independently of the tester.
Systematic processes for deriving
Functional test cases
Boundary test cases
Processes can be automated
3/19/2002
Testing at FAMU

41. Repeatability -- Test Scripts
Functional
Test Cases
Boundary
Test Cases
3/19/2002
Testing at FAMU

42. Principle: Accountability
Keep records that document test process and artifacts.
Documentation answers:
What tests were planned?
Which tests were conducted?
Who did what testing,and when?
What were the results?
How were the results interpreted?
3/19/2002
Testing at FAMU

43. Accountability -- Test Log
Test Results
Summary
Failure
Description
3/19/2002
Testing at FAMU

44. Principle: Economy
Economy: Test activities must not require excessive time or effort.
Systematic processes
Risk-based
Automation
Test drivers/harnesses (classical)
Test case generation/assistance
Test results analysis (oracle, database).
3/19/2002
Testing at FAMU

45. DBT -- Related Work State-based testing Published methods
State models a form of decision table
State explosion problem -- infeasibility of exhaustive testing
Published methods
Parnas’ Software Cost Reduction (SCR) methodology
Object-oriented testing
3/19/2002
Testing at FAMU

46. Commercial/Research Decision Table Tool
Prologa (PROcedural Logic Analyzer)
Interactive, rule-based
Table construction and manipulation
Facilitates tasks such as checking, adding or reordering conditions.
Reference
Robbin, F., Vanthienen, J., “Developing Legal Knowledge Based Systems Using Decision Tables,” Communications of the ACM, 1993.
3/19/2002
Testing at FAMU

47. TestLab Decision-Based Test Tools
DT-Editor
Testdata
DT-2-logicspec DT
Test Data
Specifications
TD-verify
DT-Reverse
BTD-Generate
Testset
Adequacy
Source
Code
Boundary
Testdata
3/19/2002
Testing at FAMU

48. Custom Decision Table Tools
Support software specification and testing
TD-verify determines functional test set adequacy based on DT rules
Post facto verification of whitebox coverage (when DT-Reverse used)
Execution-free adequacy test
Formalism to support test arcade
3/19/2002
Testing at FAMU

49. Seminal Projects Decision-based test methods
Reliability testing training simulator
Test arcade
Testing via design mutation
Test patterns / verification agents
Other Ideas
3/19/2002
Testing at FAMU

50. Test Arcade
Problem: Testbed of problem templates + problem/answer generator + presentation + response manager + score manager.
Approach: Identify problem templates and develop knowledge base for representation and generation. Target problems to specific skill levels.
Research: Refine competency levels x test instances. Accumulate problem set per (level, instance) pairs.
Tool: database + web access.
3/19/2002
Testing at FAMU

51. Seminal Projects Decision-based test methods
Reliability testing training simulator
Test arcade
Testing via design mutation
Test patterns / verification agents
Other Ideas
3/19/2002
Testing at FAMU

52. Reliability Test Simulator
Prototype from Graduate IV&V Course
Published in ISSRE 2001 Proceedings
Excerpts from Presentation
3/19/2002
Testing at FAMU

53. Reliability Simulation -- TBDs
Simulation for experimentation
Imperfect defect removal
Operational testing
Thesis topic(s)
Model validation vs. published/industry results
Management decision-support via simulation
3/19/2002
Testing at FAMU

54. Seminal Projects Decision-based test methods
Reliability testing training simulator
Test arcade
Testing via design mutation
Test patterns / verification agents
Other Ideas
3/19/2002
Testing at FAMU

55. Design Mutation Reflects design misinterpretations
Mutant killed when Failk is nonempty D1
Gen Test Sets
Tset1
Fail1
Run Tests
Design
Fail2 D2
Tset2
Mutate D3
Tset3
Fail3
...
...
...
Tsetn
Failn Dn
3/19/2002
Testing at FAMU

56. Mutation of Decision Table
Mutant
Behavior
Mutant
Behavior
3/19/2002
Testing at FAMU

57. Impact of Mutations on Functional Test Cases
Bad
test case
Bad test case
3/19/2002
Testing at FAMU

58. Design Mutation -- Prospectus
Application
Requirements/design models
Guided inspection -- ensure mutants not being created
Testing: creation of BAD test cases
Practical if automated
Facilitated by formal representation
3/19/2002
Testing at FAMU

59. Seminal Projects Decision-based test methods
Reliability testing training simulator
Test arcade
Testing via design mutation
Test patterns / verification agents
Other Ideas
3/19/2002
Testing at FAMU

60. Test Patterns (Example)
GUI input controls
Properties determine required testing
Derive black-box test patterns for GUI input controls
Propose automated test agents knowledgeable about specific test pattern
3/19/2002
Testing at FAMU

61. Test Patterns (Continued)
Test patterns for design patterns
Empirical test patterns reflecting organizational practices
Reverse engineer practices into patterns
Forward engineer patterns into practice
3/19/2002
Testing at FAMU

62. Test Agents Properties of Agents Scope of Agents Responsibility
Knowledge
Autonomy
Scope of Agents
Functional Unit (F)
Architectural Component (A)
System (S)
3/19/2002
Testing at FAMU

63. Test Agents -- Prospectus
Intelligence
Generator of checklist / procedure for human tester
Watch-dog and status reporter
Reminder to complete test action
Performer of test action
Coordinator of other test agents
3/19/2002
Testing at FAMU

64. Seminal Projects Decision-based test methods
Reliability testing training simulator
Test arcade
Testing via design mutation
Test patterns / verification agents
Other Ideas
3/19/2002
Testing at FAMU

65. Testing: A State of Practice Study
Problem: Develop a method for characterizing the state of practice in software testing.
Approach: Use frameworks like the Test Maturity Model and SPRAE to characterize/assess state of practice.
Research: Relate attributes of testing practice to qualitative and quantitative indicators of effectiveness and satisfaction. Devise easy to use evaluation system that identifies areas needing improvement, and which provide insightful measures of performance.
3/19/2002
Testing at FAMU

66. Adaptive Operational Testing
Problem: Perfecting software by OT is biased by expected feature usage. Even when errors are flushed out for one feature, the test emphasis remains the same. OT leads to slow rate of error discovery in other features.
Approach: Given features A,B,C with probabilities pA, pB, pC, and MTBF of tA, tB, tC. Shift feature probability as feature reliability increases.
Research: Determine criteria for shifting p’s so that feature starvation does not occur.
Tool: Reliability simulator to prototype solutions.
3/19/2002
Testing at FAMU

67. Testing and Reverse Engineering
Testing answers the questions
“What do we have here?” (exploratory)
“Is this what we wanted?” (acceptance)
“Is this what we expected?” (scripted)
Testing is the last chance to document the as-built system
Exploratory testing -- can it be sooner?
3/19/2002
Testing at FAMU

68. Testability via Self-Diagnostics in Objects
Problem: Lack of observability [Voas] in O-O software complicates testability. Encapsulation prevents error propagation to outside.
Approach: Design in self diagnostics along with means of propagation to outside.
Research: Research observability problem in object-based testing. Apply and extend frameworks/methods like JUnit to implement self-diagnostics.
3/19/2002
Testing at FAMU

69. Part IV
CONCLUSION
3/19/2002
Testing at FAMU

70. Support Acknowledgment
Lucent Technologies
Dell Computer 3M
Telcordia
Students
NSF (EIA )
Lockheed Martin
Abbott Laboratories
Eli Lilly and Company
Software Research, Inc.
3/19/2002
Testing at FAMU

71. Future Evolve TestLab Mentorship Model Transfer to Selected Courses
TestLab students = transfer agents
Disseminate Results
Web site, newsletter, conferences
Procure Funding for Research
Find Research Collaborators
3/19/2002
Testing at FAMU

72. Opportunities To P A R T N E
3/19/2002
Testing at FAMU

73. Questions?
Questions?
Questions?
3/19/2002
Testing at FAMU