Software System Testing IMPORTANT NOTICE TO STUDENTS: These slides are NOT to be used as a replacement for student notes. These slides are sometimes vague and incomplete on purpose to spark class discussions. Software System Testing CS 360 Lecture 19

System Testing System testing is testing conducted on an integrated system to evaluate its compliance with specified requirements. Takes, as its input, all of the integrated components that have passed integration testing. The purpose of integration testing is to detect any inconsistencies between the units that are integrated together. Detect defects within the system as a whole. “Black-box” testing approach.

System Testing Source of Software Faults During Development

Types of System tests Basic Tests Stress Tests Interoperability Tests Performance Tests Scalability Tests Stability Tests Smoke Tests Regression Tests Documentation Tests

Basic system tests Boot Upgrade/Downgrade Diagnostic Boot tests are designed to verify that the system can boot up its software image (or, build) from the supported boot options. Upgrade/Downgrade Upgrade/downgrade tests are designed to verify that the system software can be upgraded or downgraded (rollback) in a graceful manner. Diagnostic Diagnostic tests are designed to verify that the hardware components (or, modules) of the system are functioning as desired Power-On Self Test

Stress tests Determines the robustness of software by testing it beyond the limits of normal operation. The system is monitored during the overload to ensure that the system can sustain and recover from the stress. Reasons for conducting stress tests: Allows the test team to monitor performance during failures. To verify if the system has saved all data before crashing, or not. To verify if the system prints meaningful error messages while crashing, of if it prints seemly random exceptions. To verify if unexpected failures do not cause security issues.

Stress tests Scenarios: Monitor the system behavior when the maximum number of users log in at the same time. All users performing critical operations at the same time. All users accessing the same file at the same time. Critical software modules unresponsive such as a database service failure.

Interoperability tests Tests how a component or complete application interacts with another application. During interoperability testing, checks are made on how data from one application is transferred into another application and further processed to give the accepted output.

Performance tests Tests are designed to determine the performance of the actual system compared to the expected one Tests are designed to verify response time, execution time, throughput, resource utilization and traffic rate For example: End-to-end response time (as seen by external user) Database access time Network connection time Network throughput

Scalability tests Testing of a software application to measure its capacity to scale up/down or scale out/in. Scaling up/down (Vertical Scaling): Adding/removing resources to a single compute node (CPU cores, RAM, etc.). Tests how the software performs based on fluctuation of resources on a single node. Scaling in/out (Horizontal Scaling): Adding/removing complete compute nodes to a system. Tests how the software performs when components reside or are replicated on different compute nodes.

stability tests Testing the ability of the software application to continue to function over time and over its full range of use, without failing or causing failure. Stability testing advantages: Provide confidence in the software Monitor the effectiveness of your system (Availability, down time, etc.)

stability tests Possible errors related to software stability: System latencies increase over time. The system begins to encounter functionality problems. The system begins to show strange behavior. The system may crash altogether. Example test cases for stability testing: To verify the upper limit of the system. How the system handles crash scenarios and recovers. Transaction response time over a period of time. How the system behaves under heavy/light load.

Smoke tests Also call confidence testing, sanity testing, build verification testing Term “smoke test” comes from a similar type of hardware testing. Where the device passes if it doesn’t catch on fire when turned on. Tries to reveal simple failures severe enough to reject a software release. Ensures that the most important functionalities of a component or system appear to work correctly. Smoke testing is performed at different levels Integration testing System testing Acceptance testing

Regression tests Type of software testing that seeks to uncover new bugs in existing areas of a system after changes have been made to them. Ensures that a change has not introduced new faults. Also verifies intended results are still being produced.

Regression tests Required when Change in requirements and code is modified according to the requirement. A new feature is added to the software. An error is discovered and fixed. Performance/security issued are fixed.

Documentation tests Tests that ensures that the documentation of a software system matches with what actually exists in the software system. Documentation tests include Verifying requirements, diagrams, design patterns, and other approaches for software development of the application. Verifying software application test cases and results. Verifying software application source code version(s). Reviews to update inconsistent and vague sections. Executing spelling, grammar, and format checks.

Testing Web applications Questions to ask when testing web apps: What are the expected loads on the server (e.g., hits/time)? What kind of performance is required under such loads (server response time, database query times)? What kinds of tools will be needed for performance testing (such as web load testing tools, other tools already in house that can be adapted, web robot downloading tools, etc.)? Who is the target audience? What kind of browsers will they be using? What kind of connection speeds will they by using? What kind of performance is expected on the client side (e.g., how fast should pages appear, how fast should animations, applets, etc. load and run)? Will down time for server and content maintenance/upgrades be allowed? how much? What kinds of security (firewalls, encryptions, passwords, etc.) will be required and what is it expected to do? How can it be tested?

Testing Web applications Questions to ask when testing web apps: How reliable are the site's Internet connections required to be? And how does that affect backup system or redundant connection requirements and testing? What processes will be required to manage updates to the web site's content, and what are the requirements for maintaining, tracking, and controlling page content, graphics, links, etc.? Which HTML specification will be adhered to? How strictly? What browsers will be allowed? Will there be any standards or requirements for page appearance and/or graphics throughout a site or parts of a site? How will internal and external links be validated and updated? how often? Can testing be done on the production system, or will a separate test system be required? How are browser caching, variations in browser option settings, dial-up connection variabilities, and real-world internet 'traffic congestion' problems to be accounted for in testing? Do web site visits generate logs/traces of individual user interactions? How extensive or customized are the server logging and reporting requirements; are they considered an integral part of the system and do they require testing?

System Testing Questions?