Presented To: September 9, 2015 CONFIDENTIAL Copyright 2007, TopCoder, Inc. Sean Campion [scamp] 27 June 2007 TopCoder Open, Las Vegas TopCoder Component-Based Software Engineering (CBSE) Quality Assurance (QA)

CBSE Quality Assurance Specific Challenges TopCoder Process from QA perspective Wrap-up [ ] Agenda CONFIDENTIAL Copyright 2007, TopCoder, Inc.

[ ] CBSE Quality Assurance Specific Challenges CONFIDENTIAL Copyright 2007, TopCoder, Inc. The following are common QA problems cited against the use of reusable components. I will discuss each issue, and what TopCoder does to address the issue.

[ ] CBSE Quality Assurance Specific Challenges CONFIDENTIAL Copyright 2007, TopCoder, Inc. Issue: Lack of access to commercial component source code. TopCoder: We provide both the source code and all design documentation for every component that we produce.

[ ] CBSE Quality Assurance Specific Challenges CONFIDENTIAL Copyright 2007, TopCoder, Inc. Issue: Components are not readily testable. TopCoder: All of our components are required to have unit tests of stress, failure, and accuracy.

[ ] CBSE Quality Assurance Specific Challenges CONFIDENTIAL Copyright 2007, TopCoder, Inc. Issue: Difficult to use and interpret tests from component vendors TopCoder: Our documentation provides examples on how to run the tests; we also provide the test results in the form of logs.

[ ] CBSE Quality Assurance Specific Challenges CONFIDENTIAL Copyright 2007, TopCoder, Inc. Issue: Concern that components were not developed for a specific operating environment; and that developing a component test bed for all possible environments is too costly and time consuming. TopCoder: We are developing a software system that will take advantage of participants diverse environments to test components.

[ ] CONFIDENTIAL Copyright 2007, TopCoder, Inc. -Distributes component tests to participating computers. -Takes advantage of diverse environments, hardware, character sets, and so on. -Massively parallel -Can be setup at customer sites to start validating existing components during architecture phase on actual target environment. -Customers can use to test their own components. -Ideal for regression-testing when component is updated.

[ ] CONFIDENTIAL Copyright 2007, TopCoder, Inc.

[ ] TopCoder Process from a QA Perspective CONFIDENTIAL Copyright 2007, TopCoder, Inc. Specification Phase -Mid and end-point peer reviews of the application specification, conducted by 2 Project Managers and 1 Architect. -High-level test cases, tied to application level functionality, developed. -Customer review and sign-off on application specification.

[ ] TopCoder Process from a QA Perspective CONFIDENTIAL Copyright 2007, TopCoder, Inc. Architecture Phase -Mid and end-point peer reviews of the architecture design, conducted by 1 Project Manager and 2 Architects. -High-level test cases expanded into detailed test cases and scenarios. -Customer review and sign-off on architecture design. -Future: Existing components identified for reuse can be validated using including on the customer target systems if desired.

[ ] TopCoder Process from a QA Perspective CONFIDENTIAL Copyright 2007, TopCoder, Inc. Component Production Phase - Design -Design competitions; designers know they are competing against each other. -Automated submission screening tool. -Manual screening phase. -Review Board. -Scorecard-based evaluation. -Final fixes. -Winning designer required to support the development phase.

[ ] TopCoder Process from a QA Perspective CONFIDENTIAL Copyright 2007, TopCoder, Inc. Component Production Phase - Development -Development competitions; developers know they are competing against each other. -Automated submission screening tool. -Manual screening phase. -Review Board. -Scorecard-based evaluation. -Final fixes. -Unit tests: stress, failure, accuracy -Coverage Tool: minimum of 85% line coverage from unit tests.

[ ] TopCoder Process from a QA Perspective CONFIDENTIAL Copyright 2007, TopCoder, Inc. Assembly Phase -Assembly competitions; teams know they are competing against each other. -Manual screening phase. -Review Board. -Scorecard-based evaluation. -Final fixes.

[ ] TopCoder Process from a QA Perspective CONFIDENTIAL Copyright 2007, TopCoder, Inc. Assembly Phase - Testing -Testing competitions; teams know they are competing against each other. -Produces automated tests that will be run against the assembly competition submissions.

[ ] TopCoder Process from a QA Perspective CONFIDENTIAL Copyright 2007, TopCoder, Inc. Certification Phase -Replicate the customer’s target environment on a TC server. -Deploy and rigorously test the winning assembly submission. -Run automated tests against the submission. -Run manual tests against the submission. -Early customer access to the TC testing environment for immediate feedback.

[ ] TopCoder Process from a QA Perspective CONFIDENTIAL Copyright 2007, TopCoder, Inc. User Acceptance Testing Phase -Customer has option of performing their own User Acceptance Testing. -TopCoder will directly support any UAT.

[ ] TopCoder Process from a QA Perspective CONFIDENTIAL Copyright 2007, TopCoder, Inc. Deployment Phase -TopCoder will support the deployed application for 90 days. -Customer receives all source code, including component source code, and all design documentation from all phases.

[ ] Metrics CONFIDENTIAL Copyright 2007, TopCoder, Inc. Here’s how TopCoder compares to industry standards: Industry Defects per KLOC: 3.43 TopCoder Defects per KLOC: 2.10

[ ] Wrap-Up CONFIDENTIAL Copyright 2007, TopCoder, Inc. -Quality Assurance addressed at every stage of the process in one form or another. -TopCoder continuously looking to improve QA through processes and tools. addresses a primary concern of reusable components. -Quality software distinguishes TopCoder from other software.

[ ] References CONFIDENTIAL Copyright 2007, TopCoder, Inc. -“Testing and Quality Assurance for Component-Based Software”, Jerry Zeyu Gao, H.-S. Jacob Tsao, & Ye Wu; Artech House, Boston, “Risk Management in Component-Based Development: A Separation of Concerns Perspective”, Awais Rashid, Gerald Kotonya, Computing Department, Lancaster University, UK. -“The Trouble With Testing Components”, Elaine J. Weyuker, in “Component Based Software Engineering”, Chapter 28, Addison-Wesley, “CBS Testing Requirements and Test Case Process Documentation Revisited”, Nasib S. Gill, Pradeep Tomar, Department of Computer Science and Applications, Haryana India, March From ACM SIGSOFT Software Engineering Notes, Volume 32 Number 2. -“The QSM Software Alamanac: Application Development Series”, QSM Inc., 2006.