1. SAIC-F QA Internal Process (DRAFT ) Sudha Chudamani QA Team, Frederick National Lab Jan 2, 2013

2. Frederick National Laboratory for Cancer Research QA Release Process Matrix Release to QAExpected from Development teamExpected from QA teamTime to Test (in Week s) Release Notes List of Bugs fixed List of System Improvements List of new features MST / BSTRTSTPT508 TAG/Build/ Iteration xxxx1-2 W Milestone Release (Limited)xxx1-2W Release Candidate (RC) x (Cumulative) 2-4 W Post Release Patch release or Hot Fix xx TBD (determine d by NCI PO) TBD (determi ned by NCI PO) xTBD LEGENDS: MST / BST – Milestone Specific Test / Build specific Test RT – Regression Test ST – Security Test (Appscan) PT – Performance Test 508 – Section 508 Compliance

3. Frederick National Laboratory for Cancer Research QA Project Initiation Phase –QA project needs are evaluated for skills and resources –Contract awarded if sub-contractor is chosen –Else SAIC-F internal QA resource is allocated –QA Resource / POC, reviews the Project schedule, Project requirements etc., and prepares: Deliverables: –Prepares QA schedule –Creates / Updates Test Plan »Types of testing (Installation, Performance, Interface testing, Section 508 etc.,) – Sets up, Clean up & Maintain JIRA, HPQC etc.

4. Frederick National Laboratory for Cancer Research QA Project Testing (TAG/Build/Iteration) Phase –Tag/Build/Iteration Testing Phase QA POC reviews requirements and writes Test Cases –Test cases are written for functionality planned to be delivered for the TAG prior to receiving TAG, which includes test cases that cover: »New Functionality »Feature / Functionality Enhancements »Defect Fixes »Non-functional test cases, such as performance tests, security tests, or conformance Before the first tag is delivered, QA creates the test environment Once TAG is delivered to QA –QA verifies the technology stack conformance (for the first tag) –QA reviews the release notes and deployment guides –QA installs the application on the QA tier (cont.)

5. Frederick National Laboratory for Cancer Research QA Project Testing (TAG/Build/Iteration) Phase (cont.) Once TAG is delivered to QA (cont.) –Executes Smoke tests to accept the TAG –Executes the Test cases planned for the TAG –Reports Defects –Test Cases are written for next TAG Note: Prefer to run 508 compliance and performance tests on the MR as well The above steps are repeated for each TAG.

6. Frederick National Laboratory for Cancer Research QA Project Milestone Testing Phase –Milestone Testing Phase QA POC writes / updates Test Cases –Test cases are written for testing entire feature / feature enhancements delivered for the Milestone prior to receiving TAG, which includes. »Test cases that cover new features and enhancements »Defect Fixes »Limited Regression Test cases around the feature / defect are identified and updated if needed Once Milestone TAG is delivered to QA –QA reviews the release notes and deployment guides –QA installs the application on the QA tier –Executes Smoke tests to accept the Milestone TAG –Executes the planned Test cases –Reports Defects –Plan and writes Test Cases for the next TAG.

7. Frederick National Laboratory for Cancer Research QA Project Release Candidate (RC) Testing Phase –When all the planned features for the release are developed, a RC tag is cut by the development team. RC tag should not contain any new features / functionalities No show stopper bug Can contain defect fixes that are found in previous testing phases –QA will deploy (or request Systems team) the RC on the QA tier –Execute the tests planned for the Release Execute all the Test cases developed for the release –Test cases written during TAG/Build/Iteration testing phase –Test cases written during Milestone testing phase Execute the full Regression Tests Execute the Performance tests Execute the Section 508 compliance tests Retest all the bug fixes - If new defects are found, they will get fixed and a new RC tag will be cut

8. Frederick National Laboratory for Cancer Research QA Project Release Candidate (RC) Testing Phase - Continued –On successful completion of RC testing, AppScan is requested to be run on QA tier (by the Systems Team) –QA POC documents the Test Report and submits the report to SAIC-F QA Manager for final review and recommend to (NCI QA Lead) promote the RC to Stage tier. The Package submitted should include the following artifacts at the least: Test Plan Test Report in EPLC format Test cases document Test Execution Results Outstanding Defects list and their severity (The outstanding defects and their severity need to be reviewed and approved by PO) Performance Test Report AppScan report 508 Compliance report

9. Frederick National Laboratory for Cancer Research QA STAGE tier Testing Phase –Once RC is approved by the NCI QA Lead, SAIC-F QA Manager will send the sign-off to the Project Manager to be deployed to STAGE tier QA Executes Smoke Tests AppScan is scheduled –If AppScan reports High / Medium vulnerabilities, Dev team fixes the vulnerabilities and a RC is cut and deployed to QA environment where complete Regression Test is performed. Exceptions apply. –Stage  Production Test Report is submitted by QA POC. This again goes through the Approval process before it is signed-off. Test Report in EPLC format Smoke Test cases document Test Execution Results Outstanding Defects list and their severity level AppScan report

10. Frederick National Laboratory for Cancer Research QA Process –Application Release is Deployed to PRODUCTION tier.

11. Frederick National Laboratory for Cancer Research Proposal to improve testing efficiency –Challenges/Risks Every project uses Release Terminologies differently –Proposed Resolution »Create and publish (If one does not exist) a Release Process document which should include a Glossary of Terminologies and share with PMs, Developers and QA. »Should be shared with New Employees / Contractors as part of orientation. Frequent releases to QA –Proposed Resolution »QA should not accept any unplanned TAG without approval from NCI/SAIC-F QA Management »One exception - if the previous TAG failed installation or Smoke Test, this again needs approval from QA Management. (cont.)

12. Frederick National Laboratory for Cancer Research Proposal to improve testing efficiency (cont.) –Challenges/Risks (cont.) Test Maintenance effort not planned for in the project schedule –Proposed Resolution »QA should ensure that this activity is included in the Project Schedule »This activity involves Updating Regression Tests, Organizing HPQC and JIRA, Updating Master Test Plan with planned activity for the upcoming release etc. Too Many Release Candidates (RC Tag) –Proposed Resolution »Reduce Regression errors in RC by: » Implementing Regression Automation » Execute full Automated Regression Tests early on in the testing phase. »If no automation implemented, Instead of Limited manual Regression Testing in Milestone phase, run full regression tests. »For the first RC TAG, enough time should be provided to execute a complete regression tests and all release test cases. (cont.)

13. Frederick National Laboratory for Cancer Research Proposal to improve testing efficiency (cont.) –Challenges/Risks (cont.) Appscan process –Time consuming »QA runs initial Appscan in local environment »Share the Test Script with Systems –Inconsistent results »Have a way to update the Appscan Test Scripts for new features or pages rather than re-writing a complete new script. »Document the Configuration settings that can be shared across projects Number of hardware and software platforms to be considered

14. Frederick National Laboratory for Cancer Research End