1. Quality is about testing early and testing often Joe Apuzzo, Ngozi Nwana, Sweety Varghese Student/Faculty Research Day CSIS Pace University May 6th, 2005

2. Overview Introduction Introduction Overview Overview Relevance Relevance Our methodology: Quality Attributes Our methodology: Quality Attributes Main Message to project teams Main Message to project teams QA Walkthrough QA Walkthrough Iterative development and testing Iterative development and testing Unit Test case creation Unit Test case creation Conclusions Conclusions Estimate of QA successfulness Estimate of QA successfulness Lessons Learned Lessons Learned Recommendations Recommendations Summary Summary Question session Question session

3. Introduction The CS615/616 is the capstone course for Pace University’s Masters in Computer Science curriculum. The focus of which is to develop soft skills as well as introducing the student to formal software engineering procedures. The CS615/616 is the capstone course for Pace University’s Masters in Computer Science curriculum. The focus of which is to develop soft skills as well as introducing the student to formal software engineering procedures. To develop these real world skills, the students are broken up into teams, assigned a project and thus, responsible for delivering it on time. To develop these real world skills, the students are broken up into teams, assigned a project and thus, responsible for delivering it on time. The customers must accept the provided solution for the corresponding team to achieve full marks. The customers must accept the provided solution for the corresponding team to achieve full marks. Our team was assigned the duty of being the Quality Assurance team. Our goal was to assist in delivery of high quality projects, through reviews, testing and the establishment of best practices. Our team was assigned the duty of being the Quality Assurance team. Our goal was to assist in delivery of high quality projects, through reviews, testing and the establishment of best practices.

4. Relevance “It is widely accepted that a "test-early, test-often" approach can greatly assist in the production of quality software … the design of software can impact on the ease of testing so testing must be in borne in mind from project conception through to completion.” “It is widely accepted that a "test-early, test-often" approach can greatly assist in the production of quality software … the design of software can impact on the ease of testing so testing must be in borne in mind from project conception through to completion.” The Serco consulting group

5. Team approach Reviewed 2003 2004 QA team’s findings Reviewed 2003 2004 QA team’s findings Researched Quality Assurance best practices Researched Quality Assurance best practices Evaluated development environment Evaluated development environment Focus on customer satisfaction as a measure of software quality Focus on customer satisfaction as a measure of software quality

6. Our methodology: Quality Attributes Clarity of user requirements Clarity of user requirements Who are the stakeholders and who are the users? Match up each requirement with each one. Who are the stakeholders and who are the users? Match up each requirement with each one. What are the most important, least important, which have the greatest risks? What are the most important, least important, which have the greatest risks? What are the user stories? Write them down! What are the user stories? Write them down! Are there Conflicting, Confusing or Incomplete user requirements? Are there Conflicting, Confusing or Incomplete user requirements?

7. Our methodology: Quality Attributes Verifiability Verifiability Define how the program knows when to continue and when to generate an error. Define how the program knows when to continue and when to generate an error. Each component needs to verify that it's input, output and data is correct. When components fail to do this bugs will be harder to locate, since bad data may be passed from module to module before being detected. Each component needs to verify that it's input, output and data is correct. When components fail to do this bugs will be harder to locate, since bad data may be passed from module to module before being detected. Use an integrated system log that is updated with state at each input or decision point. Use an integrated system log that is updated with state at each input or decision point.

8. Our methodology: Quality Attributes Modifiability ( modularity or reusability) Modifiability ( modularity or reusability) Document how other developers who are not familiar with the project should proceed in making code change. Explain how to locate components, where to find there interdependences, where special cases are explained, etc. Document how other developers who are not familiar with the project should proceed in making code change. Explain how to locate components, where to find there interdependences, where special cases are explained, etc.

9. Our methodology: Quality Attributes Usability Usability Set the expected level of usability and document how you will communicate this to the stakeholders. Set the expected level of usability and document how you will communicate this to the stakeholders. define who and how the system will be used. define who and how the system will be used. What prior knowledge show a user have and if they should have specific skills that an average person would not. What prior knowledge show a user have and if they should have specific skills that an average person would not. All assumptions should be listed like "users will be accustomed to MS Windows and using a web browser" All assumptions should be listed like "users will be accustomed to MS Windows and using a web browser" For each customer delivery review usability and have sufficient buy in by all stakeholders. For each customer delivery review usability and have sufficient buy in by all stakeholders.

10. Our methodology: Quality Attributes Security Security Define what the security requirements are for each user and or stakeholder. Define what the security requirements are for each user and or stakeholder. Define what a security issue would be, and how it would be mitigated. Define what a security issue would be, and how it would be mitigated. How will this software protect the users data and from whom? How will this software protect the users data and from whom? Do the stakeholders agree with the security statements? Do the stakeholders agree with the security statements?

11. Main Message to project teams Test early Test early Test often Test often Test enough Test enough Designate (know your role) Designate (know your role) Communicate Communicate Iterate Iterate

12. System Test Iterative development and testing Code Unit Test Re- Design Integration test Integrate findings

13. Unit test case creation There are 4 full paths, 3 decision points, 11 sections. All of which should be included in a unit test.

14. Estimate of QA successfulness Reviewing designs early helped each team to clarified design trade-offs. Reviewing designs early helped each team to clarified design trade-offs. Setting dates for when things should be delivered helped focus each team to keeping on task. Setting dates for when things should be delivered helped focus each team to keeping on task. Problems were found during testing and corrected. Problems were found during testing and corrected.

15. Lessons learned Applying Quality Assurance from the outside of each team was not as effective as we thought it would be. Applying Quality Assurance from the outside of each team was not as effective as we thought it would be. Soft skills were as important as technical skills. Soft skills were as important as technical skills. Technological barriers still have the biggest impact to a project. Technological barriers still have the biggest impact to a project. There is not enough time to do project work. There is not enough time to do project work.

16. Recommendations Decentralize Quality Assurance, make one member of each team the QA focal point. Decentralize Quality Assurance, make one member of each team the QA focal point. Mile stone and commitments need to be mapped out as early as possible. Mile stone and commitments need to be mapped out as early as possible. Start as early as possible with the QA part of new projects, reusing or reworking the existing documents from previous Teams. Start as early as possible with the QA part of new projects, reusing or reworking the existing documents from previous Teams.

17. Summary There is still work to be done. There is still work to be done. There is opportunity to streamline many QA processes. There is opportunity to streamline many QA processes. Communication between the teams and also between team members can be improved. Communication between the teams and also between team members can be improved. As in other studies, the more frequent and thorough unit testing combined with a frequent and aggressive unit / system testing will result in a higher quality product that will have the greatest chance of delivering on time. As in other studies, the more frequent and thorough unit testing combined with a frequent and aggressive unit / system testing will result in a higher quality product that will have the greatest chance of delivering on time.

18. Question Session Question? Question? Comments? Comments? Suggestions? Suggestions? Thank you for your time