2. Error and Defect Management in context of IFDK reference product

3. About this course material -This material if for general training for Error and Defect Management -Material is more supportive in class room -Material will be updated during courses -FreeNest Portable Project Platform is used to demonstrate things only in practice. This is not limiting usage for material for other training environments (I hope ) About material

4. Customer/Business Requirements Sub System Requirements Component Requirements Component / Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Error Database Error Database MAINTENANCE Error Database & Management covers whole product life cycle PRODUCT END timelinetimeline

5. Test Plan Life Cycle Test Plan 1.0 Test Plan 1.0 Week 0 Week 26 Week 52 REL 0.1 REL 0.2 REL 0.5 REL 1.0 Test Plan 2.x Test Plan 2.x Routine Test Needs Still Work Effort Needs Still Work Effort

6. Testing Levels and Error Management UNIT TESTING – Developer's area Implementation Test Framework Error Database Error Database

7. System & Acceptace Testing Implem entati on Test Framewo rk Error Database Error Database Te st Pl an Te st Pl an Implem entati on Test Framewo rk Component 2 Test Plan Test Plan IFDK HARDWARE DESIGN IFDK HARDWARE DESIGN IFDK HARDWARE PROTOTYPE VERIFICATED IFDK HARDWARE PROTOTYPE VERIFICATED

8. Integration Testing Implementa tion Test Framework Error Database Error Database Test Plan Test Plan Implementa tion Test Framework Component 2

9. Error/Bug/Defect Report ● Defect/Burg/Error ID ● Reporter ● Time ● Founded where ● Which way? ● Test Case ● Test Setup/Configuration ● Describe scenario? ● Attachements? Picture/Log/etc.. ● Defect/Burg/Error ID ● Reporter ● Time ● Founded where ● Which way? ● Test Case ● Test Setup/Configuration ● Describe scenario? ● Attachements? Picture/Log/etc.. Error Database Error Database Id1 Id2 Id1 Id3 Id1

10. Hands On: Bugzilla Error Database http://www.bugzilla.org/ http://www.bugzilla.org/installat ion-list/ http://www.bugzilla.org/ http://www.bugzilla.org/installat ion-list/ What is Bugzilla? Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively. Most commercial defect-tracking software vendors charge enormous licensing fees. Despite being "free", Bugzilla has many features its expensive counterparts lack. Consequently, Bugzilla has quickly become a favorite of thousands of organizations across the globe. What is Bugzilla? Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively. Most commercial defect-tracking software vendors charge enormous licensing fees. Despite being "free", Bugzilla has many features its expensive counterparts lack. Consequently, Bugzilla has quickly become a favorite of thousands of organizations across the globe.

11. Error Management – Bugzilla Test Management Database Test Management Database Feature Implementation Feature Implementation TestCase Bug Report Error Database Error Database PASS FAIL TestCase Test Plan/Suite SourceCode Version Control SourceCode Version Control BUGZILLA TESTLINK

12. Other Defect Database Solutions JIRA – Commercial Requisite Pro – Commercial Rational Synergy - Commercial Mantis – Open Source

13. Bugzilla

14. Reporting, Metrics and daily usage

15. Feature or Bug? That's a question!

16. Definitions Failure - Fault, Defect, Bug - Incident, Failure, Error Example forum thread: http://www.allinterview.com/showanswers/36257.htmlhttp://www.allinterview.com/showanswers/36257.html ISTQB syllabus Yläotsikko

17. ● TEST CASE ID XXXXX ● Step1 ● Step2 ● Step3. ● Step4. INCIDENT (Huomio) Bug/Defect (Vika) Defect Database IFDK System EXECUTE TEST ! Test Engineer Notes Reports What means error reporting? Yläotsikko

18. Error/Bug/Defect Report ● Defect/Burg/Error ID ● Reporter ● Time ● Founded where ● Which way? ● Test Case ● Test Setup/Configuration ● Describe scenario? ● Attachements? Picture/Log/etc.. ● Defect/Burg/Error ID ● Reporter ● Time ● Founded where ● Which way? ● Test Case ● Test Setup/Configuration ● Describe scenario? ● Attachements? Picture/Log/etc.. Yläotsikko

19. About Error Management Requirement Management Requirement Management Implementation Process Implementation Process Verificat ion& Validatio n Verificat ion& Validatio n Failure Report Failure Report Yläotsikko

21. Example Sources for error report CRM Field Testing Testing Process Customer Feedback / Customer Feedback / Error Report Change Request? Change Request? N x Incidents Yläotsikko

22. Change Management Sometimes founded defect can lead to change Bug? Change Request? Not Clear Requirements Not Clear Requirements Feature ? Yläotsikko

23. Error Reporting and Process Yläotsikko

24. Hands On: Bugzilla Error Database http://www.bugzilla.org/ http://www.bugzilla.org/installat ion-list/ http://www.bugzilla.org/ http://www.bugzilla.org/installat ion-list/ What is Bugzilla? Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively. Most commercial defect-tracking software vendors charge enormous licensing fees. Despite being "free", Bugzilla has many features its expensive counterparts lack. Consequently, Bugzilla has quickly become a favorite of thousands of organizations across the globe. What is Bugzilla? Bugzilla is a "Defect Tracking System" or "Bug-Tracking System". Defect Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively. Most commercial defect-tracking software vendors charge enormous licensing fees. Despite being "free", Bugzilla has many features its expensive counterparts lack. Consequently, Bugzilla has quickly become a favorite of thousands of organizations across the globe. Yläotsikko

25. Other Defect Database Solutions JIRA – Commercial Requisite Pro – Commercial Rational Synergy - Commercial Mantis – Open Source Yläotsikko

27. Bugzilla Yläotsikko

28. Reporting, Metrics and daily usage Yläotsikko

29. CMMI Process framework CMMI – covers ”error management” in several process areas SP3.2 Analyze Verification results Typical Work Products: Trouble reports - Analyze the verificationd ata on defects - Record all results of the analysis in a report - Provide infromation on how defects can be resolved (including verfiocation methods, criteria, and verification environment) and initiate corrective action Project Monitoring and Control SG2 Manage Corrective Action to Closure SP2.1 Analyze Issues SP2.2 Take corrective Action SP2.3 Manage corrective Action SG2 Validate Product or Product Components SP2.2 Analyze Validation Results. Change request management & Configuration Management process SG2 Track and Control Changes SP2.1 Track Change Requests SP2.2 Control Configuration Items Yläotsikko

30. Traditional SW Project vs Open Source Project Open Source – Crowd Sourcing SW Relase tested without coordination by group of volunteers Release tested by customer Field Testing Test Group Yläotsikko

32. Example of IFDK product concept IFDK = Internal Flame Drum Kit Requirement Management

33. Example of product architecture Yläotsikko

34. Customer/Business Requirements Customer/Business Requirements Sub System Requirements Sub System Requirements Component Requirements Component Requirements Component / Unit Testing Component / Unit Testing Integration Testing Integration Testing System Testing System Testing Acceptance Testing Acceptance Testing System Requirements System Requirements IFDK System Verification and Validation IFDK System Verification and Validation Use Cases User Storys Features Requirements Validation = Are we building the right product? Verification = Are we building the product right? Validation = Are we building the right product? Verification = Are we building the product right? Architecture& Design& Implementation Architecture& Design& Implementation Product VALIDATION VERIFICATION IFDK Product Ideas Yläotsikko VALIDATION vs VERIFICATION?

35. http://www.sysml.org/ UML OMT System Engineering Google: SysML, UML, Systems engineering, Software Design, Code Software Engineering - Architecture SysML UML OMT.NET, JAVA, C++ Software Engineering - Design Requirements http://www.sysml.org/

36. Yläotsikko Seller Customer Developer Different aspects to product QA Tester

37. Different requirement levels Customer/Business/Stake Holder Requirements System Requirements Design Requirements Component Requirements Implementation Scalability Stability Performance Security Performance Stress Usabilty

38. System Testing Integration Testing Unit Testing Customer/Business/Stake Holder Requirements System Requirements Design Requirements Component Requirements Acceptance Testing Why we need requirements from testing point of view? Implementation „Developer's Area“ „Test Engineers Area“ ”Traditional Testing Levels”

39. Customer Requirements Sub System Requirements Component RequirementsComponent/Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Verification and Validation Verification = Are we building the product right? Validation = Are we building the right product? Yläotsikko

40. IDEAL Requirements help Design Requirements help Design REQ-X REQ-Y REQ-Z REQ-O REQ Yläotsikko

41. Defines Feature X * n Functional Requirements Functional Requirements Non-Functional Requirements Non-Functional Requirements Use Cases Vision of product Simple Requirement Management Process Problem DomainSolution Domain Solution Proposal Customer/Marketing/ business User Storys FEATURE VISION/NEED/PROPOSA L FEATURE VISION/NEED/PROPOSA L YOU! Design documents & implementation Test Case Yläotsikko

42. Features Test Case Use Cases User Storys Product Design & Implementation Requirements Test Case Ready to test Testing & Quality Assurance Can we deliver Product Ready to Deliver ? Customer Not ready to deliver Yläotsikko

43. SW Development Process (Waterfall) Requirement Gathering/Evaluatio n Requirement Gathering/Evaluatio n Design Implementation Verification & Validation Verification & Validation Maintenance Error Managment Process Error Managment Process Task1 Mile Stone 1 Mile Stone 2 Mile Stone 3 Task1

44. Black Box vs White Box Testing Unit Testing is White Box testing Black Box Testing for selected component ? ? ? ? Yläotsikko

45. Component /Unit Testing Class Attributes Class Methods Class Attributes Class Methods Class Attributes Class Methods Class Attributes TestClass Methods Unit Test Frame Work Result Yläotsikko

46. Component Testing Customer/Business Requirements Customer/Business Requirements Sub System Requirements Sub System Requirements Component Requirements Component Requirements Component / Unit Testing Component / Unit Testing Integration Testing Integration Testing System Testing System Testing Acceptance Testing Acceptance Testing System Requirements System Requirements Architecture& Design& Implementatio n Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Yläotsikko

47. Create a Test Case! Functional? Your Sources For Test Case Functional Test Case: ● Verify functionality of XXXX Non-Functional Test Cases ● Verify Stability of XXXX ● Verify Performance of XXX ● Verify Security of XXXX ● Verify Usability of XXXX ● Verify Scalability of XXXX ● etc... Non-Functional? Which Type? ● Requirement ● Use Case ● Feature ● User Story ● Customer's Idea ● Brainstorm ● Intitution ● Exploratory Check also..... ● Correct functionality path ● Miss-usage of functionality ● Boundary Check Check also..... ● Check Possiblity to automated testing? How to create Test Case??? Regression Test Case?? Write a Case Yläotsikko

48. Heading -Sed posuere interdum sem. -Quisque ligula eros ullamcorper quis, lacinia quis facilisis sed sapien. -Mauris varius diam vitae arcu. Sed arcu lectus auctor vitae, consectetuer et venenatis eget velit. -Sed augue orci, lacinia eu tincidunt et eleifend nec lacus. Yläotsikko

49. Feature X * n Calory Counter: Player can measure calories during training session. This can be seen as exercise result in web service eg. Facebook application Energy usage Yläotsikko

50. SW Development Process (Agile) User Story Y Sprint Task1 Task2 Task3 Sprint Task4 Task5 Task6 User Story X Design Implementation Verification Design Implementation Verification Product Backlog Design Implementation Verification Design Implementation Verification Design Implementation Verification Design Implementation Verification Design Implementation Verification Design Implementation Verification User Story Z Task7 Task8 Task9 User Story Z Yläotsikko

51. Testing according ISEB standard Functional Testing Testing based on an analysis of the specification of the functionality of a component or system. See also black box testing.) Non-Functional Testing Testing the attributes of a component or system that do not relate to functionality, e.g. scalabilty, stability, reliability, efficiency, usability, maintainability and portability. http://www.bcs.org/upload/pdf/glossary-current.pdf Yläotsikko

52. We need to capsule all projects as sub projects, which have more independence Project Teams has to have own test engineer –> Integration Test Engineer this lead’s to better defect prevention Integration Test Engineer is part of project team, part of team at start of project Test Engineer provides valuable information for design and enables better testability for project product Define new roles for validation engineers Check next page Lower level for test automation Enables regression test (method to block new implementation defects) Support for executing more complex test scenarios Continuous test execution (stability, performance gain etc) Notes Yläotsikko

53. Test Level1 – Component/Unit testing Component/Class level unit testing (eg. xUnit framework) Test Level2 - Integration Testing (Feature, Application) Integration test for feature/application with “stub” interface components Functionality SW Testing Non-Functional SW Testing according needs (eg. feature/application specific performance, stability) Test Level3 – Regression Testing (Target HW+Android platform, tool, terminal etc.) Functional SW testing for all possible features Non-Functional SW testing according needs Non-Functional HW Testing according needs Test Level4 – System Integration Testing (Target Platform, Tool, Terminal etc) Functional SW testing for all new features Non-Functional SW Performance, Stability, Scalability Test Level5 – Acceptance Testing Functional/Non-Functional test according customer requirements HWT1 – Hardware Testing HW Integration test with limited environment. Conducted performance verified Testing Level descriptions Yläotsikko

54. VERIFICATION & VALIDATION Validation = Are we building the right product? Verification = Are we building the product right? Yläotsikko

55. Integration testing using emulator Yläotsikko

57. Test Case SUT/DUT IFDK What is generated as results from test case execution LOG FILE EVENTS NOTIFICATIONS SUT = System Under Test DUT = Device Under Test ENVIRONMENT ANDROID EMULATOR TOOLS scripts/grep TEST CASE Yläotsikko

58. How to verify component implementation -Unit Testing -Code Coverage -Branch Coverage -Complexity Analyse -Unit Testing -Code Coverage -Branch Coverage -Complexity Analyse Yläotsikko

59. Code Complexity Example tool CCCC Yläotsikko

60. Branch coverage The percentage of branches that have been exercised by a test suite. 100% branch coverage implies both 100% decision coverage and 100% statement coverage. Yläotsikko

61. Code Coverage An analysis method that determines which parts of the software have been executed (covered) by the test suite and which parts have not been executed, e.g. statement coverage, decision coverage or condition coverage. http://en.wikipedia.org/wiki/Code_coverage http://www.atlassian.com/software/clover/ Yläotsikko

62. Release/Configuration Management & Integration Testing Day 6 Yläotsikko

63. Putting all tools together! Continous Integration http://hudson-ci.org/ Yläotsikko

64. Design verification - Unit Testing Yläotsikko

65. Unit Test example Re-run with Meego Example of xUnit in QT environment Yläotsikko

66. Integration Testing Customer/Business Requirements Sub System Requirements Component Requirements Component / Unit Testing Integration Testing System Testing Acceptance Testing System Requirements Architecture& Design& Implementatio n Product VALIDATION VERIFICATION Yläotsikko

67. Why Integrate first? Avoid Big Bang! HW Component Data Base Component/Application 10% tested Component/Application 10% tested Web Service Tested Component/Application Yläotsikko

68. Integration Test with stubs Tested Component/Application Log STUB/MOCK Component Scripted STUB Interface Control Configure Control Configure Simulated Interface Simulated Interface Messages/Events STUB/MOCK Component Control Interface Yläotsikko

69. in practice #1 IFDK android setup Tested Component Application Tested Component Application Activate/Control STUB/MOCK Component Scripted STUB Interface Control Configure Control Configure Simulated Interface Simulated Interface Messages/Events WEB SERVER simulating Service interface WEB SERVER simulating Service interface Control Interface Trace/Log Yläotsikko

70. in practice #2 server component testing Tested Component Application Tested Component Application Trace/Log Activate/Control Mock Server/Daemon Scripted STUB Interface Automated Test Interface Automated Test Interface Simulated Interface Simulated Interface Messages/Events WEB SERVER Control Interface Operating System Needed Fake Application Needed Fake Application Junit Scripted Interface Yläotsikko

71. Release Management and Integration Testing IFDK Application Facebook Web Service HW Component Calore Meter Enabled Drum Stick HW Component Calore Meter Enabled Drum Stick Calore Meter SW Component 10% tested Calore Meter SW Component 10% tested Data Base Schema Design Stubs neede d Stubs neede d Stubs neede d Stubs neede d Stubs neede d Stubs neede d Stubs neede d Stubs neede d First System Test With all components First System Test With all components Yläotsikko

72. Trace/Log as feedback Log contains important information. Log Simple tool for log analysing in Linux ”grep” command, TAIL -F /var/log/messages | grep error -Specific Inhouse log capturing and analyse tools Yläotsikko

73. Agenda Brief guidance for NEST Project Platform 1.3 Background story for reference project IFDK Some Definitions SW Development Process Release + Configuration Management Testing Levels and Error Management Hands On: Test Link + Bugzilla Error Reporting, Metrics and daily usage Closed Software Project vs Open Source Project Discussion Yläotsikko

75. Release Management Version 0.1 Version 0.2 Version 0.3 Version 0.2.1 Version 0.2.2.1 Version 0.2.2 Version 0.4 Trunk Customer 1 Version 0.2.3 Version 0.2.2.2 Version 0.2.2.3 Yläotsikko

76. Release & Configuration Managmement Version 0.1 Version 0.2 Version 0.3 Version 0.2.1 Version 0.2.2.1 Version 0.2.2 Version 0.4 Trunk Customer 1 Version 0.2.3 Version 0.2.2.2 Version 0.2.2.3 Feature s Release 1.0 Feature s Yläotsikko

77. Release Planning Yläotsikko

79. Validaton& Verificaton (Testing) Management Version 0.4 Version 0.2.2.2 Version 0.2.2 Test Plan Test Cases For Features Test Plan Test Cases For Features Tested Release/configuration Error/De fect Report Error/De fect Report Error/De fect Report Error/De fect Report Error/De fect Report Error/De fect Report Yläotsikko

80. Feature Pack Project Manager Designer/Coder Integration Test Engineer Integration Test Engineer Test Manager System Testing IFDK System Testing Feature Unit/Integration Testing IFDK System Acceptance Testing System Test Engineer System Test Engineer Test Automation Engineer Test Automation Engineer Acceptance Test Engineer Acceptance Test Engineer Validation Verification IFDK Verification/Validation (Testing Organization) Regression Testing Integration Testing System Testing Acceptance Testing Test Automation Unit Testing Product Release Yläotsikko