outcome Agile Testing

2© 2010 OutSystems – all rights reserved How do you ensure the quality of your delivery?

3© 2010 OutSystems – all rights reserved Purpose Structure Functionality Process Contents

4© 2010 OutSystems – all rights reserved Assures Tests are performed consistently Tries to keep it simple and nimble Easily reusable, extendable and collaborative Creates a framework for Quality Assurance Creates roadmap through the application Completes the development cycle The proof you need Purpose

5© 2010 OutSystems – all rights reserved You’ve got your brand new app, just out the 1st Sprint… To ensure quality, you’ll need to perform some tests… Structure – Test Cases

6© 2010 OutSystems – all rights reserved Test Group Edit Customer Test Group Structure – Test Groups  Page Loads  Apply Filters  Sort by Column  Page Loads  Save Action  Delete Action Test Cases Test Cases Customer List Screen Test Groups aggregate Test Cases, covering a Screen, for example.

7© 2010 OutSystems – all rights reserved Use Case Test Group Activate Customer Screen Test Group Test Groups come in 2 types: –Screen –Use Case Screen Test Groups can be seen as Unit Tests for a Screen. –To simplify the creation of these tests, Test Templates can be used, OR; –Clones from other existing Screen Test Groups –More on both options later… Use Case Test Groups are Functional Tests which can be transversal to screens and application areas Structure – Test Groups  Page Loads  Apply Filters  Sort by Column  Profile can see Activate Button  Activate changes customer status  Customer gets notified by Customer List

8© 2010 OutSystems – all rights reserved Test Module Test Group Edit Customer Test Group Structure – Test Modules  Page Loads  Apply Filters  Sort by Column … Customer List Customer Center Test Groups are organized into Test Modules Test Modules represent large areas of the application Test Group Activate Customer …

9© 2010 OutSystems – all rights reserved Edit Customer Structure – Hierarchy  Page Loads  Apply Filters  Sort by Column  Page Loads  Save Action  Delete Action Customer List Customer Center Test Group  Test Case 1  Test Case 2  Test Case 3  Test Case 1  Test Case 2  Test Case 3 Test Group Test Module Confirm record was deleted. Was feedback provided? Description

10© 2010 OutSystems – all rights reserved To a set of Test Modules, we call a Test Run. It’s called a Test Run, because it will be run several times, from release to release, or Sprint to Sprint. With each Sprint, the test set will evolve, adding new tests, groups and/or modules, and changing existing ones. Test Runs are independent from each other, so its struture and content can be freely modified. The Test Results are also independent from Test Run to Test Run, so perfect history is kept. Structure – Test Run

11© 2010 OutSystems – all rights reserved Structure – Test Runs Sprint 1Sprint 2Sprint 3 Edit Customer  Page Loads  Apply Filters  Sort by Column  Page Loads  Save Action  Delete Action Customer List Customer Center Edit Customer  Page Loads  Apply Filters  Sort by Column  Page Loads  Save Action  Disable Action Customer List Customer Center  Page Loads  Apply Filters  Sort by Column Contact List Contact Center Change Add Edit Customer  Page Loads  Apply Filters  Sort by Column  Page Loads  Save Action  Disable Action Customer List Customer Center  Page Loads  Apply Filters  Sort by Column Contact List Contact Center Edit Contact … Test Run 1 Test Run 2 Test Run 3+

12© 2010 OutSystems – all rights reserved Outcome supports several solutions, or projects, at the same time. We call these umbrellas a Test Suite. Structure – Test Suite CRM Billing Campaigns Edit Customer Page Loads Apply Filters Sort by Column Page Loads Save Action Delete Action Customer List Customer Center Test RunsTest Suites Test Modules Test Groups Test Cases Sprint 1 Sprint 2 Sprint 3

13© 2010 OutSystems – all rights reserved There are very common patterns when building applications –List Screen –Edit Record Screen –Show Record Screen –etc. Each of them with usual tests. So why not create Test Templates for each Screen? Test Group Templates contain common tests to add to Test Group for common screens. Structure – Test Templates

14© 2010 OutSystems – all rights reserved For Example, for a List Screen: Structure – Test Templates

15© 2010 OutSystems – all rights reserved A Test Case, starting from Not Run, can have 3 possible results: Success – Test executed as expected. Success, but… – Test was successful, but some glitch was identified. Failed – Test was not successful. Structure – Test Results Success Success, but… Not Run Failed

16© 2010 OutSystems – all rights reserved Structure – Test Results Edit Customer  Page Loads  Apply Filters  Sort by Column  Page Loads  Save Action  Delete Action Customer List Customer Center Success FailedDatabase error ocurred ORA-2341 Unkown… Success, but…Typo in feedback message Success Confirm record was deleted. Was feedback provided? Be detailed! Nice ! Uoh, oh!!

17© 2010 OutSystems – all rights reserved Functionality Introducing outcome

18© 2010 OutSystems – all rights reserved When a Test Suite is created, its first Test Run is created. Export All Data can be used to backup all application data, when database backups are not in place. Functionality – Test Suites New Test Suite Import/Export All Data Edit Test Suite Export Test Suite Delete Test Suite Help on Test Suites

19© 2010 OutSystems – all rights reserved Full Test Runs contain all tests from your application. It represents your full application. Delivery Test Runs contain only the most recently added or modified tests, along with the Regression Tests. Functionality – Test Runs New Test Run Edit Test Run Delete Test Run Help on Test Runs more on this later

20© 2010 OutSystems – all rights reserved Functionality – Test Run Overview Help on Test Run Overview Test Distribution per Status New Test Module Edit Test Group Delete Test Group Clone Test Group Go to Group’s Tests View All Tests for Test Run Manage Test Templates

21© 2010 OutSystems – all rights reserved Functionality – Test List Add Test to Module New Test Add Test to Group Edit Test Select Module Select Group New Module New Group Attach File

22© 2010 OutSystems – all rights reserved Functionality – Test List Test Result DescriptionSet ResultDelete Test Clone Test Attach Result File Filter by Module Filter by GroupFilter by Name or Description Filter by Status, (Recent), (Regression) or (Not Fixed) Set as Regression Test Regression Tests will be used in Delivery Test Runs.

23© 2010 OutSystems – all rights reserved When there are a lot of failed tests, it might be useful to prioritize and even estimate effort for resolution. The failed tests can be exported in excel, in a format which is compliant with the ANP (ex-AMT) Issue Import excel format. Functionality – Manage Failed Tests

24© 2010 OutSystems – all rights reserved Functionality Using outcome

25© 2010 OutSystems – all rights reserved How deep and detailed should the Test Cases be? –Outcome allows you to keep it real, specifying the tests at te level you have time to do. Keep it concise, non-formal. –Nevertheless, you can, and should, review the Test Cases, with each run, adding detail and context. Enough to be understood and executed by a team colleague. –In the end, it’ll be a prime tool to fully describe your application’s functionality – it becomes a great starting point for any new team element. Developers, testers or analysts. Process – Test Cases Construction

26© 2010 OutSystems – all rights reserved Some extra hints for Building the Tests –Seems to work best when built from an Applications User’s perspective – easier to follow and get into. –Some Test Modules/Groups/Cases are specific to some profile. Be sure to add the Test User with the specific role or permissions. ex. Test with Anna (Catalog Manager), password – not applicable to production environment tests. –It’s also common to have a comprehensive, common-sense tempered, correspondence between: Test Modules  ANP Stories Test Groups  Work Items Structure – Test Cases

27© 2010 OutSystems – all rights reserved So, how can it be used in projects? Process – Sprint 1 Sprint 1 1st Test Run (Full) Run Tests (again) before Delivery Run Tests as part of development Create the Test Suite for project, which will create the 1st Full Test Run. Developers should create the tests for the developed funcionality. Create/review/run Test Cases as part of Work Item development. Usually, the Test set will begin by “Screen” Test Groups (unit tests) and evolve into more business- oriented tests, in the form of “Use Case” Test Groups. At the end of the first Sprint, a new Test Run can be created, or new executions of same tests can be performed, to ensure the quality of the delivery.

28© 2010 OutSystems – all rights reserved At the beggining of the next Sprint… Create the 2nd Full Test Run, which will contain all the tests from the previous Full Test Run. These tests can be freely modified, as this set of tests is completely independent from the previous Test Run. Process – Sprint 2+ Sprint 1Sprint 2 2nd Test Run (Full) Run Tests as part of development

29© 2010 OutSystems – all rights reserved At the end of the Sprint… At the end of the Sprint, create a Delivery Test Run. A Delivery Test Run contains all added or modified Tests, in the last Full Test Run, along with the identified Regression Tests. The Regression Tests are the tests which are repeated every Sprint before delivery to make sure nothing critical is broken. Process – Sprint 2+ Sprint 1Sprint 2 2nd Test Run (Full) Delivery Test Run Run Tests as part of development

30© 2010 OutSystems – all rights reserved Process – Sprint 2+ The Delivery Test Run naturally comprises the contents for the Sprint Demo Script. –Added and Modified Tests Thanks to Paulo Ramos for the hints

31© 2010 OutSystems – all rights reserved Sometimes acting it not enough. You must show it. When your client requires Quality Assurance evidence: –Export the Test Suite or latest Full Test Run. Output format is Excel. –Format it into a nice document. Process – Evidence

32© 2010 OutSystems – all rights reserved Happy Testing! Please provide feedback to