Software testing in Elekta November 30, 2012 Software testing in Elekta Hans Axelsson Olle Morell

Agenda Presentation Elekta Radiation Therapy November 30, 2012 November 30, 2012 Agenda Presentation Elekta Radiation Therapy External Beam Therapy - Brachy Therapy Treatment Planning Regulations and Law – FDA - Therac 25 Prioritization General Test concepts What is testing? Test Methodologies Testing as a profession Working in a team Questions 2

Elekta the Company Elekta was formed 1972, headquarters in Stockholm November 30, 2012 Elekta the Company Elekta was formed 1972, headquarters in Stockholm Since 1997 Elekta has acquired Philips’ radiotherapy division and various other radiotherapy companies over the world The Uppsala office was acquired by Elekta in 2011; we have been around since 1986 in various incarnations Elekta makes the Leksell Gamma knife, linear accelerators, information management and treatment planning systems, etc.

Elekta the Company Over 3000 employees world wide November 30, 2012 Elekta the Company Over 3000 employees world wide On the Stockholm stock exchange Involved in the treatment of almost 1,000,000 patients every year. Research collaborations, sponsors PhD students, scientific publications

Radiation Therapy Cure or help patients with cancer November 30, 2012 Radiation Therapy Cure or help patients with cancer Used by itself, or together with surgery or chemotherapy Cancer is basically uncontrolled cell division Attack with high-energy radiation to break DNA Given in ”fractions”, allowing normal tissue to heal

External Beam Therapy – Linear Accelerator November 30, 2012 External Beam Therapy – Linear Accelerator

External Beam Therapy – Linear Accelerator November 30, 2012 External Beam Therapy – Linear Accelerator

Brachytherapy - Afterloader November 30, 2012 Brachytherapy - Afterloader

Brachytherapy - Applicators November 30, 2012 Brachytherapy - Applicators

Treatment Planning Planning is done to get November 30, 2012 Treatment Planning Planning is done to get Enough radiation into the tumor As little as possible into organs at risk Simulates the treatment in software, calculating dose Evaluated by doctor to determine best plan

Importing images (CT, MR, PET, …) Defining patient volume November 30, 2012 Treatment planning systems are rather complex, Oncentra is around 2,000,000 lines of source code Importing images (CT, MR, PET, …) Defining patient volume Creation of plans Dose calculation Evaluation Export of plans

November 30, 2012 Treatment Planning

November 30, 2012 Regulations and Law Medical devices require approvals by ”regulatory bodies” One example is FDA in the USA Rules are strict, for example about documentation Products must be validated, for example with test Not following rules is not good…

FDA FDA is a military organisation November 30, 2012 FDA FDA is a military organisation Responsible for food, medication, medtech devices, etc. Can arrest you, can shut you down, can use guns… So… is the FDA what we should be afraid of?

Therac-25 - Do or Die At http://www.ccnr.org/fatal_dose.html November 30, 2012 Therac-25 - Do or Die At http://www.ccnr.org/fatal_dose.html you can find the following headline “In 1985 a Canadian-built radiation-treatment device began blasting holes through patients' bodies. How a series of simple computer errors sabotaged a state-of-the-art medical wonder.”

Live and Let Live So let’s test everything! November 30, 2012 Live and Let Live So let’s test everything! Let’s do Exhaustive Testing! But what’s that? Students’ answers found on the net:

November 30, 2012 Exhaustive testing? “All the testers are exhausted. All the planned tests have been executed.” “Exhaustive testing is the process that causes carbon monoxide to be emitted from ones exhaust.” Or? “Exhaustive testing is executing a program with all possible combinations of inputs or values for program variables.” For all applications of practical value, this is impossible

Prioritization Can not test everything in all ways November 30, 2012 Prioritization Can not test everything in all ways Lo and behold: Developers don’t even fix all found bugs So find the right bugs! What’s important? Analyze the product, using people’s skills, specification, etc. Assess the risks Test the most important stuff first and work yourself downwards

November 30, 2012 Risk Assessment

General test concepts - Start November 30, 2012 General test concepts - Start Software functionality is often abstract and contains a lot of hidden assumptions.

November 30, 2012 General test concepts Interesting assumptions about software development: For every 10-percent increase in problem complexity, there is a 100-percent increase in the software solution’s complexity. (Robert L. Glass) The most important factor in software work is the quality of the programmers. (Jeff Atwood) Software Errors Cost U.S. Economy $59.5 Billion Annually (NIST 2002) Mistakes are a part of being human. (Old proverb)

General test concepts – The login window November 30, 2012 General test concepts – The login window How should it work? ”The user should be able to login using his/her personal user name and password.” ”Only the correct combination user/psw should be let through. If user name or password is incorrect, the user should be told.” What character encoding should be supported? If incorrect login, what should we tell the user? What string length is accepted for input? What if the user forgot login/password? How do we treat special characters? (’,”,[,}) Should you be able to copy/paste text?

General test concepts – Different views November 30, 2012 General test concepts – Different views

General test concepts – Requirements November 30, 2012 General test concepts – Requirements We use different kinds of requirements: FUNCTIONAL: Describe the expected system functionality or services. Examples: As a user, I would like the system to let me store patients. As a user , I want to be able to edit, add and remove patients. As a sysadmin, I would like to be able to set different privileges for different users. NON-FUNCTIONAL: Describe the expected reliability, response time and storage requirements, etc. Examples: The retrieval of patient data must not take more than five seconds The system should be capable of handling up to 500.000 patients. It should be possible to export data in a standard text based format.

General test concepts – Humans are fallible November 30, 2012 General test concepts – Humans are fallible Error types and their origin (From “Reducing error and influencing behavior”, HSE Books)

What is testing? Some ideas November 30, 2012 What is testing? Some ideas Try it and see if it works. (Basic conception?) Finding bugs. (Basic conception?) Questioning a product in order to evaluate it. (James Bach) A technichal investigation for the purpose of revealing the quality of a software product on behalf of the stake holders. (Cem Kaner)

What is testing? – Sounds easy, doesnt it? November 30, 2012 What is testing? – Sounds easy, doesnt it? Try it and see if it works Get different versions Set them up Try simple things Try complex things Try sequences Try combinations Try weird things Try them again Where to look? How to look? What‘s there? What‘s not there What‘s invisible? Did it change? Will it change? How about now? Read the spec (There‘s no spec? Oh.) (There is a spec! Oh it‘s old and wrong) Find inconsistencies Find obvious problems Find obcsure problems Find BAD problems

November 30, 2012

What is testing then? „Try to learn sufficiently, November 30, 2012 What is testing then? So, testing is (for us) basically about: „Try to learn sufficiently, everything that matters about whether it can work and how it might not work.“

November 30, 2012 Test Methodologies There are many test techniques, you will hear things like: black or white box testing functional testing decision tables equivalence partitioning use-case based testing etc. But when are they used? Some are used purely by developers, and are typically applied by automated low-level unit tests before or after design. However, for system and integration testing there are some fundamentally different methodologies:

November 30, 2012 Scripted Testing ”Script” not as a program, but as pre-written test steps with defined pass and fail criteria Based on specification or knowledge Repeatable Can be written on a level not requiring skill to execute Can be automated with the right tools Takes time to write and maintain Stops finding defects after a while

November 30, 2012 Scripted Testing

Ad Hoc Testing Just go at it Requires little preparation November 30, 2012 Ad Hoc Testing Just go at it Requires little preparation Requires skilled tester Can find many defects Difficult to repeat Difficult to show coverage of requirements

Exploratory Testing Not Ad Hoc testing November 30, 2012 Exploratory Testing Not Ad Hoc testing Change testing approach based on what is found when testing Document while testing for repeatability Efficient in finding bugs Requires skilled tester Still issues with demonstrating requirement coverage

Session Based Testing Based on Exploratory testing November 30, 2012 Session Based Testing Based on Exploratory testing Undisturbed sessions of about 2 +/- 1 hours Input is test charter, with test aspects to cover Output is session log with results, logged defects, and possible input to new sessions Allowed to deviate some from charter Suitable for pair testing For large test efforts, debriefing by test leader Test coverage can be traced by documenting functionality covered by sessions

November 30, 2012 Example of a charter

Testing as a profession November 30, 2012 Testing as a profession What is expected of a tester? Curious: A tester wants to know about the domain he/she operates in. If there is something that cannot be understood, a tester is quick to learn and hungry for knowledge. This way of thinking leads many testers to be good in a lot of different areas. They are often Jack of all trades.

Testing as a profession November 30, 2012 Testing as a profession What is expected of a tester? Organized: The software development cycle is often tightly time driven. The tester must keep track of how to reproduce the abnormalities encountered. The tester need to keep track of the baseline versions and branches where problems are found as well as fixed. The tester is often asked to reproduce bugs. That means you must keep track on the parameters involved in the scenario causing the bug.

Testing as a profession November 30, 2012 Testing as a profession What is expected of a tester? Skeptical: Question everything, but choose your main areas of effort. A tester thinks there are several ways of crashing an application or part of it. It’s just a matter of finding it. The basic fact that people are fallible in combination with an ever increasing complexity of development gives the testers work to do.

Testing as a profession November 30, 2012 Testing as a profession What is expected of a tester? People Skills: Communication is the most important tool for making changes. Testing is many times about showing that something DOESN’T work. The tester often have the bad news. Reporting bugs is not the same as telling developers they are doing a bad job.

Testing as a profession November 30, 2012 Testing as a profession How is a day af the office? Setting up systems (Configuration and installation) Populating data (Generating or deploying a fixed setup) Modeling and preparing tests (Tactics) Running tests (Either small series or long integrated chains) Looking at the right places (Logs, graphs, databases) Evaluating the result (Comparing and discussing) Communicating changes, deviations or suspect observations.

Working in a team The local team – using SCRUM Application designer November 30, 2012 Working in a team The local team – using SCRUM Under the same roof Easy to meet Close social interaction Application designer Developers Product owner Testers

November 30, 2012 Working in a team Separated departments – often using non-agile methods Separated geographically Physical meetings are rare or not at all. Social interaction mainly through mail or phone, via product management Product owner Developers Testers Product manager Project manager Application designer

Memo for testers As a tester, you have a very supportive role. November 30, 2012 Memo for testers As a tester, you have a very supportive role. It is not your job to decide whether the software should be released or not. Your job is to do your best to show, highlight and inform about abnormalities, bugs and strange behavior in the software under test. That way, your input becomes an important basis for decisions.

Questions? Thank you all for listening! November 30, 2012 Questions? Thank you all for listening! Is there anything more you want to know or something you didn‘t understand? Please feel free to ask!