1. Software testing in Elekta
November 30, 2012
Software testing in Elekta
Hans Axelsson
Olle Morell

2. Agenda Presentation Elekta Radiation Therapy
November 30, 2012
November 30, 2012
Agenda
Presentation Elekta
Radiation Therapy
External Beam Therapy - Brachy Therapy
Treatment Planning
Regulations and Law – FDA - Therac 25
Prioritization
General Test concepts
What is testing?
Test Methodologies
Testing as a profession
Working in a team
Questions 2

3. Elekta the Company Elekta was formed 1972, headquarters in Stockholm
November 30, 2012
Elekta the Company
Elekta was formed 1972, headquarters in Stockholm
Since 1997 Elekta has acquired Philips’ radiotherapy division and various other radiotherapy companies over the world
The Uppsala office was acquired by Elekta in 2011; we have been around since 1986 in various incarnations
Elekta makes the Leksell Gamma knife, linear accelerators, information management and treatment planning systems, etc.

4. Elekta the Company Over 3000 employees world wide
November 30, 2012
Elekta the Company
Over 3000 employees world wide
On the Stockholm stock exchange
Involved in the treatment of almost 1,000,000 patients every year.
Research collaborations, sponsors PhD students, scientific publications

5. Radiation Therapy Cure or help patients with cancer
November 30, 2012
Radiation Therapy
Cure or help patients with cancer
Used by itself, or together with surgery or chemotherapy
Cancer is basically uncontrolled cell division
Attack with high-energy radiation to break DNA
Given in ”fractions”, allowing normal tissue to heal

6. External Beam Therapy – Linear Accelerator
November 30, 2012
External Beam Therapy – Linear Accelerator

7. External Beam Therapy – Linear Accelerator
November 30, 2012
External Beam Therapy – Linear Accelerator

8. Brachytherapy - Afterloader
November 30, 2012
Brachytherapy - Afterloader

9. Brachytherapy - Applicators
November 30, 2012
Brachytherapy - Applicators

10. Treatment Planning Planning is done to get
November 30, 2012
Treatment Planning
Planning is done to get
Enough radiation into the tumor
As little as possible into organs at risk
Simulates the treatment in software, calculating dose
Evaluated by doctor to determine best plan

11. Importing images (CT, MR, PET, …) Defining patient volume
November 30, 2012
Treatment planning systems are rather complex, Oncentra is around 2,000,000 lines of source code
Importing images (CT, MR, PET, …)
Defining patient volume
Creation of plans
Dose calculation
Evaluation
Export of plans

12. November 30, 2012
Treatment Planning

13. November 30, 2012
Regulations and Law
Medical devices require approvals by ”regulatory bodies”
One example is FDA in the USA
Rules are strict, for example about documentation
Products must be validated, for example with test
Not following rules is not good…

14. FDA FDA is a military organisation
November 30, 2012
FDA
FDA is a military organisation
Responsible for food, medication, medtech devices, etc.
Can arrest you, can shut you down, can use guns…
So… is the FDA what we should be afraid of?

15. Therac-25 - Do or Die At http://www.ccnr.org/fatal_dose.html
November 30, 2012
Therac-25 - Do or Die At
you can find the following headline
“In 1985 a Canadian-built radiation-treatment device began blasting holes through patients' bodies.
How a series of simple computer errors sabotaged a state-of-the-art medical wonder.”

16. Live and Let Live So let’s test everything!
November 30, 2012
Live and Let Live
So let’s test everything!
Let’s do Exhaustive Testing!
But what’s that?
Students’ answers found on the net:

17. November 30, 2012
Exhaustive testing?
“All the testers are exhausted. All the planned tests have been executed.”
“Exhaustive testing is the process that causes carbon monoxide to be emitted from ones exhaust.”
Or?
“Exhaustive testing is executing a program with all possible combinations of inputs or values for program variables.”
For all applications of practical value, this is impossible

18. Prioritization Can not test everything in all ways
November 30, 2012
Prioritization
Can not test everything in all ways
Lo and behold: Developers don’t even fix all found bugs
So find the right bugs!
What’s important?
Analyze the product, using people’s skills, specification, etc.
Assess the risks
Test the most important stuff first and work yourself downwards

19. November 30, 2012
Risk Assessment

20. General test concepts - Start
November 30, 2012
General test concepts - Start
Software functionality is often abstract and contains a lot of hidden assumptions.

21. November 30, 2012
General test concepts
Interesting assumptions about software development:
For every 10-percent increase in problem complexity, there is a 100-percent increase in the software solution’s complexity. (Robert L. Glass)
The most important factor in software work is the quality of the programmers. (Jeff Atwood)
Software Errors Cost U.S. Economy $59.5 Billion Annually (NIST 2002)
Mistakes are a part of being human. (Old proverb)

22. General test concepts – The login window
November 30, 2012
General test concepts – The login window
How should it work?
”The user should be able to login using his/her personal user name and password.”
”Only the correct combination user/psw should be let through. If user name or password is incorrect, the user should be told.”
What character encoding should be supported?
If incorrect login, what should we tell the user?
What string length is accepted for input?
What if the user forgot login/password?
How do we treat special characters? (’,”,[,})
Should you be able to copy/paste text?

23. General test concepts – Different views
November 30, 2012
General test concepts – Different views

24. General test concepts – Requirements
November 30, 2012
General test concepts – Requirements
We use different kinds of requirements:
FUNCTIONAL: Describe the expected system functionality or services.
Examples:
As a user, I would like the system to let me store patients.
As a user , I want to be able to edit, add and remove patients.
As a sysadmin, I would like to be able to set different privileges for different users.
NON-FUNCTIONAL: Describe the expected reliability, response time and storage requirements, etc.
Examples:
The retrieval of patient data must not take more than five seconds
The system should be capable of handling up to patients.
It should be possible to export data in a standard text based format.

25. General test concepts – Humans are fallible
November 30, 2012
General test concepts – Humans are fallible
Error types and their origin (From “Reducing error and influencing behavior”, HSE Books)

26. What is testing? Some ideas
November 30, 2012
What is testing? Some ideas
Try it and see if it works. (Basic conception?)
Finding bugs. (Basic conception?)
Questioning a product in order to evaluate it. (James Bach)
A technichal investigation for the purpose of revealing the quality of a software product on behalf of the stake holders. (Cem Kaner)

27. What is testing? – Sounds easy, doesnt it?
November 30, 2012
What is testing? – Sounds easy, doesnt it?
Try it
and see
if it works
Get different versions Set them up Try simple things Try complex things Try sequences Try combinations Try weird things Try them again
Where to look? How to look? What‘s there? What‘s not there What‘s invisible? Did it change? Will it change? How about now?
Read the spec (There‘s no spec? Oh.) (There is a spec! Oh it‘s old and wrong) Find inconsistencies Find obvious problems Find obcsure problems Find BAD problems

28. November 30, 2012

29. What is testing then? „Try to learn sufficiently,
November 30, 2012
What is testing then?
So, testing is (for us) basically about:
„Try to learn sufficiently,
everything that matters
about whether it can work and
how it might not work.“

30. November 30, 2012
Test Methodologies
There are many test techniques, you will hear things like:
black or white box testing
functional testing
decision tables
equivalence partitioning
use-case based testing
etc.
But when are they used?
Some are used purely by developers, and are typically applied by automated low-level unit tests before or after design.
However, for system and integration testing there are some fundamentally different methodologies:

31. November 30, 2012
Scripted Testing
”Script” not as a program, but as pre-written test steps with defined pass and fail criteria
Based on specification or knowledge
Repeatable
Can be written on a level not requiring skill to execute
Can be automated with the right tools
Takes time to write and maintain
Stops finding defects after a while

32. November 30, 2012
Scripted Testing

33. Ad Hoc Testing Just go at it Requires little preparation
November 30, 2012
Ad Hoc Testing
Just go at it
Requires little preparation
Requires skilled tester
Can find many defects
Difficult to repeat
Difficult to show coverage of requirements

34. Exploratory Testing Not Ad Hoc testing
November 30, 2012
Exploratory Testing
Not Ad Hoc testing
Change testing approach based on what is found when testing
Document while testing for repeatability
Efficient in finding bugs
Requires skilled tester
Still issues with demonstrating requirement coverage

35. Session Based Testing Based on Exploratory testing
November 30, 2012
Session Based Testing
Based on Exploratory testing
Undisturbed sessions of about 2 +/- 1 hours
Input is test charter, with test aspects to cover
Output is session log with results, logged defects, and possible input to new sessions
Allowed to deviate some from charter
Suitable for pair testing
For large test efforts, debriefing by test leader
Test coverage can be traced by documenting functionality covered by sessions

36. November 30, 2012
Example of a charter

37. Testing as a profession
November 30, 2012
Testing as a profession
What is expected of a tester?
Curious: A tester wants to know about the domain he/she operates in. If there is something that cannot be understood, a tester is quick to learn and hungry for knowledge. This way of thinking leads many testers to be good in a lot of different areas. They are often Jack of all trades.

38. Testing as a profession
November 30, 2012
Testing as a profession
What is expected of a tester?
Organized: The software development cycle is often tightly time driven.
The tester must keep track of how to reproduce the abnormalities encountered.
The tester need to keep track of the baseline versions and branches where problems are found as well as fixed. The tester is often asked to reproduce bugs. That means you must keep track on the parameters involved in the scenario causing the bug.

39. Testing as a profession
November 30, 2012
Testing as a profession
What is expected of a tester?
Skeptical: Question everything, but choose your main areas of effort. A tester thinks there are several ways of crashing an application or part of it. It’s just a matter of finding it. The basic fact that people are fallible in combination with an ever increasing complexity of development gives the testers work to do.

40. Testing as a profession
November 30, 2012
Testing as a profession
What is expected of a tester?
People Skills: Communication is the most important tool for making changes.
Testing is many times about showing that something DOESN’T work. The tester often have the bad news.
Reporting bugs is not the same as telling developers they are doing a bad job.

41. Testing as a profession
November 30, 2012
Testing as a profession
How is a day af the office?
Setting up systems (Configuration and installation)
Populating data (Generating or deploying a fixed setup)
Modeling and preparing tests (Tactics)
Running tests (Either small series or long integrated chains)
Looking at the right places (Logs, graphs, databases)
Evaluating the result (Comparing and discussing)
Communicating changes, deviations or suspect observations.

42. Working in a team The local team – using SCRUM Application designer
November 30, 2012
Working in a team
The local team – using SCRUM
Under the same roof
Easy to meet
Close social interaction
Application designer
Developers
Product owner
Testers

43. November 30, 2012
Working in a team
Separated departments – often using non-agile methods
Separated geographically
Physical meetings are rare or not at all.
Social interaction mainly through mail or phone, via product management
Product owner
Developers
Testers
Product manager
Project manager
Application designer

44. Memo for testers As a tester, you have a very supportive role.
November 30, 2012
Memo for testers
As a tester, you have a very supportive role.
It is not your job to decide whether the software should be released or not.
Your job is to do your best to show, highlight and inform about abnormalities, bugs and strange behavior in the software under test.
That way, your input becomes an important basis for decisions.

45. Questions? Thank you all for listening!
November 30, 2012
Questions?
Thank you all for listening!
Is there anything more you want to know
or something you didn‘t understand?
Please feel free to ask!