James A. Whittaker Principal Architect Visual Studio Team Test Microsoft blogs.msdn.com/james_whittaker

to err is HUMAN

nothing is PERFECT

WE all make MISTAKES

IT could have happened to ANYONE

We have many expressions to capture the fact that humans are fallible

Our creations and inventions echo this unnerving tendency

to FAIL

if there was an Olympics for FAILURE

Or a World Cup for that which sucks

surely SOFTWARE would reign supreme

Software … … stocks supermarket shelves … delivers electricity and water … stores our personal information … runs nuclear power plants … controls doomsday weapons …

Are these failures typical? How many of you have seen software fail? How many users share this same experience?

Any reason to think things will improve? Let’s take a look at the future and find outfuture

How will we test the applications of tomorrow?

Virtualization Information Visualization

Virtualization Information Visualization

Virtualization

Users are effective testers Let’s leverage their power – They have the environments we seek – They have the data we lack – They run the tests we don’t think of “Crowd-sourcing” is one answer I like Virtualization is the next step

Virtualization The story today: – A user (or tester) finds a bug, developer can’t repro it – A tester wants to reuse test cases, but can’t get them to run Virtualization solves this – User creates a VM at the point of failure These VMs have a market value Developers could have access to the sum total of all the environments in which their application can run – A tester creates VMs for every test suite We now have reusable, fully transferable testing assets An automated historical record of all testing on our app If we can store every video in the world, why not every test case/test data/user environment? – It’s not been possible before but it is now

Virtualization What does the virtualized testing world look like? – Think crowd-sourcing on steroids – Test VM’s have great value, businesses emerge owning and managing these environments – Testers analyze an app, license the appropriate Test VM’s, conduct the test and report results Companies no longer need QA departments Why test a subset of possible usage when (in the future) we can test them all? Software is important, it deserves this kind of coverage

Virtualization Information Visualization

Virtualization Information Visualization

Information

Information about me (Status, Health, etc.) Quick Access to oft- used abilities. Information about my target. Information about my location in the world.

Information What is the information we need? – Architects see Visio diagrams and flow charts – Program Managers see PowerPoint decks and Word storyboards – Developers see Visual Studio – Testers see binaries and interfaces In this model, testers see only what the user sees … I object!

Information What the user sees What the tester sees

Information Testers need more information – The architecture, design, code … they are all useful to testers in the right context – I want to hover over a UI element and see code, data types, value ranges, previous bugs, test history … – ‘Cheating’ is not wrong when you are trying to build a great product!

Virtualization Information Visualization

Virtualization Information Visualization

Well, there’s: – Input – Output – Data flow – Control flow – Modules – Dependencies – Environment variables – Files – Interfaces – Bugs Visualization

Software will continue to run our economy, our businesses and our lives Software will be a critical part of the solutions that will grow food, cure disease and produce sustainable energy Remove software from the equation and humankind’s problems get a lot harder It is imperative that we get software right!

20 years from now – Software testing will look fundamentally different – The tools will be more powerful and allow us to be more impactful – Software testers will contribute more fundamentally to the development process

20 years from now – Will the quality of software be taken for granted? Will users be genuinely surprised when it fails? – Will researchers look back in wonder that there was ever even a need for dedicated bug finders? – What hard problems will simply cease to be hard because the software industry of this time did what needed to be done?

James A. Whittaker, PhD Blog: blogs.msdn.com/james_whittaker URL: msdn.microsoft.com/testercenter