Running a Successful Beta Test Program Adam Long (CTO) Darren Gage (QA Mgr) Sam Soubra (Dev Mgr) September 2010

Agenda About Us Running a Beta Test Program Runtime Intelligence Service – Overview – Code Demo Questions

About our Company Established in 1995 Independent Software Vendor based in Melbourne, Australia 90% of revenue from exports > 400,000 customers in 150 countries Dell Small Business Excellence Award in 2009 Microsoft Gold Partner

About our Software Realize Visualize Analyse Organize Qualitative Data Analysis – Unstructured Data Documents Audio / Video Images Spreadsheets What MS Excel does for structured numerical data, we do for unstructured non- numerical data

About our Software NVivo 9 –.NET 3.5 Win Forms WPF WCF – SQL Server 2008 R2 Express NVivo Server 9 – Silverlight 3 – SQL Server 2008 R2 Standard

Beta Test What Beta testing is conducted by a sample of customers to identify flaws in feature complete software Generally occurs after alpha testing (unit & system) Why Validate objectives – Market – Features – Quality Improve outcomes – Collaboration / Innovation Assist marketing – Quotes / Case Studies Plan future direction – Opportunities

7 Steps of a Beta Test Program Plan 2 weeks Promote 4 weeks Select 2 weeks Collect 4 weeks Analyse 2 weeks Prioritize 1 week Recognize 1 week

1. Plan Scope – Goals & Success Criteria Approach – Open vs. Closed Responsibilities – Management, Communications, Defects, Suggestions Schedule – Balance between having enough features available to test versus enough time to incorporate feedback

2. Promote Internal – Sales, Marketing, Training External – Existing Customers – Potential Customers Website & Social Media announcements Customer Newsletter Select Customers Best Beta Testers – Interested – Independent – Vocal Avoid pressure to include – Sales Leads – Evaluators

3. Select Online Application – Outline Program Expectations – 20 Questions Technical Environment Market Segment Research Background Competitor Software Motivation Result – 523 Applicants – 78 Testers Selected Selection Process – Top Previous Beta Testers – Top Forum Posters – Subject Matter Experts – Random Applicants Simulation Program to Optimize Spread of Answers to Questions Obtain Signed Agreement

4. Collect Online Webcasts (Citrix GoToWebinar) – 5 recorded videos & 88 feedback forms Online Forums (IP.Board) – 412 suggestion posts & 335 defect posts Online Survey (Survey Monkey) – 58 survey responses Application Analytics (RIS) – 2,448 hours of usage statistics

5. Analyse Identify – Defects – Suggestions Understand – Most liked features – Most disliked features – Most important improvements needed Dogfooding – Used NVivo Beta to analyse feedback Videos Forums Surveys – In conjunction with Application Analytics

6. Prioritize Prioritize Suggestions – 1 Must Address for RTM – 2 Preferably Address for RTM – 3 Consider for Service Pack – 4 Defer for Future Release – 5 Rejected Raise Defects in Bug Tracking

7. Recognize Active Participants (85%) – Activated the software & provided some feedback – Average of 37 hours testing over 4 weeks – Receive free copy of software Top 20 Participants (25%) – Also receive Amazon voucher

Beta Test Tips Allocate sufficient time & resources Observe usage patterns using Application Analytics and encourage testers to explore specific features Reprioritize effort based on final feedback Hold weekly status meetings Communicate outcomes to internal stakeholders

Runtime Intelligence – What is it? The purpose of runtime intelligence as a component of SDLC management is to support better development, support and R&D investment decision making. Leverages technologies, managed services and practices for the collection, integration, analysis, and presentation of application usage levels, patterns and practices

Runtime Intelligence – Why use it? Gauge Effectiveness of the beta test Collection Phase Feature usage Analysis Phase Platform usage Beta tester effectiveness

Runtime Intelligence – Why use it? Improve Decision Making Usage context while Analysing feedback Prioritization Phase Analysis Phase Usage context while Prioritising feedback Understand Quality

Runtime Intelligence – Tool Choice PreEmptive RIS – Features matched those in competitor’s offerings – We had used other PreEmptive tools (Dotfuscator) – Integrated into Visual Studio 2010 Other tools out there; – EQATEC Analytics – Gibraltar Software

Runtime Intelligence – Architecture Instrumented App is run Usage data sent to Data Repository Dashboard and Reports

Runtime Intelligence – Implementation Credentials, Entry point, Endpoint & Security Features & Custom data System and Performance Opt-in policy Configure analytics Inject Instrumentation Package and Deploy Application distribution Beta Mgmt., UX program, Support Business Intelligence & ROI Operations Mgmt. Runtime Intelligence Services

Runtime Intelligence – Services Preemptive Cloud Yours Aggregate Validate Manage Publish Runtime Intelligence endpoint Runtime Intelligence repository Aggregate, validate, manage, & publish Managed or on-premise For internal dev. and multi-tenant constituencies Runtime Intelligence Reports & Dashboards Owner, application, feature, user, & custom RESTful API and export IDE, CRM, ERP, & IT Operations

Runtime Intelligence – Reports and Dashboard PreEmptives RIS Portal

Runtime Intelligence – Export Data is rich You will need to spend time to mine it!

Runtime Intelligence – Export

Runtime Intelligence – Where to next? Move into Production Improve Development Decision Making for future releases Development Performance tuning Usability Requirements User needs Product direction QA Risk based testing Documentation Material coverage Help usage Application Usage

Code Demo Dotfuscator by PreEmptive Solutions Runtime Intelligence defines several message types: – Application and Session Start – Application and Session Stop – Feature – Performance Probe – System Profile – Tamper Detected

Runtime Intelligence Portal’s dashboards To have your application send these messages, you must: Be a Runtime Intelligence Service subscriber Activate Runtime Intelligence from within Dotfuscator Attribute your application with Runtime Intelligence attributes, including Setup and Teardown Run your application through Dotfuscator with the Send Analytics Messages option turned on

Business Attribute

Application Attribute

Setup Attribute

Teardown Attribute

Feature Attribute

Sample Application

Sample Code

ILDASM Before

ILDASM After

Analytical Data

Questions?