As Simple As Possible, But No Simpler Sam Guckenheimer

Simple Project Management Functionality Quality Resources Time “The Iron Triangle” (err… tetrahedron)

21 st Century Mantra Do more with less! But if your only variables are: But if your only variables are:  Functionality  Quality  Resources  Time …then how are you going to do that? …then how are you going to do that?

An Older Truth Happy families are all alike; every unhappy family is unhappy in its own way. Tolstoy, Anna Karenina Все счастливые семьи похожи друг на друга, каждая несчастливая семья несчастлива по-своему.

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Code Some why-nots Some why-nots  Use managed code  Use modern frameworks  Use service- oriented architecture  Use available tools Transparency Transparency  Responsible costing  Visible results Available tools Available tools  Unit tests  Code coverage  Static analysis  Profiling performance  Source control  Work item tracking  Build automation

Unit Tests and Code Coverage Unit Test Results Code Under Test not covered during the test run

Code Analysis Code Analysis recommendations as build warnings Direct jump to code from the warning

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Product Definition Personas and Scenarios Personas and Scenarios Qualities of Service Qualities of Service Capture implicit requirements Capture implicit requirements  Kano analysis Stack ranking Stack ranking Continually challenge your assumptions!

Personas and Scenarios PM Starts New Portfolio Project PM Enumerates Requirements in Excel PM Schedules Work in MS Project PM Monitors Project Status PM Reviews Project Status PM Promotes For Deployment CEO Signs Contract Architect Updates Design Architect Adds Tasks & Checks In Dev Writes Code Dev Writes & Runts Unit Tests Dev Reviews Work Dev Runs Code Analysis Dev Writes Load Tests Dev Checks In Work Dev Diagnoses & Fixes Dev Checks In Work Tester Checks Build Status Tester Runs Load Test Tester Reports Bug PROJECT MANAGEMENTARCHITECTDEVELOPERTEST Jacqui Ackerman Project Manager Art Benson Architect Martin Gaines Developer Renee Davis Tester Renee Davis Tester

Qualities of Service Performance Performance  Responsiveness  Concurrency  Efficiency  Fault tolerance  Scalability Trustworthiness Trustworthiness  Security  Privacy  Conformance to standards  Interoperability Usability Usability  Accessibility  Attractiveness  Compatibility  Discoverability  Ease of use  Localizability Manageability Manageability  Availability  Reliability  Installability and uninstallability  Maintainability  Monitorability  Recoverability  Testability  Supportability

Kano Analysis Hinshitsu (Quality), The Journal of the Japanese Society for Quality Control, XIV:2, pp.39-48, April 1984

Challenging Assumptions Customer in usability lab Customer’s desktop

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Architecture Service-Oriented Architecture Service-Oriented Architecture Infrastructure Architecture Infrastructure Architecture Legacy Legacy

Service Orientation Build systems using autonomous services that adhere to the four tenets of Service Orientation: 1. Boundaries are explicit 2. Services are autonomous 3. Services share schema and contract, not class 4. Service compatibility is determined based on policy

Application Designer Service-Oriented Architecture model Port Details editor

Infrastructure Architecture Points of Failure Points of Failure Points of Observation Points of Observation Points of Attack Points of Attack Manageability Manageability

Logical Infrastructure Designer Services assigned to logical infrastructure Architecture validated against operational settings and constraints

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Build Automation Nightly build Nightly build  Project heartbeat Pre check-in tests Pre check-in tests  Validation of code prior against current base prior to check-in  Variant is continuous integration Build verification tests Build verification tests  Functional tests (from unit tests)  Component integration tests Build reporting Build reporting  Against backlog, by check-in/changeset

Build Reporting

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Versions Track versions for each of Track versions for each of  Source  Tests  Executables and other runtimes you create  XML, HTML, images, docs & databases  Environmental/deployment components  Bugs Report them together & relate them Report them together & relate them

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Environment Production environment Production environment Test environment Test environment Capturing environment Capturing environment Tools Tools  Microsoft Virtual PC  Microsoft Virtual Server Maintain lab images Maintain lab images

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Security The core problem The core problem Threat modeling Threat modeling Code analysis Code analysis Security testing Security testing Michael Howard, Writing Secure Code, 2003 J.D. Meier et al., Improving Web Application Security, 2003

Security: Core Problem Odds of securing a single level is 1 / ∞ Odds of securing a single level is 1 / ∞  Bad guy has to find only one vulnerability  Infinite time Microsoft as example Microsoft as example  100’s of different IT environments  2,500 unique attacks per day  125,000 incoming virus-infected s per month Need to secure at every level Need to secure at every level  Design  Default  Deployment Multiple layers of defense needed Multiple layers of defense needed

Threat Modeling Analyze the design for vulnerability Analyze the design for vulnerability Model data flows Model data flows  S- Spoofing Identity  T- Tampering with Data  R- Repudiation  I- Information Disclosure  D- Denial of Service  E- Elevation of Privilege

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Performance Deployment configuration Deployment configuration  Model performance as part of product definition  Replicate environment in lab  Test it as part of development  Fix it where it hurts Three-tiered problem Three-tiered problem  System  Components  Code

System and Component Performance measures of test and Systems Under Test Alerts and warnings on Systems Under Test

Code Performance Timeline of memory consumption Suspect functions, drillable to code

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Manageability Operations documented and current for every service or application Operations documented and current for every service or application Service level agreement in place Service level agreement in place Security scanning in place Security scanning in place Proactively monitor and fix Proactively monitor and fix Reactive and proactive problem management Reactive and proactive problem management

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Testing Mission & Approach Marick’s Framework Marick’s Framework Different missions and approaches apply for each quadrant Different missions and approaches apply for each quadrant Technology Facing Business Facing Support Programming Critique Product

Let the punishment fit the crime! A good test approach is: A good test approach is:  Diversified  Risk-focused  Product-specific  Practical  Defensible Fit the technique and its data to its purpose in the quadrant Fit the technique and its data to its purpose in the quadrant Gilbert & Sullivan, The Mikado Kaner, Bach & Pettichord, Lessons Learned in Software Testing, 2002

Testing Mission & Approach Representative techniques Technology Facing Business Facing Support Programming Unit testing, code coverage, code analysis Test-Driven Development Granularity matches code Discrete scenarios Example-driven data Realistic 80% cases Prioritized regression testing Critique Product Specialize by QoS Model-driven tests Generated data Exploratory testing Soap operas

Test Coverage Identify the Scenario, QoS or Code that the test tests Identify the Scenario, QoS or Code that the test tests  If they’re newly discovered, capture them  If you can’t name them, question the value of the test Measure coverage against these dimensions Measure coverage against these dimensions

Test Automation and Its Discontents Technology Facing Business Facing Support Programming Lowest cost Critique Product Highest cost ROI= Σ t (Value of Information) - Σ t (Cost to Maintain) Σ t (Cost to Implement) (adjusted for net present value and risk)

Test Automation and Its Discontents Value depends on context Value depends on context Automation is a programming exercise Automation is a programming exercise Opportunity cost high due to resource constraints Opportunity cost high due to resource constraints Options theory problem Options theory problem  Very sensitive to volatility  Often incalculable ROI= Σ t (Value of Information) - Σ t (Cost to Maintain) Σ t (Cost to Implement)

Testing Web Applications View of content as rendered Content validation http request & response Performance breakdown Data substitution

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Metrics Consider many dimensions at once Consider many dimensions at once  Single metrics easily mislead  Test results  Bug rates  Code churn  Code coverage  Requirements coverage  Never use metrics for reward or punishment Flow of value, not completion of tasks Flow of value, not completion of tasks Planned and unplanned work Planned and unplanned work Robert Austin, Measuring and Managing Performance In Organizations, 1996

Which Component is Healthiest? Contrast two views of project data Contrast two views of project data Fewest bugsHighest test pass rate

Which Component is Healthiest? Conclusions: Conclusions:  Tests are stale  Highest risk here Lowest code coverage Highest code churn

Focus on Flow of Value David J. Anderson, Managing with Cumulative Flow, Control height of work in progress Value measured on completion

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Processes Differ for Good Reasons… Economics Economics  Regulation  Liability Plan-Driven vs. Adaptive Plan-Driven vs. Adaptive Iteration length Iteration length Documentation required Documentation required Sign-off gates Sign-off gates Time tracking requirements Time tracking requirements

Infrastructure Architect Project Manager Developer Tester Solution Architect Business Stakeholder …and for Bad Reasons

Solution is Transparency Infrastructure Architect Solution Architect Project Manager Developer Tester End User

Transparency Single product backlog Single product backlog Task-aware versioning Task-aware versioning Project portals Project portals Process handbook Process handbook

Single Product Backlog Single backlog of all Work Items (Reqts, Tasks, Bugs, etc.) Queries to filter, view, report Details for each entry Complete change history

Task-aware Versioning Source files to check in … …with Work Items done… …and Check-in Notes and Policy Status

Project Portal

Process Handbook

13 Symptoms of Unhappiness It’s the code, stupid! It’s the code, stupid! Actually it’s the requirements! Actually it’s the requirements! No, the problem is that you neglected the architecture! No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build. Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?! What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it? Not code versions, but the environments, don’t you get it? Ever heard of security?! Ever heard of security?! Yeah, but you ignored performance, duh! Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable! So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing? Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway! Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect? With a process like that, what do you expect? It’s our culture – you’ll never change that. It’s our culture – you’ll never change that.

Culture Productivity and predictability Productivity and predictability Responsibility over assignment Responsibility over assignment Team and individual Team and individual Product mentality Product mentality

13 Symptoms of Unhappiness  It’s the code, stupid!  Actually it’s the requirements!  No, the problem is that you neglected the architecture!  Architecture, schmarchitecture. I just want a working build.  What good is that the way we mix up versions?!  Not code versions, but the environments, don’t you get it?  Ever heard of security?!  Yeah, but you ignored performance, duh!  So what if it worked in the lab -- it’s still unmanageable!  Oh, and did we mention testing?  Since you’re not measuring it, you can’t manage it anyway!  With a process like that, what do you expect?  It’s our culture – you’ll never change that.

Sam Guckenheimer