Presented to: SOA Brown Bag #12 By: The SWIM Governance Team & Steve Thomas, iTKO Representative Date: January 11, 2012 Federal Aviation Administration System Wide Information Management (SWIM) iTKO LISA

2 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 The product is the entire brand and customer experience Service-oriented products are delivered late, over budget, and with questionable quality… WHY? Innovate Or Die

3 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 Mainframe Client/Server (including packaged apps like SAP, Siebel, Oracle…) Composite (Custom Applications, SOA, Cloud) 1980 … 1985 … 1990 … 1995 … 2000 … 2005 … 2010 … 2015 What iTKO Does  Founded in 1999  LISA Software Suite: –Virtualization, Automated Testing, Pathfinder, Cloud DevTest Environments  Optimize the Application Lifecycle

4 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 UAT Performance Test Integration Test Many Others… New Product Offering New Partner Support Summer Promotions 80+ Fixes Business Requirements 1000s of Technical Requirements Parallel Development Challenges (chicken & egg) Every team needs their own environment but shares only one Testing can’t be automated due to tech & environment issues Developers’ stubs wreak havoc downstream 2M+ Hours Later… 30+ Interdependent Development Teams Design, Develop System Test Manual testing rules because automation has failed, labor and time costs are way too high MTTR of issues is too high and growing-“100 person conference calls” Performance testing often squeezed or meaningless because it’s not production scale Most of their defects, even requirements and design, are discovered here Integration testing waits until every system is ready–this wrecks the schedule One shared environment, very often down, the “manufacturing line” stops constantly Defect discovery very difficult, 80% of the interfaces are machine interfaces Manual testing rules because automation has failed, labor and time costs are way too high Reality: Major Releases 3+ Times Per Year plus monthly minors, weekly “content” changes and urgent BreakFix Results: costs explode, quality suffers, time to market increases

5 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012  Composite Applications  More Labor Required  Missed Deadlines  Production Failures  Increased Infrastructure Cost Growing Pressures Of Application Development

6 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 Gain transparency and traceability across complex, distributed application architectures Presentation Layer Service Layer Integration Layer Data/ Applications Other UI CORBA J2EE Web ServiceBPM ESB Web Service RMICustom LegacyDatabase File System Packaged App Web 2.0 Identify bottlenecks and defects at every level Message traffic performance through the hardware iTKO is a solution for an IT problem that CA has not previously addressed

7 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 RMI Objects Messaging Service BI Tools Business Rules File System ESB Web Interface Content Database Data Warehouse Financials Mainframe Web/WAP Interface EJB Virtual Service Routing Service SOAP CRM App Products Collaboration App Portal Help Engine BPMS Legacy Infrastructure Internal External Cloud Partners # of Interconnected Components # of Inter-dependent Teams # of Heterogeneous Technologies Rate of Change Today’s Environment

8 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 Root Causes Of Application Development Problems

9 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 SYSTEM UNAVAILABLE ACCESS FEES INCOMPLETE DEVELOPMENT “I can’t do anything until I have everything…and I never have everything!” INVALID DATA Problem: Constraints

10 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, /7 availability Saved $30M Reduced Timeline by 50% Solution: Service Virtualization

11 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012  80% of all new applications are composite and highly integrated  Agile has replaced waterfall development methodologies  Most new application development is self-service and customer-facing making complexity exponential  Customers expect Facebook and Google quality RMI Objects Messaging Service BI Tools Business Rules File System ESB Web Interface Content Database Data Warehouse Financials Mainframe Web/WA P Interface EJB Virtual Service Routing Service SOAP CRM App Products Collaboration App Portal Help Engine BPMS Infrastructure Internal External Cloud Partners Modern Application Development Architecture  70% of the information developers need is outside of their control when they need it  Limited predictability on how it’s going to perform and where it’s going to break  Traditional products and methodologies don’t work in this new architecture

12 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012  iTKO invented, patented and released Service Virtualization in 2007  We clone the behavior of constrained systems  We solved the 70% availability issue  Proven value with 100+ F500 customers  Mature 4th generation product virtualizes everything Service Virtualization RMI Objects Messaging Service BI Tools Business Rules File System ESB Web Interface Content Database Data Warehouse Financials Mainframe Web/WAP Interface EJB Virtual Service Routing Service SOAP CRM App Products Collaboration App Portal Help Engine BPMS Infrastructure Internal External Cloud Partners

13 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012  LISA was built specifically for this architecture to solve the predictability issue  Continuous Validation of performance and usability  Increase collaboration through automation  Applications are more predictable  Fixing problems in production is 10x the cost  Enables enterprise agile RMI Objects Messaging Service BI Tools Business Rules File System ESB Web Interface Content Database Data Warehouse Financials Mainframe Web/WA P Interface EJB Virtual Service Routing Service SOAP CRM App Products Collaboration App Portal Help Engine BPMS Infrastructure Internal External Cloud Partners Built For Modern Architecture

14 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 This Is What LISA Addresses

15 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 “My team of 50 is now a team of 250 in just 5 years due to more components and releases.” Problem: Manual Labor

16 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 Achieve and maintain 85%-95% coverage after 1st test cycle. Solution: Automation

17 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 Challenges  Manual testing incurring delays and costs  No traceability or transparency across tiers  SMEs required to manually validate results Manual UI Testing Manual Validation Before After Automate end-to- end coverage to find bugs “between the boxes” Results with LISA  95% reduction in per-test costs  Regression testing reduced from 18 weeks of labor to 1 day Validate With Complete Automated Testing (National Cable Service Provider)

18 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 “We spend far more time figuring out where the bug is than we do fixing the bug” Problem: Complexity

19 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 Reduced “non-reproducible” defect closure by > 66% DEVELOPERTESTER “Alt-Click!” Solution: Collaboration

20 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 UPSET CUSTOMERS SERVICE DOWN PRODUCTION ISSUES “Too many things go into production without adequate testing” Problem: Change And Unintended Consequences

21 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 Reduced problems delivered to QA to <1% and took 5 months off integration release cycle Solution: Continuous Validation

22 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 LISA Is Helping Reshape Application Development  50-80% reduction in production bugs  20-30% decrease in bug fix cycle UAT Performance Test Integration Test System Test Dev Test Development Environment BuildQC/RQMMonitoring  20-30% reduction in release cycle time  Millions saved in infrastructure costs

23 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 Value Analysis: Ask Yourself… How many requirements are tested manually vs. automated? What % of QA’s reported defects are rejected as "not reproducible“ by Dev? Time spent “proving” an issue that's been rejected as not reproducible? % of resolved cases from Development re-opened by QA as still failing? Avg. Dev & QA time recoding and retesting each unresolved issue? The big one: % of Total Test time in the next release spent retesting what you are testing in the current release? Typical Results: Cost Savings > 25% | Releases > 20% Faster “90% manual” “At least 35%” “3-4 hours each” “25% or more” “4-5 hours avg.” “80%+”

24 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012  Reduced infrastructure cost  Parallel development and testing  Increased quality and performance  Shorter release times Value delivered with LISA

25 Federal Aviation Administration SOA Brown Bag #12: iTKO LISA January 11, 2012 Questions and Comments