1. Microsoft Australia Security Summit Tools for Quality Code Nigel Watson, Microsoft Australia Sean Salisbury, Compuware Corp Nigel Watson, Microsoft Australia Sean Salisbury, Compuware Corp

2. Microsoft Australia Security Summit Agenda Testing – so what? Testing in Visual Studio Team System Extending VSTS – Compuware DevPartner Summary Testing – so what? Testing in Visual Studio Team System Extending VSTS – Compuware DevPartner Summary

3. Microsoft Australia Security Summit Projects and Testing Often an expensive afterthought Strategies for minimising impact Often an expensive afterthought Strategies for minimising impact Requirements Coding Integration Beta Test Post-Release 5 10 15 20 25 30 Relative Cost To Fix Bugs...

4. Microsoft Australia Security Summit Problems... It is expensive to find and fix bugs that get past daily development practices Potential security flaws need to be caught early It is hard to diagnose errors at runtime Why does an application run slowly? Individual Developers and Testers need to know if they are on track Test and development are often out of synch Final test phase for shipping is often ad-hoc How much testing is enough? It is expensive to find and fix bugs that get past daily development practices Potential security flaws need to be caught early It is hard to diagnose errors at runtime Why does an application run slowly? Individual Developers and Testers need to know if they are on track Test and development are often out of synch Final test phase for shipping is often ad-hoc How much testing is enough?

5. Microsoft Australia Security Summit Defense In Depth Microsoft uses a 'defense in depth' strategy Unit testing Code reviews Frequent builds Catch bugs early Static checks Runtime checks Microsoft uses a 'defense in depth' strategy Unit testing Code reviews Frequent builds Catch bugs early Static checks Runtime checks

6. Microsoft Australia Security Summit Testing in VSTS Change Management Work Item Tracking Reporting Project Site Visual Studio Team Foundation Project Management Visual Studio Team Architect Visio and UML Modeling VS Pro Class Modeling Application Modeling Logical Infra. Modeling Deployment Modeling Visual Studio Team Developer Visual Studio Team Test Project Site Work Item Tracking Reporting Project Management Integration Services Load Testing Manual Testing Test Case Management Unit Testing Code Coverage Dynamic Code Analyzer Static Code Analyzer Code Profiler Team Foundation Client

7. Microsoft Australia Security Summit Testing in VSTS Change Management Work Item Tracking Reporting Project Site Visual Studio Team Foundation Project Management Visual Studio Team Architect Visio and UML Modeling VS Pro Class Modeling Application Modeling Logical Infra. Modeling Deployment Modeling Visual Studio Team Developer Visual Studio Team Test Project Site Work Item Tracking Reporting Project Management Integration Services Load Testing Manual Testing Test Case Management Unit Testing Code Coverage Dynamic Code Analyzer Static Code Analyzer Code Profiler Team Foundation Client

8. Microsoft Australia Security Summit Test-Driven Development Integrate testing into the development process Tests define what code will do Tests come from specifications Write code to pass tests Don't write code that doesn't contribute to passing a test... Integrate testing into the development process Tests define what code will do Tests come from specifications Write code to pass tests Don't write code that doesn't contribute to passing a test... Code

9. Microsoft Australia Security Summit VSTS Unit Testing Integrated into VS Automatic generation of test classes Comprehensive test management Code coverage testing Integrated into VS Automatic generation of test classes Comprehensive test management Code coverage testing [TestMethod()] public void GetValueTest() { double d = myObject.getValue(); double d = myObject.getValue(); if (d < 10.0) if (d < 10.0) Assert.Fail("Bad return value"); Assert.Fail("Bad return value"); }

10. Microsoft Australia Security Summit Unit Testing

11. Microsoft Australia Security Summit Code Reviews For the Visual Studio 7.0 product cycle 86% of bugs occurred in reviewed code 60% of all bugs were coding errors Static analysis helps catch bugs Source code analysis PREfast for C and C++ FxCop for.NET For the Visual Studio 7.0 product cycle 86% of bugs occurred in reviewed code 60% of all bugs were coding errors Static analysis helps catch bugs Source code analysis PREfast for C and C++ FxCop for.NET

12. Microsoft Australia Security Summit PREFast Static analysis for C/C++ code Managed and unmanaged C++ Catches common bugs Buffer overruns, uninitialized memory Memory leaks, null pointer dereference Reported as compiler warnings Display path to problem Use #pragma to turn off Static analysis for C/C++ code Managed and unmanaged C++ Catches common bugs Buffer overruns, uninitialized memory Memory leaks, null pointer dereference Reported as compiler warnings Display path to problem Use #pragma to turn off

13. Microsoft Australia Security Summit FxCop Static analysis for.NET assemblies Not just C++ Uses design guidelines (including many in the.NET Class Design Guidelines) Customizable Which checks to include Whether to report as error or warning Create custom rules Static analysis for.NET assemblies Not just C++ Uses design guidelines (including many in the.NET Class Design Guidelines) Customizable Which checks to include Whether to report as error or warning Create custom rules

14. Microsoft Australia Security Summit Static code analysis

15. Microsoft Australia Security Summit Integrating Dev and Test Tests are just another form of source code: Stored in source code control Versioned with the product “Test Complete” Test writing is scheduled along with development work Tracked by work items Testers are notified when bugs are fixed Tests are just another form of source code: Stored in source code control Versioned with the product “Test Complete” Test writing is scheduled along with development work Tracked by work items Testers are notified when bugs are fixed

16. Microsoft Australia Security Summit VSTS Test Types Unit Tests Test class methods Web Tests Record and playback interactions Load Tests Simulate multiple users Manual Tests Provide scripts for manual tasks Third-party Tests Integrated into VSTS Unit Tests Test class methods Web Tests Record and playback interactions Load Tests Simulate multiple users Manual Tests Provide scripts for manual tasks Third-party Tests Integrated into VSTS

17. Microsoft Australia Security Summit Application Quality Best Practices and Tools Sean Salisbury Senior Regional Tech Specialist Compuware Corporation sean.salisbury@compuware.com

18. Microsoft Australia Security Summit Compuware Corporation 30 years of delivering proven IT value to the world’s largest businesses 8,500+ employees 84 offices in 16 countries Rock-solid financially $1 billion+ in revenue $700 million+ in cash Our commitment: Real VALUE and ROI for every single dollar you spend with us. 30 years of delivering proven IT value to the world’s largest businesses 8,500+ employees 84 offices in 16 countries Rock-solid financially $1 billion+ in revenue $700 million+ in cash Our commitment: Real VALUE and ROI for every single dollar you spend with us.

19. Microsoft Australia Security Summit Development & Integration Automated Software Quality Production Readiness Performance & Availability Management Application Life Cycle

20. Microsoft Australia Security Summit l l Integrated development and test automation tools l l Rich process management l l Detailed and relevant project information Microsoft and Compuware Production Readiness Automated Software Quality Development & Integration Performance & Availability Management QACenter Extends quality assurance testing DevPartner Extends quality in development VS & Team System integration platform, base tools

21. Microsoft Australia Security Summit Quality Begins in Development Addressing application quality needs including security during development is most cost-effective Profile performance and optimize memory utilization Carry out in-depth unit testing Monitor code coverage Seamlessly capture defect information Addressing application quality needs including security during development is most cost-effective Profile performance and optimize memory utilization Carry out in-depth unit testing Monitor code coverage Seamlessly capture defect information

22. Microsoft Australia Security Summit Compuware DevPartner Studio enhance and extend Visual Studio Native and Managed Code Analysis Local and Remote Data Collection: Performance Analysis.NET Memory Analysis Code Coverage Analysis Distributed Application Analysis VB, VB.NET, ASP.Net and C# Source Code Review with >600 Rules C/C++ Memory Error & Thread Deadlock Detection Native and Managed Code Analysis Local and Remote Data Collection: Performance Analysis.NET Memory Analysis Code Coverage Analysis Distributed Application Analysis VB, VB.NET, ASP.Net and C# Source Code Review with >600 Rules C/C++ Memory Error & Thread Deadlock Detection

23. Microsoft Australia Security Summit Example: Profiling

24. Microsoft Australia Security Summit Code Analysis 600+ Rules enhance problem resolution Supports VS6/2002/2003/2005 Accelerates learning curves Improves code quality and maintainability Supports Visual Basic, VB.NET, C#, ASP.Net 600+ Rules enhance problem resolution Supports VS6/2002/2003/2005 Accelerates learning curves Improves code quality and maintainability Supports Visual Basic, VB.NET, C#, ASP.Net

25. Microsoft Australia Security Summit

26. Memory Analysis Optimize Local or Remote Memory Use View allocations/deallocations over time: get an overall feel for memory use Identify Objects That: Consume a lot of memory Create a lot of temporary objects Stay around longer than they need to, including leaks Compare Runs- Did Code Changes Help? Tune Garbage Collection Optimize Local or Remote Memory Use View allocations/deallocations over time: get an overall feel for memory use Identify Objects That: Consume a lot of memory Create a lot of temporary objects Stay around longer than they need to, including leaks Compare Runs- Did Code Changes Help? Tune Garbage Collection

27. Microsoft Australia Security Summit Memory Analysis at Run Time Real-Time trace of memory usage System Allocations Your Code RAM usage Time

28. Microsoft Australia Security Summit Memory Analysis Many Different Data Views with Details Available

29. Microsoft Australia Security Summit Automatic Error Detection Memory/Resource/ Interface Leaks API Errors Threading Issues Event Debugging C/C++/VC++ Memory/Resource/ Interface Leaks API Errors Threading Issues Event Debugging C/C++/VC++

30. Microsoft Australia Security Summit

31. Thread Deadlock Detection Locate Actual or Potential Thread Deadlocks or Other Synchronization Issues Deadlock: 2 or more code paths running at the same time, contending for the same resource(s) Benefits Thread deadlock are difficult to detect: automating detection is very useful Locate Actual or Potential Thread Deadlocks or Other Synchronization Issues Deadlock: 2 or more code paths running at the same time, contending for the same resource(s) Benefits Thread deadlock are difficult to detect: automating detection is very useful

32. Microsoft Australia Security Summit Performance Profiling Pinpoint bottlenecks across app Tiers/Versions Optimize application performance Increase usability Pinpoint bottlenecks across app Tiers/Versions Optimize application performance Increase usability

33. Microsoft Australia Security Summit

34. Compare Performance Runs

35. Microsoft Australia Security Summit Code Coverage Quickly identify untested code across tiers & VS6/02/03/05 Ensure test coverage during unit testing More reliable components and applications Quickly identify untested code across tiers & VS6/02/03/05 Ensure test coverage during unit testing More reliable components and applications

36. Microsoft Australia Security Summit

37. Distributed Analysis

38. Microsoft Australia Security Summit What’s New….

39. Microsoft Australia Security Summit IT Challenges Identifying what errors can occur & when Tools lacking for error simulation and analysis Errors corrupt the debugging environment Impossible to trace error handling execution Difficult to create repeatable tests Time-consuming, manual process Identifying what errors can occur & when Tools lacking for error simulation and analysis Errors corrupt the debugging environment Impossible to trace error handling execution Difficult to create repeatable tests Time-consuming, manual process

40. Microsoft Australia Security Summit Quality of Error Handlers Applications are comprised of between 10% and 30% error handling code It’s difficult/impossible to safely and repeatably simulate errors to test error- handling code The result: Overall code coverage during test is lower than it could be Undetected bugs are passed into the production environment Applications are comprised of between 10% and 30% error handling code It’s difficult/impossible to safely and repeatably simulate errors to test error- handling code The result: Overall code coverage during test is lower than it could be Undetected bugs are passed into the production environment

41. Microsoft Australia Security Summit What If You Could… Quickly determine what errors could occur at any point in your application? Ensure you have error handlers in place to cope Simulate errors safely and efficiently? With no impact on the OS,.NET framework or any other running application Observe and debug your error handlers Build reusable fault test libraries? Create repeatable tests that are reusable by development & QA Quickly determine what errors could occur at any point in your application? Ensure you have error handlers in place to cope Simulate errors safely and efficiently? With no impact on the OS,.NET framework or any other running application Observe and debug your error handlers Build reusable fault test libraries? Create repeatable tests that are reusable by development & QA

42. Microsoft Australia Security Summit DevPartner Fault Simulator Developer Insight What errors can occur at what point in the code Integrated with Visual Studio debugging features to monitor error handling execution Break at fault occurrence Developer Insight What errors can occur at what point in the code Integrated with Visual Studio debugging features to monitor error handling execution Break at fault occurrence

43. Microsoft Australia Security Summit DevPartner Fault Simulator Error handling validation Simulate Environmental and.NET Framework faults Simple method of selection of errors to validate, with user defined conditions Reusable Fault Sets for repeat and QA testing VS 2003/05 IDE integrated, standalone and command line operation Error handling validation Simulate Environmental and.NET Framework faults Simple method of selection of errors to validate, with user defined conditions Reusable Fault Sets for repeat and QA testing VS 2003/05 IDE integrated, standalone and command line operation

44. Microsoft Australia Security Summit DevPartner Fault Simulator Results analysis Simulate Stack tracing & error details “Go to source” linking for detailed analysis Live view and summary of fault execution Saved Results files for later review Results analysis Simulate Stack tracing & error details “Go to source” linking for detailed analysis Live view and summary of fault execution Saved Results files for later review

45. Microsoft Australia Security Summit DevPartner Fault Simulator Demonstration

46. Microsoft Australia Security Summit Securing ASP.Net Applications

47. Microsoft Australia Security Summit Security Vulnerability “Today over 70% of attacks against a company’s network come at the Application Layer, not the Network or System Layer” John Pescatore, Gartner chief security analyst The responsibility for application security is shifting to the development organization How do they address this aspect of application quality? How do they gain the skills they need to assess and correct security vulnerabilities? “Today over 70% of attacks against a company’s network come at the Application Layer, not the Network or System Layer” John Pescatore, Gartner chief security analyst The responsibility for application security is shifting to the development organization How do they address this aspect of application quality? How do they gain the skills they need to assess and correct security vulnerabilities?

48. Microsoft Australia Security Summit What If You Could… Quickly locate security vulnerabilities in your application during development? Minimize the cost and mean-time-to-repair Improve the quality/reliability of your application Have a wealth of security expertise and advice at your fingertips? Have the information you need, when you need it Quickly locate security vulnerabilities in your application during development? Minimize the cost and mean-time-to-repair Improve the quality/reliability of your application Have a wealth of security expertise and advice at your fingertips? Have the information you need, when you need it

49. Microsoft Australia Security Summit DevPartner SecurityChecker A vulnerability assessment scanner that locates security vulnerabilities in ASP.NET (C# or VB.NET) Locates complex & hard-to-find security problems Organizes results by priority and category Pinpoints vulnerabilities to the line of source code Explains why it is an issue Suggests steps to repair each vulnerability Provides links to additional technical information A vulnerability assessment scanner that locates security vulnerabilities in ASP.NET (C# or VB.NET) Locates complex & hard-to-find security problems Organizes results by priority and category Pinpoints vulnerabilities to the line of source code Explains why it is an issue Suggests steps to repair each vulnerability Provides links to additional technical information

50. Microsoft Australia Security Summit DevPartner SecurityChecker Integrity Analysis (attach simulation) Replays a series of known security attacks against the application Secures the interface to the application Compile-time Analysis Scans source code for known security problems Test while coding Run-time Analysis Monitors execution of the application Observes interior/hidden facets, beyond the external interface Expert Advisor Go to line of source code Detailed assistance Allows the developer to quickly: Find & fix the vulnerability Become more knowledgeable about security Accelerates secure application development

51. Microsoft Australia Security Summit Security Issues by Analyzer Integrity Analyzer Compile-time Analyzer Run-time Analyzer Security Context√ √ Insecure Coding Practices √√√ Execution Errors√√ Application Integrity√√ Deployment Issues√√

52. Microsoft Australia Security Summit DevPartner SecurityChecker Demonstration

53. Microsoft Australia Security Summit Quality Continues in Testing Automate functional testing and validation Manage test plans and execution Comparison of complex data results Seamlessly capture defect information Simulate application under load Simulate load conditions ‘000,000’s of users Determine application scalability Compuware QACenter Enterprise Wide Compuware Vantage - Network and Server monitoring Automate functional testing and validation Manage test plans and execution Comparison of complex data results Seamlessly capture defect information Simulate application under load Simulate load conditions ‘000,000’s of users Determine application scalability Compuware QACenter Enterprise Wide Compuware Vantage - Network and Server monitoring

54. Microsoft Australia Security Summit Microsoft & Compuware Tools to: Improve application reliability & performance Increase team productivity Lower costs Deliver better applications to the market faster Production Readiness Automated Software Quality Development & Integration Performance & Availability Management

55. Microsoft Australia Security Summit Summary Appreciated the importance of testing to the development process Had a quick look at some of the testing tools in Visual Studio Team System Sean showed us how Compuware DevPartner Studio uses the integration capabilities of Visual Studio to extend the power of the IDE Appreciated the importance of testing to the development process Had a quick look at some of the testing tools in Visual Studio Team System Sean showed us how Compuware DevPartner Studio uses the integration capabilities of Visual Studio to extend the power of the IDE

56. Microsoft Australia Security Summit © 2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.