1. ® IBM Software Group © 2006 IBM Corporation Agility in the Database: Data Doesnt Have to be a Four-Letter Word Any More Scott W. Ambler Practice Leader Agile Development

2. IBM Software Group | Rational software Scott Ambler - Background Practice Leader Agile Development Fellow – International Association of Software Architects www-306.ibm.com/software/rational/bios/ambler.html

3. IBM Software Group | Rational software Presentation Overview Warning! Critical Observations Adoption Rate Evolutionary/Agile Data Modeling Database Refactoring Database Testing Other Important Techniques Conclusion

4. IBM Software Group | Rational software Warning! Im spectacularly blunt at times Many new ideas will be presented Some may not fit well into your existing environment Some will challenge your existing notions about software development Some will confirm your unvoiced suspicions Dont make any career-ending moves Be skeptical but open minded

5. IBM Software Group | Rational software Critical Observations Modern Development Leading development lifecycles are iterative and incremental New applications typically use a combination of object and data technologies – the world isnt pure There is growing proof that serial/traditional approaches are ineffective Data in only one of many important aspects, and it is rarely the primary one There is growing evidence that craftspeople, or generalizing specialists, are significantly more effective than specialists Traditional IT organizations appear to be vulnerable to offshore outsourcers Cultural There is a serious rift between the data and development communities Each community is relatively ignorant of what the other one is doing There are few books within either community that do the other justice The agile movement is coming from the development community Many developers are prepared to ignore data professionals, and do so quite often

6. IBM Software Group | Rational software 66% of Development Teams Go Around Data Groups: Why? Source: www.ambysoft.com/surveys/

7. IBM Software Group | Rational software Adoption Rate of Agile Database Techniques

8. ® IBM Software Group © 2006 IBM Corporation Evolutionary/Agile Data Modeling What is Evolutionary/Agile Data Modeling An Example Why This Works

9. IBM Software Group | Rational software Agile Data Modeling Data modeling is the act of exploring data-oriented structures. Evolutionary data modeling is data modeling performed in an iterative and incremental manner. Agile data modeling is evolutionary data modeling done in a collaborative manner. Data modeling is only one of many aspects of modeling, see www.agilemodeling.com/artifacts www.agilemodeling.com/artifacts See Agile Model Driven Development (AMDD) www.agilemodeling.com/essays/amdd.htm www.agilemodeling.com/essays/amdd.htm

10. IBM Software Group | Rational software Karate School Management System Example Based on an actual organization Did the same problem two ways: Without an initial conceptual model www.ddj.com – A three-issue series With an initial conceptual model www.agiledata.org/essays/agileDataModeling.html

11. IBM Software Group | Rational software The Initial Requirements (A Karate School Management System) IterationUser Stories 1 Maintain student contact information Enroll student Drop student Record payment 2 Promote student to higher belt Invite student to grading Email membership to student Print membership for student 3 Schedule gradings Print certificate Put membership on hold 4 Maintain product information Sell product 5 Print catalog of products Order product for inventory Order product for student 6 Organize tournament Enroll participant in tournament Send out tournament announcement email to past participants Print tournament announcement letters to past participants

12. IBM Software Group | Rational software Initial Domain Model

13. IBM Software Group | Rational software Iteration 1 PDM

14. IBM Software Group | Rational software Iteration 2 PDM

15. IBM Software Group | Rational software Iteration 3 PDM

16. IBM Software Group | Rational software Updated Requirements Iter.User Stories 4Enroll child student Offer family membership plan Support child belt system 5Enroll student in Tai Chi Support Tai Chi belt system Enroll student in cardio kick boxing Support Tai Chi belt system Support the belt order for each style 6Maintain product information Sell products 7Print catalog of products Order product for inventory Order product for student 8Organize internal special event (special classes, internal tournaments, …) Enroll student in special event Print special event certificate for student

17. IBM Software Group | Rational software Updated Domain Model

18. IBM Software Group | Rational software Iteration 4 PDM

19. IBM Software Group | Rational software Iteration 5 PDM

20. IBM Software Group | Rational software Iteration 6 PDM

21. IBM Software Group | Rational software Why This Works Up front, high-level modeling defines the landscape JIT model storming enables: You to ask more intelligent questions because you know more about the domain the longer you work in it Stakeholders to answer more intelligently because they understand the solution space better Focus on functionality that is actually needed, not what might be needed

22. ® IBM Software Group © 2006 IBM Corporation Refactoring Databases Database Refactoring Why its Hard The Process Examples Why This Works

23. IBM Software Group | Rational software Uncomfortable Question How long would it take you to safely rename a column in a production database table? Could you do it in a single day?

24. IBM Software Group | Rational software Survey: How Long to Rename a Column in a Production Database? Source: www.ambysoft.com/surveys/

25. IBM Software Group | Rational software Length of Time to Rename Production Column (% Respondents)

26. IBM Software Group | Rational software 61.9% of Respondents Have Production Data Problems: Their Strategy to Address Data Source Problems Source: www.ambysoft.com/surveys/

27. IBM Software Group | Rational software Database Refactoring A database refactoring is a simple change to a database schema that improves its design while retaining both its behavioral and informational semantics. A database schema includes both structural aspects such as table and view definitions as well as functional aspects such as stored procedures and triggers. Important: Database refactorings are a subset of schema transformations, but they do not add functionality. www.agiledata.org/essays/databaseRefactoring.html

28. IBM Software Group | Rational software Why DB Refactoring is Hard

29. IBM Software Group | Rational software The Process of Database Refactoring

30. IBM Software Group | Rational software Rename Column

31. IBM Software Group | Rational software Example: Replace Column

32. IBM Software Group | Rational software Example: Remove View

33. IBM Software Group | Rational software Example: Merge Columns

34. IBM Software Group | Rational software Why This Works The database schema is evolved in very small steps in a low-risk manner Each refactoring is simple to implement It enables an evolutionary approach while supporting the existing infrastructure

35. ® IBM Software Group © 2006 IBM Corporation Database Regression Testing Possible Things to Test Current Database Testing

36. IBM Software Group | Rational software Current State of Database Regression Testing (July 2006) 95.7% of organizations considered data to be a corporate asset Of those, only 40.3% had a regression test suite in place Of those, Only 63.3% allowed developers to run this test suite when they needed to Of the organizations that didnt have a test suite, only 31.6% had discussed putting one in place 63.7% of organizations implemented mission-critical functionality in the database Of those, only 46% had a regression test suite in place Of those, Only 66.3% allowed developers to run this test suite when they needed to Of the organizations that didnt have a test suite, only 38.6% had discussed putting one in place

37. IBM Software Group | Rational software Database Regression Testing Agilists develop regression unit and acceptance test suites for their applications, why not for databases too? You must test: Within the database (clear box) At the interface to the database (black box) You must be able to put the database into a known state, therefore you need test data (generation)

38. IBM Software Group | Rational software Things to Test For

39. IBM Software Group | Rational software Do You Do Database Testing? (September 2006)

40. IBM Software Group | Rational software When Are You Testing the Database? (% Respondents)

41. IBM Software Group | Rational software Types of Database Testing (% Respondents)

42. IBM Software Group | Rational software Other Important Techniques Configuration management of all project artifacts Continuous integration Encapsulate database access Train developers in data-oriented skills Train data professionals in development skills Promote the concept of generalizing specialists www.agilemodeling.com/essays/generalizingSpecialists.htm

43. IBM Software Group | Rational software Conclusion Evolutionary development is the norm, not the exception Agile development techniques are quickly being adopted There is no reason why data professionals cannot work in an evolutionary, or even agile, manner The software world has changed. Have you? Visit www.agiledata.org

44. ® IBM Software Group © 2006 IBM Corporation Keep In Touch! Scott W. Ambler www-306.ibm.com/software/rational/bios/ambler.html

45. IBM Software Group | Rational software References and Recommended Reading www.agilealliance.com www.agilemodeling.com www.agiledata.org www.ambysoft.com www.databaserefactoring.com www.enterpriseunifiedprocess.com Ambler, S.W. (2002). Agile Modeling: Effective Practices for XP and the UP. New York: John Wiley & Sons. Ambler, S.W. (2003). Agile Database Techniques. New York: John Wiley & Sons. Ambler, S.W. (2004). The Object Primer 3 rd Edition: AMDD with UML 2. New York: Cambridge University Press. Ambler, S.W. and Sadalage, P.J. (2006). Refactoring Databases: Evolutionary Database Design. Reading, MA: Addison Wesley Longman, Inc. Larman, C. (2004). Agile and Iterative Development: A Managers Guide. Reading, MA: Addison Wesley McGovern, J., Ambler, S.W., Stevens, M., Linn, J., Sharan, V., & Jo, E. (2003). The Practical Guide to Enterprise Architecture. Prentice Hall PTR.