1. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 1 No Respect Danny, Be the Ball

2. NA-MIC National Alliance for Medical Image Computing http://na-mic.org Dart2 Quality Framework Software Quality: Past, Present, Future Dan Blezek Jim Miller Bill Lorensen

3. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 3 A brief history of quality Very First VTK Dashboard Update information Builds – Irix, Solaris, WinNT Regression tests WinNT “Catastropic failure”

4. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 4 Last VTK Dashboard 8 Platforms 650 Nightly Tests 70% coverage (Still) Nightly Purify

5. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 5 A Bad Day

6. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 6 The Continuous Build

7. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 7 My first VTK checkin Continuous Update Continuous Build

8. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 8 The Big Bat of Quality

9. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 9 Bill “Yogi” Lorensen

10. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 10 Lessons learned Test on different platforms Test nightly Make it easy to add a test Track changes daily Keep historical information

11. NA-MIC National Alliance for Medical Image Computing http://na-mic.org Two roads diverged in a wood, and I-- I took the one less traveled by, And that has made all the difference. Robert Frost Closed Source

12. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 12 Dart (v1) Tests Reports Dashboards Captures state of the system Distills data into information –Convert build log  errors/warnings –Summarize test execution –Rollup Coverage statistics

13. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 13

14. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 14

15. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 15 Someone broke the build!

16. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 16 Someone broke the build!

17. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 17

18. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 18 Dart’s Power Distributed testing If I don’t have a platform, you do Distill data from many tools Distributed Extreme Programming Know the state of the system Instant feedback on changes

19. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 19 Quality Statistics Original VTK dashboard 8 platforms / 650 tests 13.6 G over 4 years Current VTK dashboard 29 Nightly platforms / 500 tests 1-2 G / week Insight dashboard 60 Nightly builds / 970 tests 1-2 G / week

20. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 20 Dart2 Design Goals One Server, multiple Projects Simple, flexible setup and management Configurable presentation Persist data on dashboard over time Aggregate Dashboards Authenticated submission, if desired Resource management tools

21. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 21 Implementation Java –Many, many available packages –Cross-platform Everything in one package –No extra OS packages required –Distribute as Jar and/or platform exe Should be easily extensible –Even to non-Java programmers

22. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 22 Packages

23. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 23 Components/Concepts Client, Submission Test Hierarchy Results RDBMS XML-RPC Server Task Manager Scheduler Tracks HTTP / Template Engine

24. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 24 Client, Submission Client: a unique platform –Need to define criteria –Currently Site / BuildName Submission –One TimeStamped set of Test data –Particular to a sub-Project slicer.itk slicer.vtk

25. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 25 Test Hierarchy Test is a logic group of Results Has a Pass/Fail/NotRun status May contain other Tests Has Hierarchial naming convention –itk.common.PrintSelf SubTest information rolled up

26. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 26 Results Data produced by a Test Examples: –Log of standard out –Image –ExecutionTime Typed –text/string, text/url, text/xml, text/text –numeric/integer, numeric/double –image/png,image/jpeg

27. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 27 RDBMS Core of Dart2 –Bundled with Derby embedded RDBMS –Any JDBC compliant DB works Stores “small” data –Images, large blocks of text in files Jaxor –Object – Relational Bridge package –No fancy SQL required –Creates objects from rows in DB

28. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 28 XML-RPC Server Accepts Submissions Administrative functions HTTP transport –Easy submission through firewalls Digester used to process XML –Executes code when tags found

29. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 29 Task Manager Tasks are units of work for the server –Project and Server Tasks –Scheduled, Event driven When a Submission arrives, a Task is queued QueueManager executes Tasks Plug-ins allow Project specific Tasks –Simply implement the Task Interface

30. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 30 Scheduler Quartz Enterprise Scheduler Executes Tasks –Uses enhanced “cron” syntax Uses –Regular DB maintained –Purge unnecessary data –Archive aging data

31. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 31 Tracks Groups of Submissions –Dashboard consists of intersecting Tracks Temporal Tracks –Time based, i.e. 12am start, 24hr duration “Most Recent” Track –Last 5 Continuous builds Project specific

32. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 32 HTTP / Template Engine Jetty is HTTP/Servlet server FreeMarker –Data prepared in Servelet –Template processed –Returned to client via HTTP Flexible Easy to add new pages No XSLT!

33. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 33 Dart2 Current Status Alpha version ready Test server –http://www.na-mic.org:8081/Insight/Dashboard/http://www.na-mic.org:8081/Insight/Dashboard/ –Populated with Build & Test from public.kitware.com Subversion Code Repository –svn co http://svn.na-mic.org:8000/svn/Dart –Web SVN: http://www.na-mic.org:8000/websvn/http://www.na-mic.org:8000/websvn/

34. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 34 Acknowledgements Andy Cedilnik Bill Hoffman Will Schroeder Ken Martin Amitha Perera Fred Wheeler

35. NA-MIC National Alliance for Medical Image Computing http://na-mic.org Dart2 Quality Framework Software Quality: Past, Present, Future Dan Blezek Jim Miller Bill Lorensen

36. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 36 Desirable Qualities Frequent testing –Identify defects as soon as they are introduced –Hard to find cause if not done frequently Minimally invasive to daily activities Automated testing Automated report generation/summaries –Must be concise yet informative Track results over time

37. NA-MIC National Alliance for Medical Image Computing http://na-mic.org NAMIC Software Process Dan Blezek Jim Miller Bill Lorensen

38. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 38 Motivation Many algorithms, many platforms –VTK, ITK, Slicer, LONI –Linux, Windows, Mac OSX, Solaris, SGI Many users Many datasets Many sources of problems!

39. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 39 Motivation Negative example MIT codes ITK algorithm for LONI pipeline UCLA developer changes LONI GE changes ITK Time for release, everything’s broken

40. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 40 Motivation Ensuring high quality software System’s state must be known –If UCLA knew about MIT code, they would have been more careful w/changes All the code works, all the time –As often as is feasible, compile and test the code

41. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 41 Extreme Programming

42. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 42 NAMIC Process Light weight Based on Extreme Programming –High intensity cycle Design Test Implement –Supported with web-enabled tools –Automated testing integrated with the software development

43. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 43 Software Process Design Process Coding Standards Testing Bug Tracker Communication –Mailing lists, Discussion forum, Wiki –Tcons Documentation Releases

44. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 44 Design Process Take the time to design a good API Plan for future use Plan for future extension Two routes –Code something, check it in Others will tear it down & make it better –Put together a strawman Solicit ideas, implement

45. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 45 Coding Standards Follow the package’s rules ITK has certain coding standards –Style guidelines –Naming conventions –Macros

46. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 46 Testing If it isn’t tested, it’s broken. Tests –Ensure your code works –Documents expected results –Others free to change

47. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 47 Bug Tracker Bugs assigned / taken by developers Tracks progress to releases Captures feature requests Communication mechanism

48. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 48 Documentation Doxygen –Automatic API documentation –Algorithm references –Implementation details Books / Manuals –Insight Book

49. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 49 Communication Email lists Discussion forum Wiki Tcon

50. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 50 Extreme Programming Compression of standard analyze, design, implement, test cycle into a continuous process

51. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 51 Daily Testing Is The Key Testing anchors the development process (Dart) Developers monitor the testing dashboard constantly Problems are identified and fixed immediately Developers receive e-mail if they “Break the Build”

52. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 52 Daily rhythm Design, implement algorithm write regression test check it in Dart takes over Code is updated, built, tested Guilty parties are blamed Core 2 Build breakers are notified Work on cleaning up dashboard Repeat

53. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 53 Dart Testing Reports Dashboards Central site for state of the system –Updates –Builds –Test –Coverage

54. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 54

55. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 55

56. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 56 Someone broke the build!

57. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 57

58. National Alliance for Medical Image Computing http://na-mic.org Salt Lake City AHM February, 2005 58 Conclusion Have fun Process extends your impact –Many can use your code –Many can improve your code Communicate, Communicate, Communicate

59. NA-MIC National Alliance for Medical Image Computing http://na-mic.org NAMIC Software Process Dan Blezek Jim Miller Bill Lorensen