National Alliance for Medical Image Computing Salt Lake City AHM February, No Respect Danny, Be the Ball

NA-MIC National Alliance for Medical Image Computing Dart2 Quality Framework Software Quality: Past, Present, Future Dan Blezek Jim Miller Bill Lorensen

National Alliance for Medical Image Computing Salt Lake City AHM February, A brief history of quality Very First VTK Dashboard Update information Builds – Irix, Solaris, WinNT Regression tests WinNT “Catastropic failure”

National Alliance for Medical Image Computing Salt Lake City AHM February, Last VTK Dashboard 8 Platforms 650 Nightly Tests 70% coverage (Still) Nightly Purify

National Alliance for Medical Image Computing Salt Lake City AHM February, A Bad Day

National Alliance for Medical Image Computing Salt Lake City AHM February, The Continuous Build

National Alliance for Medical Image Computing Salt Lake City AHM February, My first VTK checkin Continuous Update Continuous Build

National Alliance for Medical Image Computing Salt Lake City AHM February, The Big Bat of Quality

National Alliance for Medical Image Computing Salt Lake City AHM February, Bill “Yogi” Lorensen

National Alliance for Medical Image Computing Salt Lake City AHM February, Lessons learned Test on different platforms Test nightly Make it easy to add a test Track changes daily Keep historical information

NA-MIC National Alliance for Medical Image Computing Two roads diverged in a wood, and I-- I took the one less traveled by, And that has made all the difference. Robert Frost Closed Source

National Alliance for Medical Image Computing Salt Lake City AHM February, Dart (v1) Tests Reports Dashboards Captures state of the system Distills data into information –Convert build log  errors/warnings –Summarize test execution –Rollup Coverage statistics

National Alliance for Medical Image Computing Salt Lake City AHM February,

National Alliance for Medical Image Computing Salt Lake City AHM February,

National Alliance for Medical Image Computing Salt Lake City AHM February, Someone broke the build!

National Alliance for Medical Image Computing Salt Lake City AHM February, Someone broke the build!

National Alliance for Medical Image Computing Salt Lake City AHM February,

National Alliance for Medical Image Computing Salt Lake City AHM February, Dart’s Power Distributed testing If I don’t have a platform, you do Distill data from many tools Distributed Extreme Programming Know the state of the system Instant feedback on changes

National Alliance for Medical Image Computing Salt Lake City AHM February, Quality Statistics Original VTK dashboard 8 platforms / 650 tests 13.6 G over 4 years Current VTK dashboard 29 Nightly platforms / 500 tests 1-2 G / week Insight dashboard 60 Nightly builds / 970 tests 1-2 G / week

National Alliance for Medical Image Computing Salt Lake City AHM February, Dart2 Design Goals One Server, multiple Projects Simple, flexible setup and management Configurable presentation Persist data on dashboard over time Aggregate Dashboards Authenticated submission, if desired Resource management tools

National Alliance for Medical Image Computing Salt Lake City AHM February, Implementation Java –Many, many available packages –Cross-platform Everything in one package –No extra OS packages required –Distribute as Jar and/or platform exe Should be easily extensible –Even to non-Java programmers

National Alliance for Medical Image Computing Salt Lake City AHM February, Packages

National Alliance for Medical Image Computing Salt Lake City AHM February, Components/Concepts Client, Submission Test Hierarchy Results RDBMS XML-RPC Server Task Manager Scheduler Tracks HTTP / Template Engine

National Alliance for Medical Image Computing Salt Lake City AHM February, Client, Submission Client: a unique platform –Need to define criteria –Currently Site / BuildName Submission –One TimeStamped set of Test data –Particular to a sub-Project slicer.itk slicer.vtk

National Alliance for Medical Image Computing Salt Lake City AHM February, Test Hierarchy Test is a logic group of Results Has a Pass/Fail/NotRun status May contain other Tests Has Hierarchial naming convention –itk.common.PrintSelf SubTest information rolled up

National Alliance for Medical Image Computing Salt Lake City AHM February, Results Data produced by a Test Examples: –Log of standard out –Image –ExecutionTime Typed –text/string, text/url, text/xml, text/text –numeric/integer, numeric/double –image/png,image/jpeg

National Alliance for Medical Image Computing Salt Lake City AHM February, RDBMS Core of Dart2 –Bundled with Derby embedded RDBMS –Any JDBC compliant DB works Stores “small” data –Images, large blocks of text in files Jaxor –Object – Relational Bridge package –No fancy SQL required –Creates objects from rows in DB

National Alliance for Medical Image Computing Salt Lake City AHM February, XML-RPC Server Accepts Submissions Administrative functions HTTP transport –Easy submission through firewalls Digester used to process XML –Executes code when tags found

National Alliance for Medical Image Computing Salt Lake City AHM February, Task Manager Tasks are units of work for the server –Project and Server Tasks –Scheduled, Event driven When a Submission arrives, a Task is queued QueueManager executes Tasks Plug-ins allow Project specific Tasks –Simply implement the Task Interface

National Alliance for Medical Image Computing Salt Lake City AHM February, Scheduler Quartz Enterprise Scheduler Executes Tasks –Uses enhanced “cron” syntax Uses –Regular DB maintained –Purge unnecessary data –Archive aging data

National Alliance for Medical Image Computing Salt Lake City AHM February, Tracks Groups of Submissions –Dashboard consists of intersecting Tracks Temporal Tracks –Time based, i.e. 12am start, 24hr duration “Most Recent” Track –Last 5 Continuous builds Project specific

National Alliance for Medical Image Computing Salt Lake City AHM February, HTTP / Template Engine Jetty is HTTP/Servlet server FreeMarker –Data prepared in Servelet –Template processed –Returned to client via HTTP Flexible Easy to add new pages No XSLT!

National Alliance for Medical Image Computing Salt Lake City AHM February, Dart2 Current Status Alpha version ready Test server – –Populated with Build & Test from public.kitware.com Subversion Code Repository –svn co –Web SVN:

National Alliance for Medical Image Computing Salt Lake City AHM February, Acknowledgements Andy Cedilnik Bill Hoffman Will Schroeder Ken Martin Amitha Perera Fred Wheeler

NA-MIC National Alliance for Medical Image Computing Dart2 Quality Framework Software Quality: Past, Present, Future Dan Blezek Jim Miller Bill Lorensen

National Alliance for Medical Image Computing Salt Lake City AHM February, Desirable Qualities Frequent testing –Identify defects as soon as they are introduced –Hard to find cause if not done frequently Minimally invasive to daily activities Automated testing Automated report generation/summaries –Must be concise yet informative Track results over time

NA-MIC National Alliance for Medical Image Computing NAMIC Software Process Dan Blezek Jim Miller Bill Lorensen

National Alliance for Medical Image Computing Salt Lake City AHM February, Motivation Many algorithms, many platforms –VTK, ITK, Slicer, LONI –Linux, Windows, Mac OSX, Solaris, SGI Many users Many datasets Many sources of problems!

National Alliance for Medical Image Computing Salt Lake City AHM February, Motivation Negative example MIT codes ITK algorithm for LONI pipeline UCLA developer changes LONI GE changes ITK Time for release, everything’s broken

National Alliance for Medical Image Computing Salt Lake City AHM February, Motivation Ensuring high quality software System’s state must be known –If UCLA knew about MIT code, they would have been more careful w/changes All the code works, all the time –As often as is feasible, compile and test the code

National Alliance for Medical Image Computing Salt Lake City AHM February, Extreme Programming

National Alliance for Medical Image Computing Salt Lake City AHM February, NAMIC Process Light weight Based on Extreme Programming –High intensity cycle Design Test Implement –Supported with web-enabled tools –Automated testing integrated with the software development

National Alliance for Medical Image Computing Salt Lake City AHM February, Software Process Design Process Coding Standards Testing Bug Tracker Communication –Mailing lists, Discussion forum, Wiki –Tcons Documentation Releases

National Alliance for Medical Image Computing Salt Lake City AHM February, Design Process Take the time to design a good API Plan for future use Plan for future extension Two routes –Code something, check it in Others will tear it down & make it better –Put together a strawman Solicit ideas, implement

National Alliance for Medical Image Computing Salt Lake City AHM February, Coding Standards Follow the package’s rules ITK has certain coding standards –Style guidelines –Naming conventions –Macros

National Alliance for Medical Image Computing Salt Lake City AHM February, Testing If it isn’t tested, it’s broken. Tests –Ensure your code works –Documents expected results –Others free to change

National Alliance for Medical Image Computing Salt Lake City AHM February, Bug Tracker Bugs assigned / taken by developers Tracks progress to releases Captures feature requests Communication mechanism

National Alliance for Medical Image Computing Salt Lake City AHM February, Documentation Doxygen –Automatic API documentation –Algorithm references –Implementation details Books / Manuals –Insight Book

National Alliance for Medical Image Computing Salt Lake City AHM February, Communication lists Discussion forum Wiki Tcon

National Alliance for Medical Image Computing Salt Lake City AHM February, Extreme Programming Compression of standard analyze, design, implement, test cycle into a continuous process

National Alliance for Medical Image Computing Salt Lake City AHM February, Daily Testing Is The Key Testing anchors the development process (Dart) Developers monitor the testing dashboard constantly Problems are identified and fixed immediately Developers receive if they “Break the Build”

National Alliance for Medical Image Computing Salt Lake City AHM February, Daily rhythm Design, implement algorithm write regression test check it in Dart takes over Code is updated, built, tested Guilty parties are blamed Core 2 Build breakers are notified Work on cleaning up dashboard Repeat

National Alliance for Medical Image Computing Salt Lake City AHM February, Dart Testing Reports Dashboards Central site for state of the system –Updates –Builds –Test –Coverage

National Alliance for Medical Image Computing Salt Lake City AHM February,

National Alliance for Medical Image Computing Salt Lake City AHM February,

National Alliance for Medical Image Computing Salt Lake City AHM February, Someone broke the build!

National Alliance for Medical Image Computing Salt Lake City AHM February,

National Alliance for Medical Image Computing Salt Lake City AHM February, Conclusion Have fun Process extends your impact –Many can use your code –Many can improve your code Communicate, Communicate, Communicate

NA-MIC National Alliance for Medical Image Computing NAMIC Software Process Dan Blezek Jim Miller Bill Lorensen