1. September 6, 2007Open Source Software Practice Overview of the Software Process Bill Lorensen GE Research (retired) bill.lorensen@gmail.com

2. September 6, 2007Open Source Software Practice Course Wiki http://public.kitware.com/OpenSourceSoftwarePractice

3. September 6, 2007Open Source Software Practice Outline Motivation Motivation The Process The Process Case Study – Insight Toolkit Case Study – Insight Toolkit Lessons Learned Lessons Learned

4. September 6, 2007Open Source Software Practice A Personal Look Back 1976- NASA’s Computer Software Management and Information System (COSMIC) 1976- NASA’s Computer Software Management and Information System (COSMIC) 1978 - MOVIE.BYU 1978 - MOVIE.BYU 1984 - GE Research Workstation 1984 - GE Research Workstation 1984 - LYMB 1984 - LYMB 1994 - Visualization Toolkit 1994 - Visualization Toolkit 2000 - Insight Toolkit 2000 - Insight Toolkit

5. September 6, 2007Open Source Software Practice Flattener #4 Open Sourcing Self-organizing collaborative communities

6. September 6, 2007Open Source Software Practice Open Source Anecdote * IBM Manager: “So, walk me through the development process for e-commerce. What’s the underlying web server?” Developer: “It’s built on top of Apache.” Manager: “Apache?” Developer: “A shareware program for web server technology, produced for free by a bunch of geeks working online in some open-source chat room.” Manager: “How do you buy it?” Developer: “You download it off a Web site for free.” Manager: “Who supports it if something goes wrong?” Developer: “I don’t know, it just works!” *from The World is Flat

7. September 6, 2007Open Source Software Practice Successful Open Source Projects

8. September 6, 2007Open Source Software Practice Visualization Toolkit - vtk Open source toolkit for scientific visualization, computer graphics, and image processing www.vtk.org

9. September 6, 2007Open Source Software Practice Computer Vision Library - vxl Based on Target jr and Image Understanding Environment Based on Target jr and Image Understanding Environment Numerics Numerics Imaging Imaging Geometry Geometry Camera models Camera models vxl.sourceforge.net

10. September 6, 2007Open Source Software Practice Slicer Developed by Brigham and Womens Surgical Planning Lab Developed by Brigham and Womens Surgical Planning Lab User interface plus plug-ins for applications User interface plus plug-ins for applications Segmentation Segmentation Registration Registration Image Guidance Image Guidance http://www.slicer.org

11. September 6, 2007Open Source Software Practice SCIRun http://www.sci.utah.edu Developed by U Utah SCI Institute Developed by U Utah SCI Institute Computational Workbench Computational Workbench Visual Programming Visual Programming Modeling, Simulation and Visualization Modeling, Simulation and Visualization

12. September 6, 2007Open Source Software Practice National Library of Medicine Segmentation and Registration Toolkit $14 million over 6 years Leading edge algorithms Open Source Software www.itk.org

13. September 6, 2007Open Source Software Practice Open Source Menu for Success A Community with a common vision A Community with a common vision A pool of talented and motivated developers/scientists A pool of talented and motivated developers/scientists A mix of academic and commercial A mix of academic and commercial An organized, light weight approach to software development An organized, light weight approach to software development A leadership structure A leadership structure Communication Communication A business model A business model

14. September 6, 2007Open Source Software Practice Project Stages Startup Startup Early Early Middle Middle Mature Mature

15. September 6, 2007Open Source Software Practice Project Stages - Startup Define goals and/or requirements Define goals and/or requirements Select developer community Select developer community Define developer roles Define developer roles Select host server Select host server Roll your own Roll your own SourceForge, CollabNet, GForge SourceForge, CollabNet, GForge Select software development tools Select software development tools Revision control system Revision control system Compilers Compilers Code coverage Code coverage Dynamic analysis Dynamic analysis Build system Build system Testing system Testing system Select an open source license Select an open source license

16. September 6, 2007Open Source Software Practice Project Stages - Early Refine tools Refine tools Change ASAP if necessary Change ASAP if necessary Establish coding guidelines Establish coding guidelines Setup communication channels Setup communication channels Mailing lists Mailing lists Developers Developers Users Users Establish web presence Establish web presence Project home pages Project home pages Wiki for developers Wiki for developers Establish nightly build/test Establish nightly build/test

17. September 6, 2007Open Source Software Practice Project Stages - Middle Establish release mechanisms Establish release mechanisms Establish policy to add additional developers Establish policy to add additional developers Establish backward compatibility policy Establish backward compatibility policy Establish procedure to add new features Establish procedure to add new features Marketing Marketing Papers Papers Tutorials Tutorials Web buzz Web buzz

18. September 6, 2007Open Source Software Practice Project Stages - Mature Project has an installed base Project has an installed base Everything should be hard to change Everything should be hard to change Code Code Tools Tools Process Process

19. September 6, 2007Open Source Software Practice Inside Insight (itk)

20. September 6, 2007Open Source Software Practice What is itk? A common Application Programmers Interface (API). A common Application Programmers Interface (API). A framework for software development A framework for software development A toolkit for registration and segmentation A toolkit for registration and segmentation An Open Source resource for future research An Open Source resource for future research A validation model for segmentation and registration. A validation model for segmentation and registration. A framework for validation development A framework for validation development Assistance for algorithm designers Assistance for algorithm designers A seed repository for validation case studies A seed repository for validation case studies

21. September 6, 2007Open Source Software Practice Insight - Open Source Products

22. September 6, 2007Open Source Software Practice itk by the Numbers March 2000 March 2000 First code checkin First code checkin 1600 1600 # of nightly builds # of nightly builds 1200 1200 # tests run nightly # tests run nightly 50 50 # of platforms # of platforms 730 730 # of classes # of classes 2000 2000 # of files with code # of files with code 400K 400K # of lines of code # of lines of code 100K 100K # of lines of test code # of lines of test code 45K 45K # of lines of examples # of lines of examples 160K 160K # of lines of Applications # of lines of Applications > 300 > 300 # weekly t-cons # weekly t-cons 85 85 # unique developers # unique developers

23. September 6, 2007Open Source Software Practice itk by the Numbers 186 186 # of subscribers to the developers mailing list # of subscribers to the developers mailing list 900 900 # of subscribers to the users mailing list # of subscribers to the users mailing list 350 350 # of monthly posts to users-list # of monthly posts to users-list

24. September 6, 2007Open Source Software Practice http://ohloh.net

25. September 6, 2007Open Source Software Practice http://ohloh.net

26. September 6, 2007Open Source Software Practice A Common Vision Create a dynamic, self-sustaining, public domain and extensible toolkit that will empower researchers throughout the world to develop new segmentation and registration algorithms and create new applications that leverage the NLM’s investment in the Visible Human Male and Female data sets

27. September 6, 2007Open Source Software Practice The Original Team Six prime contractors Six prime contractors Industrial Industrial Kitware (Schroeder) Kitware (Schroeder) Insightful (Ng) Insightful (Ng) UPenn (Gee) UPenn (Gee) GE Research (Lorensen) GE Research (Lorensen) Brigham and Womens (Kikinis) Brigham and Womens (Kikinis) Academic Academic UNC (Aylward) UNC (Aylward) Pitt (Stetton) Pitt (Stetton) Utah (Whitaker) Utah (Whitaker) Rutgers (Metaxas) Rutgers (Metaxas) Columbia (Imielenski) Columbia (Imielenski) UPenn (Udupa) UPenn (Udupa)

28. September 6, 2007Open Source Software Practice Insight Consortium A competitive process selected the development team A competitive process selected the development team Each group was evaluated individually, without regard to how they might integrate with each other Each group was evaluated individually, without regard to how they might integrate with each other Each team had strengths in software development, validation or algorithm development, but no one group had the necessary skills to do the entire project Each team had strengths in software development, validation or algorithm development, but no one group had the necessary skills to do the entire project Distributed software developers with varying software engineering experience Distributed software developers with varying software engineering experience

29. September 6, 2007Open Source Software Practice Insight Software Architecture Object-oriented design Object-oriented design Generic Programming Generic Programming Design Patterns Design Patterns Frameworks Frameworks Separation of Algorithms from Interfaces Separation of Algorithms from Interfaces

30. September 6, 2007Open Source Software Practice Object-Oriented Design Dominated software systems throughout the 1990’s Dominated software systems throughout the 1990’s Continues to be the accepted software design technique Continues to be the accepted software design technique Particularly useful for dealing with complexity Particularly useful for dealing with complexity Provides programmatic abstractions to deal with generalization and encapsulation Provides programmatic abstractions to deal with generalization and encapsulation C++ and Java have mechanisms to support OOD C++ and Java have mechanisms to support OOD

31. September 6, 2007Open Source Software Practice Generic Programming Organize libraries consisting of generic— or reusable—software components. Organize libraries consisting of generic— or reusable—software components. The essential ideas of generic programming are containers to hold data, iterators to access the data, and generic algorithms that use containers and iterators to create efficient, fundamental algorithms. The essential ideas of generic programming are containers to hold data, iterators to access the data, and generic algorithms that use containers and iterators to create efficient, fundamental algorithms. ITK uses generic programming to process n-dimensional “images”. ITK uses generic programming to process n-dimensional “images”.

32. September 6, 2007Open Source Software Practice Design Patterns Good object-oriented software systems have recurring designs (patterns) that occur frequently Good object-oriented software systems have recurring designs (patterns) that occur frequently ITK employs a number of powerful design patterns ITK employs a number of powerful design patterns object factories object factories command/observer command/observer smart pointer memory management smart pointer memory management

33. September 6, 2007Open Source Software Practice Frameworks Define how a group of participants can be put together to solve a particular task. Define how a group of participants can be put together to solve a particular task. Particularly suitable for describing complex flows or algorithms that have a number of steps that can be varied Particularly suitable for describing complex flows or algorithms that have a number of steps that can be varied ITK Frameworks ITK Frameworks A demand-driven data processing pipeline that connects algorithms to process n-dimensional image data A demand-driven data processing pipeline that connects algorithms to process n-dimensional image data Registration framework Registration framework Level-set framework Level-set framework

34. September 6, 2007Open Source Software Practice Registration Framework

35. September 6, 2007Open Source Software Practice Level-Set Framework

36. September 6, 2007Open Source Software Practice Separation of Algorithms from Interfaces Implement the algorithms with a clear separation from the applications and especially the user interfaces. Implement the algorithms with a clear separation from the applications and especially the user interfaces. Uses the Command/Observer design pattern that permits applications to watch for significant events during the execution of an algorithm Uses the Command/Observer design pattern that permits applications to watch for significant events during the execution of an algorithm ITK has no built-in visualization, but has been interfaced to several systems including 3D Slicer, Analyze, SciRun and Volview. ITK has no built-in visualization, but has been interfaced to several systems including 3D Slicer, Analyze, SciRun and Volview.

37. September 6, 2007Open Source Software Practice In search of a new software development process

38. Google Hits “Extreme Programming”: 1,860,000

39. September 6, 2007Open Source Software Practice Extreme Programming

40. September 6, 2007Open Source Software Practice A Light Weight Software Engineering Process Based on the new Extreme Programming process Based on the new Extreme Programming process High intensity design, test, implement cycle High intensity design, test, implement cycle Supported with web-enabled tools Supported with web-enabled tools Automated testing integrated with the software development Automated testing integrated with the software development

41. September 6, 2007Open Source Software Practice Extreme Programming Compresses the standard analyze, design, implement, test cycle into a continuous process

42. September 6, 2007Open Source Software Practice A process supported by a suite of portable, open source tools Apache, perl, php Apache, perl, php Web services Web services cvs, subversion cvs, subversion Revision control Revision control MediaWiki MediaWiki Collaborative content capture Collaborative content capture Doxygen Doxygen Automated documentation Automated documentation CMake CMake Cross-platform program build Cross-platform program build Dart Dart Continuous and distributed test reporting Continuous and distributed test reporting

43. September 6, 2007Open Source Software Practice Extreme Programming The community owns the code Although the identity of the original author is kept, other developers are free to correct defects and enhance each other's code Although the identity of the original author is kept, other developers are free to correct defects and enhance each other's code In the end, all of the software should appear as though one author wrote it

44. September 6, 2007Open Source Software Practice Extreme Programming Release early, release often Although developers are tempted to keep their code under wraps until it is perfect, the process encourages them to release their code as soon as it passes some minimum tests The longer the code is visible to the community, the better integrated it will be

45. September 6, 2007Open Source Software Practice Extreme Programming Continuous integration There is no scheduled porting to computer platforms All new software builds supported platforms every evening

46. September 6, 2007Open Source Software Practice Extreme Programming All developers agree to keep the software defect free Although everyone is encouraged to submit their code early, the code must compile and pass tests nightly A continuous build process sends e-mails to developers who check in code that does not compile More effectively, the community enforces the commitment though peer pressure

47. September 6, 2007Open Source Software Practice Insight - Development Cycles Daily – dashboard Daily – dashboard Weekly – telephone conferences Weekly – telephone conferences Periodic – architecture reviews Periodic – architecture reviews Quarterly – developer meetings Quarterly – developer meetings Yearly – work assignments Yearly – work assignments

48. September 6, 2007Open Source Software Practice Extreme Programming Daily Testing Is The Key Testing anchors and drives the development process (Dart) Testing anchors and drives the development process (Dart) Opens up the development process to everyone Opens up the development process to everyone Developers monitor the testing dashboard constantly Developers monitor the testing dashboard constantly Problems are identified and fixed immediately Problems are identified and fixed immediately Developers receive e-mail if they “Break the Build” Developers receive e-mail if they “Break the Build”

49. September 6, 2007Open Source Software Practice How DART Enables Collaboration CVS Results posted on web (the dashboard) CVS maintains source code revisions DART compiles source code, runs tests Developers review results Developers check-in code

50. September 6, 2007Open Source Software Practice Dart

51. September 6, 2007Open Source Software Practice

52. September 6, 2007Open Source Software Practice Multi-Platform Builds

53. September 6, 2007Open Source Software Practice Regression Testing

54. September 6, 2007Open Source Software Practice Continuous System Monitoring

55. September 6, 2007Open Source Software Practice Someone broke the build!

56. September 6, 2007Open Source Software Practice Insight Toolkit Lessons Learned

57. September 6, 2007Open Source Software Practice Lessons Learned: The Good Good mix of commercial and academic Good mix of commercial and academic Communication is critical Communication is critical The daily rhythm of Extreme Testing The daily rhythm of Extreme Testing The Whole >>> Sum of the parts The Whole >>> Sum of the parts Process automation reduces the burden on developers Process automation reduces the burden on developers The process is open and repeatable The process is open and repeatable

58. September 6, 2007Open Source Software Practice Lessons Learned: The Bad Communication Early in the project, communication was limited to the mailing list and the Quarterly meetings. Almost a year passed before we added the weekly telephone conferences. Developers felt isolated and the impersonal electronic communication methods failed to build the personal relationships required for any software development process. Early in the project, communication was limited to the mailing list and the Quarterly meetings. Almost a year passed before we added the weekly telephone conferences. Developers felt isolated and the impersonal electronic communication methods failed to build the personal relationships required for any software development process.Solution Weekly developer t-cons Weekly developer t-cons Wiki Wiki

59. September 6, 2007Open Source Software Practice Lessons Learned: The Bad Lack of Design Reviews The lightweight process has its drawbacks. The current process emphasizes coding and testing. This process is well defined and benefits from automation. But, the design process is not well supported by the current mechanisms. The lightweight process has its drawbacks. The current process emphasizes coding and testing. This process is well defined and benefits from automation. But, the design process is not well supported by the current mechanisms.Solution Insight Journal review process Insight Journal review process

60. September 6, 2007Open Source Software Practice Lessons Learned: The Bad Unstable Application Programming Interfaces (API’s) The nightly test/build was so effective in empowering programmers to make changes, that API changes occurred too frequently without the necessary buy-in from the development community. The nightly test/build was so effective in empowering programmers to make changes, that API changes occurred too frequently without the necessary buy-in from the development community.Solution Established backward compatibility policy Established backward compatibility policy

61. September 6, 2007Open Source Software Practice Insight Observations: The Bad Insufficient Automation Automation is critical to maintain consistent style and robust software. Although the project provided many automatic techniques to catch developer errors, the project would benefit from more automation. Automation is critical to maintain consistent style and robust software. Although the project provided many automatic techniques to catch developer errors, the project would benefit from more automation.Solution KWStyle – tool to do style checking KWStyle – tool to do style checking

62. September 6, 2007Open Source Software Practice Insight Observations: The Bad Complexity and Feature Creep Like most software development projects, this software suffers from complexity and feature creep. Early designs and implementations used many features of the C++ language to an “extreme”. The team realized and began removing the complexity. Like most software development projects, this software suffers from complexity and feature creep. Early designs and implementations used many features of the C++ language to an “extreme”. The team realized and began removing the complexity.Solution Ongoing reviews Ongoing reviews

63. September 6, 2007Open Source Software Practice Lessons Learned: The Ugly Still need a stick Still need a stick Someone needs to monitor the daily process. Someone needs to monitor the daily process. XP is not everyone’s cup of tea XP is not everyone’s cup of tea Almost all bought into the process. A few tolerated it. Almost all bought into the process. A few tolerated it.

64. September 6, 2007Open Source Software Practice Open Source Resources http://www.opensource.org http://www.opensource.org http://www.opensource.org http://sourceforge.net http://sourceforge.net http://sourceforge.net http://gforge.org http://gforge.org http://gforge.org http://www.itk.org http://www.itk.org http://www.itk.org http://www.vtk.org http://www.vtk.org http://www.vtk.org http://vxl.sourceforge.net http://vxl.sourceforge.net http://vxl.sourceforge.net http://www.slicer.org http://www.slicer.org http://www.slicer.org http://www.sci.utah.edu http://www.sci.utah.edu http://www.sci.utah.edu http://www.google.com/codesearch http://www.google.com/codesearch http://www.google.com/codesearch http://ohloh.net http://ohloh.net http://ohloh.net

65. September 6, 2007Open Source Software Practice Quiz Monday – Reading List Free Culture, Lawrence Lessig, Free Culture, Lawrence Lessig, Available online at: http://www.free-culture.cc/freecontent/ Available online at: http://www.free-culture.cc/freecontent/http://www.free-culture.cc/freecontent/ Sections to read Sections to read Introduction Introduction Chapter Four Chapter Four Free Software Free Society, Richard Stallman Free Software Free Society, Richard Stallman Available online at Available online at http://www.gnu.org/doc/book13.html (HTML) http://www.gnu.org/doc/book13.html (HTML) http://www.gnu.org/doc/book13.html http://www.gnu.org/philosophy/fsfs/rms-essays.pdf (PDF) http://www.gnu.org/philosophy/fsfs/rms-essays.pdf (PDF) http://www.gnu.org/philosophy/fsfs/rms-essays.pdf Sections to read Sections to read Chapter One: "The GNU Project" Chapter One: "The GNU Project" Chapter Two: "The GNU Manifesto" Chapter Two: "The GNU Manifesto" Chapter Three: "Free Software Definition" Chapter Three: "Free Software Definition" Chapter Four: "Why software should not have owners" Chapter Four: "Why software should not have owners" Open Source Software Licensing, Lawrence Rosen Open Source Software Licensing, Lawrence Rosen Available online at: http://www.rosenlaw.com/oslbook.htm Available online at: http://www.rosenlaw.com/oslbook.htmhttp://www.rosenlaw.com/oslbook.htm Sections to read Sections to read Chapter Two: "Intellectual Property" Chapter Two: "Intellectual Property""Intellectual Property""Intellectual Property" Chapter Four: "Taxonomy of Licenses" Chapter Four: "Taxonomy of Licenses""Taxonomy of Licenses""Taxonomy of Licenses" Retrieved from "http://public.kitware.com/OpenSourceSoftwarePractice/index.php/Intellectual_Monopolies_Reading_Assignement_I" Retrieved from "http://public.kitware.com/OpenSourceSoftwarePractice/index.php/Intellectual_Monopolies_Reading_Assignement_I"http://public.kitware.com/OpenSourceSoftwarePractice/index.php/Intellectual_Monopolies_Reading_Assignement_I

66. September 6, 2007Open Source Software Practice Overview of the Software Process Bill Lorensen GE Research (retired) bill.lorensen@gmail.com