September 6, 2007Open Source Software Practice Overview of the Software Process Bill Lorensen GE Research (retired)

September 6, 2007Open Source Software Practice Course Wiki

September 6, 2007Open Source Software Practice Outline Motivation Motivation The Process The Process Case Study – Insight Toolkit Case Study – Insight Toolkit Lessons Learned Lessons Learned

September 6, 2007Open Source Software Practice A Personal Look Back NASA’s Computer Software Management and Information System (COSMIC) NASA’s Computer Software Management and Information System (COSMIC) MOVIE.BYU MOVIE.BYU GE Research Workstation GE Research Workstation LYMB LYMB Visualization Toolkit Visualization Toolkit Insight Toolkit Insight Toolkit

September 6, 2007Open Source Software Practice Flattener #4 Open Sourcing Self-organizing collaborative communities

September 6, 2007Open Source Software Practice Open Source Anecdote * IBM Manager: “So, walk me through the development process for e-commerce. What’s the underlying web server?” Developer: “It’s built on top of Apache.” Manager: “Apache?” Developer: “A shareware program for web server technology, produced for free by a bunch of geeks working online in some open-source chat room.” Manager: “How do you buy it?” Developer: “You download it off a Web site for free.” Manager: “Who supports it if something goes wrong?” Developer: “I don’t know, it just works!” *from The World is Flat

September 6, 2007Open Source Software Practice Successful Open Source Projects

September 6, 2007Open Source Software Practice Visualization Toolkit - vtk Open source toolkit for scientific visualization, computer graphics, and image processing

September 6, 2007Open Source Software Practice Computer Vision Library - vxl Based on Target jr and Image Understanding Environment Based on Target jr and Image Understanding Environment Numerics Numerics Imaging Imaging Geometry Geometry Camera models Camera models vxl.sourceforge.net

September 6, 2007Open Source Software Practice Slicer Developed by Brigham and Womens Surgical Planning Lab Developed by Brigham and Womens Surgical Planning Lab User interface plus plug-ins for applications User interface plus plug-ins for applications Segmentation Segmentation Registration Registration Image Guidance Image Guidance

September 6, 2007Open Source Software Practice SCIRun Developed by U Utah SCI Institute Developed by U Utah SCI Institute Computational Workbench Computational Workbench Visual Programming Visual Programming Modeling, Simulation and Visualization Modeling, Simulation and Visualization

September 6, 2007Open Source Software Practice National Library of Medicine Segmentation and Registration Toolkit $14 million over 6 years Leading edge algorithms Open Source Software

September 6, 2007Open Source Software Practice Open Source Menu for Success A Community with a common vision A Community with a common vision A pool of talented and motivated developers/scientists A pool of talented and motivated developers/scientists A mix of academic and commercial A mix of academic and commercial An organized, light weight approach to software development An organized, light weight approach to software development A leadership structure A leadership structure Communication Communication A business model A business model

September 6, 2007Open Source Software Practice Project Stages Startup Startup Early Early Middle Middle Mature Mature

September 6, 2007Open Source Software Practice Project Stages - Startup Define goals and/or requirements Define goals and/or requirements Select developer community Select developer community Define developer roles Define developer roles Select host server Select host server Roll your own Roll your own SourceForge, CollabNet, GForge SourceForge, CollabNet, GForge Select software development tools Select software development tools Revision control system Revision control system Compilers Compilers Code coverage Code coverage Dynamic analysis Dynamic analysis Build system Build system Testing system Testing system Select an open source license Select an open source license

September 6, 2007Open Source Software Practice Project Stages - Early Refine tools Refine tools Change ASAP if necessary Change ASAP if necessary Establish coding guidelines Establish coding guidelines Setup communication channels Setup communication channels Mailing lists Mailing lists Developers Developers Users Users Establish web presence Establish web presence Project home pages Project home pages Wiki for developers Wiki for developers Establish nightly build/test Establish nightly build/test

September 6, 2007Open Source Software Practice Project Stages - Middle Establish release mechanisms Establish release mechanisms Establish policy to add additional developers Establish policy to add additional developers Establish backward compatibility policy Establish backward compatibility policy Establish procedure to add new features Establish procedure to add new features Marketing Marketing Papers Papers Tutorials Tutorials Web buzz Web buzz

September 6, 2007Open Source Software Practice Project Stages - Mature Project has an installed base Project has an installed base Everything should be hard to change Everything should be hard to change Code Code Tools Tools Process Process

September 6, 2007Open Source Software Practice Inside Insight (itk)

September 6, 2007Open Source Software Practice What is itk? A common Application Programmers Interface (API). A common Application Programmers Interface (API). A framework for software development A framework for software development A toolkit for registration and segmentation A toolkit for registration and segmentation An Open Source resource for future research An Open Source resource for future research A validation model for segmentation and registration. A validation model for segmentation and registration. A framework for validation development A framework for validation development Assistance for algorithm designers Assistance for algorithm designers A seed repository for validation case studies A seed repository for validation case studies

September 6, 2007Open Source Software Practice Insight - Open Source Products

September 6, 2007Open Source Software Practice itk by the Numbers March 2000 March 2000 First code checkin First code checkin # of nightly builds # of nightly builds # tests run nightly # tests run nightly # of platforms # of platforms # of classes # of classes # of files with code # of files with code 400K 400K # of lines of code # of lines of code 100K 100K # of lines of test code # of lines of test code 45K 45K # of lines of examples # of lines of examples 160K 160K # of lines of Applications # of lines of Applications > 300 > 300 # weekly t-cons # weekly t-cons # unique developers # unique developers

September 6, 2007Open Source Software Practice itk by the Numbers # of subscribers to the developers mailing list # of subscribers to the developers mailing list # of subscribers to the users mailing list # of subscribers to the users mailing list # of monthly posts to users-list # of monthly posts to users-list

September 6, 2007Open Source Software Practice

September 6, 2007Open Source Software Practice

September 6, 2007Open Source Software Practice A Common Vision Create a dynamic, self-sustaining, public domain and extensible toolkit that will empower researchers throughout the world to develop new segmentation and registration algorithms and create new applications that leverage the NLM’s investment in the Visible Human Male and Female data sets

September 6, 2007Open Source Software Practice The Original Team Six prime contractors Six prime contractors Industrial Industrial Kitware (Schroeder) Kitware (Schroeder) Insightful (Ng) Insightful (Ng) UPenn (Gee) UPenn (Gee) GE Research (Lorensen) GE Research (Lorensen) Brigham and Womens (Kikinis) Brigham and Womens (Kikinis) Academic Academic UNC (Aylward) UNC (Aylward) Pitt (Stetton) Pitt (Stetton) Utah (Whitaker) Utah (Whitaker) Rutgers (Metaxas) Rutgers (Metaxas) Columbia (Imielenski) Columbia (Imielenski) UPenn (Udupa) UPenn (Udupa)

September 6, 2007Open Source Software Practice Insight Consortium A competitive process selected the development team A competitive process selected the development team Each group was evaluated individually, without regard to how they might integrate with each other Each group was evaluated individually, without regard to how they might integrate with each other Each team had strengths in software development, validation or algorithm development, but no one group had the necessary skills to do the entire project Each team had strengths in software development, validation or algorithm development, but no one group had the necessary skills to do the entire project Distributed software developers with varying software engineering experience Distributed software developers with varying software engineering experience

September 6, 2007Open Source Software Practice Insight Software Architecture Object-oriented design Object-oriented design Generic Programming Generic Programming Design Patterns Design Patterns Frameworks Frameworks Separation of Algorithms from Interfaces Separation of Algorithms from Interfaces

September 6, 2007Open Source Software Practice Object-Oriented Design Dominated software systems throughout the 1990’s Dominated software systems throughout the 1990’s Continues to be the accepted software design technique Continues to be the accepted software design technique Particularly useful for dealing with complexity Particularly useful for dealing with complexity Provides programmatic abstractions to deal with generalization and encapsulation Provides programmatic abstractions to deal with generalization and encapsulation C++ and Java have mechanisms to support OOD C++ and Java have mechanisms to support OOD

September 6, 2007Open Source Software Practice Generic Programming Organize libraries consisting of generic— or reusable—software components. Organize libraries consisting of generic— or reusable—software components. The essential ideas of generic programming are containers to hold data, iterators to access the data, and generic algorithms that use containers and iterators to create efficient, fundamental algorithms. The essential ideas of generic programming are containers to hold data, iterators to access the data, and generic algorithms that use containers and iterators to create efficient, fundamental algorithms. ITK uses generic programming to process n-dimensional “images”. ITK uses generic programming to process n-dimensional “images”.

September 6, 2007Open Source Software Practice Design Patterns Good object-oriented software systems have recurring designs (patterns) that occur frequently Good object-oriented software systems have recurring designs (patterns) that occur frequently ITK employs a number of powerful design patterns ITK employs a number of powerful design patterns object factories object factories command/observer command/observer smart pointer memory management smart pointer memory management

September 6, 2007Open Source Software Practice Frameworks Define how a group of participants can be put together to solve a particular task. Define how a group of participants can be put together to solve a particular task. Particularly suitable for describing complex flows or algorithms that have a number of steps that can be varied Particularly suitable for describing complex flows or algorithms that have a number of steps that can be varied ITK Frameworks ITK Frameworks A demand-driven data processing pipeline that connects algorithms to process n-dimensional image data A demand-driven data processing pipeline that connects algorithms to process n-dimensional image data Registration framework Registration framework Level-set framework Level-set framework

September 6, 2007Open Source Software Practice Registration Framework

September 6, 2007Open Source Software Practice Level-Set Framework

September 6, 2007Open Source Software Practice Separation of Algorithms from Interfaces Implement the algorithms with a clear separation from the applications and especially the user interfaces. Implement the algorithms with a clear separation from the applications and especially the user interfaces. Uses the Command/Observer design pattern that permits applications to watch for significant events during the execution of an algorithm Uses the Command/Observer design pattern that permits applications to watch for significant events during the execution of an algorithm ITK has no built-in visualization, but has been interfaced to several systems including 3D Slicer, Analyze, SciRun and Volview. ITK has no built-in visualization, but has been interfaced to several systems including 3D Slicer, Analyze, SciRun and Volview.

September 6, 2007Open Source Software Practice In search of a new software development process

Google Hits “Extreme Programming”: 1,860,000

September 6, 2007Open Source Software Practice Extreme Programming

September 6, 2007Open Source Software Practice A Light Weight Software Engineering Process Based on the new Extreme Programming process Based on the new Extreme Programming process High intensity design, test, implement cycle High intensity design, test, implement cycle Supported with web-enabled tools Supported with web-enabled tools Automated testing integrated with the software development Automated testing integrated with the software development

September 6, 2007Open Source Software Practice Extreme Programming Compresses the standard analyze, design, implement, test cycle into a continuous process

September 6, 2007Open Source Software Practice A process supported by a suite of portable, open source tools Apache, perl, php Apache, perl, php Web services Web services cvs, subversion cvs, subversion Revision control Revision control MediaWiki MediaWiki Collaborative content capture Collaborative content capture Doxygen Doxygen Automated documentation Automated documentation CMake CMake Cross-platform program build Cross-platform program build Dart Dart Continuous and distributed test reporting Continuous and distributed test reporting

September 6, 2007Open Source Software Practice Extreme Programming The community owns the code Although the identity of the original author is kept, other developers are free to correct defects and enhance each other's code Although the identity of the original author is kept, other developers are free to correct defects and enhance each other's code In the end, all of the software should appear as though one author wrote it

September 6, 2007Open Source Software Practice Extreme Programming Release early, release often Although developers are tempted to keep their code under wraps until it is perfect, the process encourages them to release their code as soon as it passes some minimum tests The longer the code is visible to the community, the better integrated it will be

September 6, 2007Open Source Software Practice Extreme Programming Continuous integration There is no scheduled porting to computer platforms All new software builds supported platforms every evening

September 6, 2007Open Source Software Practice Extreme Programming All developers agree to keep the software defect free Although everyone is encouraged to submit their code early, the code must compile and pass tests nightly A continuous build process sends s to developers who check in code that does not compile More effectively, the community enforces the commitment though peer pressure

September 6, 2007Open Source Software Practice Insight - Development Cycles Daily – dashboard Daily – dashboard Weekly – telephone conferences Weekly – telephone conferences Periodic – architecture reviews Periodic – architecture reviews Quarterly – developer meetings Quarterly – developer meetings Yearly – work assignments Yearly – work assignments

September 6, 2007Open Source Software Practice Extreme Programming Daily Testing Is The Key Testing anchors and drives the development process (Dart) Testing anchors and drives the development process (Dart) Opens up the development process to everyone Opens up the development process to everyone Developers monitor the testing dashboard constantly Developers monitor the testing dashboard constantly Problems are identified and fixed immediately Problems are identified and fixed immediately Developers receive if they “Break the Build” Developers receive if they “Break the Build”

September 6, 2007Open Source Software Practice How DART Enables Collaboration CVS Results posted on web (the dashboard) CVS maintains source code revisions DART compiles source code, runs tests Developers review results Developers check-in code

September 6, 2007Open Source Software Practice Dart

September 6, 2007Open Source Software Practice

September 6, 2007Open Source Software Practice Multi-Platform Builds

September 6, 2007Open Source Software Practice Regression Testing

September 6, 2007Open Source Software Practice Continuous System Monitoring

September 6, 2007Open Source Software Practice Someone broke the build!

September 6, 2007Open Source Software Practice Insight Toolkit Lessons Learned

September 6, 2007Open Source Software Practice Lessons Learned: The Good Good mix of commercial and academic Good mix of commercial and academic Communication is critical Communication is critical The daily rhythm of Extreme Testing The daily rhythm of Extreme Testing The Whole >>> Sum of the parts The Whole >>> Sum of the parts Process automation reduces the burden on developers Process automation reduces the burden on developers The process is open and repeatable The process is open and repeatable

September 6, 2007Open Source Software Practice Lessons Learned: The Bad Communication Early in the project, communication was limited to the mailing list and the Quarterly meetings. Almost a year passed before we added the weekly telephone conferences. Developers felt isolated and the impersonal electronic communication methods failed to build the personal relationships required for any software development process. Early in the project, communication was limited to the mailing list and the Quarterly meetings. Almost a year passed before we added the weekly telephone conferences. Developers felt isolated and the impersonal electronic communication methods failed to build the personal relationships required for any software development process.Solution Weekly developer t-cons Weekly developer t-cons Wiki Wiki

September 6, 2007Open Source Software Practice Lessons Learned: The Bad Lack of Design Reviews The lightweight process has its drawbacks. The current process emphasizes coding and testing. This process is well defined and benefits from automation. But, the design process is not well supported by the current mechanisms. The lightweight process has its drawbacks. The current process emphasizes coding and testing. This process is well defined and benefits from automation. But, the design process is not well supported by the current mechanisms.Solution Insight Journal review process Insight Journal review process

September 6, 2007Open Source Software Practice Lessons Learned: The Bad Unstable Application Programming Interfaces (API’s) The nightly test/build was so effective in empowering programmers to make changes, that API changes occurred too frequently without the necessary buy-in from the development community. The nightly test/build was so effective in empowering programmers to make changes, that API changes occurred too frequently without the necessary buy-in from the development community.Solution Established backward compatibility policy Established backward compatibility policy

September 6, 2007Open Source Software Practice Insight Observations: The Bad Insufficient Automation Automation is critical to maintain consistent style and robust software. Although the project provided many automatic techniques to catch developer errors, the project would benefit from more automation. Automation is critical to maintain consistent style and robust software. Although the project provided many automatic techniques to catch developer errors, the project would benefit from more automation.Solution KWStyle – tool to do style checking KWStyle – tool to do style checking

September 6, 2007Open Source Software Practice Insight Observations: The Bad Complexity and Feature Creep Like most software development projects, this software suffers from complexity and feature creep. Early designs and implementations used many features of the C++ language to an “extreme”. The team realized and began removing the complexity. Like most software development projects, this software suffers from complexity and feature creep. Early designs and implementations used many features of the C++ language to an “extreme”. The team realized and began removing the complexity.Solution Ongoing reviews Ongoing reviews

September 6, 2007Open Source Software Practice Lessons Learned: The Ugly Still need a stick Still need a stick Someone needs to monitor the daily process. Someone needs to monitor the daily process. XP is not everyone’s cup of tea XP is not everyone’s cup of tea Almost all bought into the process. A few tolerated it. Almost all bought into the process. A few tolerated it.

September 6, 2007Open Source Software Practice Open Source Resources

September 6, 2007Open Source Software Practice Quiz Monday – Reading List Free Culture, Lawrence Lessig, Free Culture, Lawrence Lessig, Available online at: Available online at: Sections to read Sections to read Introduction Introduction Chapter Four Chapter Four Free Software Free Society, Richard Stallman Free Software Free Society, Richard Stallman Available online at Available online at (HTML) (HTML) (PDF) (PDF) Sections to read Sections to read Chapter One: "The GNU Project" Chapter One: "The GNU Project" Chapter Two: "The GNU Manifesto" Chapter Two: "The GNU Manifesto" Chapter Three: "Free Software Definition" Chapter Three: "Free Software Definition" Chapter Four: "Why software should not have owners" Chapter Four: "Why software should not have owners" Open Source Software Licensing, Lawrence Rosen Open Source Software Licensing, Lawrence Rosen Available online at: Available online at: Sections to read Sections to read Chapter Two: "Intellectual Property" Chapter Two: "Intellectual Property""Intellectual Property""Intellectual Property" Chapter Four: "Taxonomy of Licenses" Chapter Four: "Taxonomy of Licenses""Taxonomy of Licenses""Taxonomy of Licenses" Retrieved from " Retrieved from "

September 6, 2007Open Source Software Practice Overview of the Software Process Bill Lorensen GE Research (retired)