Software Tools Group & Release Process Alain Roy Mine Altunay

November 20, 2008 OSG Software Providers Meeting Welcome! We are really happy you could attend Main goal of this meeting: This meeting has LOTS of time for discussion. Please discuss! 2 Improve OSG’s interaction with and understanding of our software providers

November 20, 2008 OSG Software Providers Meeting Achieving our goal 1.We tell you about our efforts to improve and formalize our software processes 2.You give us feedback on these efforts 3.You tell us about where you’re going with your software 3

November 20, 2008 OSG Software Providers Meeting So what are our efforts? The all new Software Tools Group Improving our software release process Let’s talk through both of them in this talk. 4

November 20, 2008 OSG Software Providers Meeting The OSG Software Tools Group The Software Tools Group (STG) oversees the development and delivery of all software tools and utilities owned or distributed by the OSG.  We oversee development of OSG-created software.  No, we aren’t overseeing development of externally developed software  We oversee delivery of all software we ship.  Mostly software we don’t develop 5

November 20, 2008 OSG Software Providers Meeting The STG has responsibilities Responsibilities to stakeholders (OSG users) Responsibilities to OSG Responsibilities to you (software providers) 6

November 20, 2008 OSG Software Providers Meeting STG responsibilities to stakeholders We will collecting requests/requirements for software to be included in the OSG Software Stack. Software developed and distributed should benefit stakeholders. 7

November 20, 2008 OSG Software Providers Meeting STG responsibilities to OSG Good, well-documented, software life-cycle. Provide requirements, design, implementation, testing, documentation and deployment. Allocate OSG’s efforts for specific software tools and services based on criticality and benefit to OSG. Maintain a schedule for timely delivery of the software tools. Setting policies for OSG software stack, including how the OSG Software Stack is released, versioned, tested, and deployed. 8

November 20, 2008 OSG Software Providers Meeting STG responsibilities to you (Software Providers) We will help get your software distributed to OSG users. We will maintain a relationship with you. We will coordinate with you about release dates. Develop policies and procedures on how software is included in OSG software stack We want to make this as smooth as possible: your feedback is desired. 9

November 20, 2008 OSG Software Providers Meeting STG interactions with providers STG works software providers Three kinds of software providers 1.OSG software: OSG pays for some or all of the development  Bestman  Gratia  Panda  … 2.Close Relationship  Condor  dCache  EGEE-owned software (VOMS, BDII)  Globus  Xrootd  …. 3.Not close relationship  Apache  MySQL  Squid  … 10

November 20, 2008 OSG Software Providers Meeting 1. OSG-owned software Hopefully we don’t have much OSG- owned software  We strongly prefer to get software from external software providers! STG will appoint technical lead and team of developers (STG won’t develop software) STG will request and review yearly plans STG will periodically review progress 11

November 20, 2008 OSG Software Providers Meeting 2. Software providers with close relationship OSG assigns liaison to each provider  Not the only source of communication, but well-known point of contact. Will request periodic meetings to:  Understand future development  Request features or report bugs 12

November 20, 2008 OSG Software Providers Meeting 2. Software providers with close relationship 2 modes of operations:  Development  Maintenance During Development: that is related to OSG (perhaps requested by OSG)  Frequent contact, either via liaison or directly  Sharing expectations, functionality requirements  Software development plans, update on development status 13

November 20, 2008 OSG Software Providers Meeting 2. Software providers with close relationship Maintenance mode (no new software development)  Less frequent contact  How little? Once a year or twice a year?  We expect you to ping us when you make new plans or changes in software  How far in advance ?  How frequently do you make your plans?  Should we call you or you will call us?  Routine support and bug fixes and etc 14

November 20, 2008 OSG Software Providers Meeting 3. Other software providers No close relationship.  We use bug tracking systems and mailing lists, just like the rest of the world. We identify candidate software based on stakeholder needs. We maintain information about who maintains the software. We file bug reports as needed. 15

November 20, 2008 OSG Software Providers Meeting What we hope for from you When we add software to the software stack, we consider:  License: Can we distribute it? Can we modify it?  Robustness: Is it stable? How is it tested? How can we test it?  Security: When do we learn of security problems? How? Can we hear about them befor eyou release fixes? How do you release security fixes? (Talk on this later!)  Feature Requests: Can we make them?  Support: How do we get it? How much can you provide?  Future: How can we learn about your software plans? 16

November 20, 2008 OSG Software Providers Meeting How to capture our mutual understanding? We have almost nothing written down so far What is best way to capture our mutual understanding?  Written MOUs?  Agreed interactions and expectations  Would that be feasible for you?  Informal, but written agreement?  What should go into the agreements? 17

November 20, 2008 OSG Software Providers Meeting How does this fit into software release process? Software Tools Group’s work is one part of the software release process. The whole process fits together  Gathering of requirements  Gathering of software  Validation and testing  Delivery of software Picture on next slide  18

November 20, 2008 OSG Software Providers Meeting Software Release Process 19 Requirement Gathering Process Software Tools Group Process Validation & Integration Process Software Delivery process OSG Software Release Planning Process (People who watch over the above process) Software providers

November 20, 2008 OSG Software Providers Meeting Requirements Gathering Requirements come from:  OSG VOs/Users. Collected by VO contact or VO users group  Internal OSG needs 20 Reqs STGTestingDeliver Provide rs

November 20, 2008 OSG Software Providers Meeting STG/Provider Interaction 21 Reqs STG TestingDeliver Provide rs STG SP Question: Today most folks do not have clear agreements. Can we set them up? Focus on support and security? STG SP Req Agreement here is smaller. Iterative process to work out requests. Initial Agreement Initial Agreement New Work Agreement Req

November 20, 2008 OSG Software Providers Meeting Termination of agreements We will tell you if we no longer need your software, or if we want to drop a request. If you can’t continue support, we hope you will tell us. 22 STG SP

November 20, 2008 OSG Software Providers Meeting Testing Validation testbed (Small)  Only sites (not VOs) test  Installation works  Small tests pass Integration testbed (Larger)  Installed on more sites  Multiple VOs verify their software works properly Specific testing  Like integration testbed, but more focused testing  Example: ATLAS tested LFC tool before release 23 ReqsSTG Testing Deliver Provide rs

November 20, 2008 OSG Software Providers Meeting Delivery: Software cycle Old software cycle  New major release every 6-8 months  Very few intermediate releases New software cycle  Decreased frequency of major releases (once a year?)  More frequent and significant intermediate releases “Hey Alain, when is the next big OSG release for me to plan for?”  I’m not sure… can we do it as an incremental release? 24 ReqsSTGTesting Deliver Provide rs

November 20, 2008 OSG Software Providers Meeting Incremental Updates Late 2008/January 2009:  Update to Globus  Update to major new GIP  Update to authorization components  PRIMA, GUMS, glexec, VOMS Exact schedule depends on testing 25

November 20, 2008 OSG Software Providers Meeting OSG 1.2: The next big release Major goal (probably): Packaging  Right now incremental updates are hard  Hard to add or subtract a platform  Hard to apply updates  Next major release to fix these problems Date: Spring 2008? 26

November 20, 2008 OSG Software Providers Meeting Questions to stimulate discussion How should support ticketing work?  Interaction with our ticketing system?  Our ticket system vs. yours Flow of questions:  Users  GOC  Provider  Users  Provider (cc: OSG) How do you want to get details of users problem?  Logs?  Do you have enough information?  Do you want login to the users system? (VNC?) What should we do differently for critical bugs? How much can you help us test the software?  Should we integrate you into the validation/integration testing? How?  Can we do software certification? Would you help us? What is the best way to learn your plans?  Meet at the OSG all-hands meeting?  Meet you at home twice a year?  Regular phone conferences?  Joint with other providers (like today), or joint meetings? 27

November 20, 2008 OSG Software Providers Meeting More Questions When is the best time to send you requests?  How do you usually collect requests/requirements? How do you feel about your current interaction with OSG? How do you feel about your interaction with other software providers  does your software interact with other software?  Are you affected by changes in other software components in the stack?  How do you hear about the changes?  How do you wish to hear about the changes? How do you feel about your interaction with stakeholders?  Would you prefer more feedback, usability etc, new features  Should OSG be more active in bringing you together?  Would you have time and effort ? 28