University of Southern California Center for Systems and Software Engineering 1 Architecture Review Boards Barry Boehm 10/14/2009

University of Southern California Center for Systems and Software Engineering 2 AT&T/Lucent ARB Concept -Overview -Results -Recommendations USC CS577 ARB Concept -Participants -Procedures -Results Outline

University of Southern California Center for Systems and Software Engineering 3

University of Southern California Center for Systems and Software Engineering 4

University of Southern California Center for Systems and Software Engineering 5

University of Southern California Center for Systems and Software Engineering 6

University of Southern California Center for Systems and Software Engineering 7

University of Southern California Center for Systems and Software Engineering 8

University of Southern California Center for Systems and Software Engineering 9

University of Southern California Center for Systems and Software Engineering 10

University of Southern California Center for Systems and Software Engineering 11

University of Southern California Center for Systems and Software Engineering 12

University of Southern California Center for Systems and Software Engineering 13

University of Southern California Center for Systems and Software Engineering 14

University of Southern California Center for Systems and Software Engineering 15

University of Southern California Center for Systems and Software Engineering 16

University of Southern California Center for Systems and Software Engineering 17

University of Southern California Center for Systems and Software Engineering 18

University of Southern California Center for Systems and Software Engineering 19

University of Southern California Center for Systems and Software Engineering 20

University of Southern California Center for Systems and Software Engineering 21

University of Southern California Center for Systems and Software Engineering 22

University of Southern California Center for Systems and Software Engineering 23

University of Southern California Center for Systems and Software Engineering 24 AT&T/Lucent ARB Concept -Overview -Results -Recommendations USC CS577 ARB Concept -Participants -Procedures -Results Outline

University of Southern California Center for Systems and Software Engineering 25 USC CS577 ARB Participants Project Team -Everybody presents something Reviewers -Clients -Instructors and TA’s -Industry participants 80 minute time slots

University of Southern California Center for Systems and Software Engineering ARB/milestones for two-semester team FCR ARB: October 19 th to 23 rd –Based on preliminary FC package –Focus on FCR success criteria DCR ARB: November 30 th to December 4 th –Based on draft DC package –Focus on DCR success criteria 26

University of Southern California Center for Systems and Software Engineering ARB/milestones for one-semester team DCR ARB: October 19 th to 23 rd –Based on DC package –Focus on DCR success criteria CCD: November 11 th –Core Capability Drive-through –Client(s) will have hands-on experience on your core capabilities TRR: November 23 rd –Informal review with TA; readiness for transition OCR: November 30 th to December 4 th –Based on AsBuilt package 27

University of Southern California Center for Systems and Software Engineering ARB Review Success Criteria 28 FCR For at least one architecture, a system built to arch. will: Support the Ops Concept Satisfy the Requirements Be faithful to the Prototype Be buildable within the budgets and schedules in the Plan Show viable business case Key stakeholders committed to support Foundations (nee Architecting or Elaboration) Phase (to DCR) DCR For the selected architecture, a system built to the arch. will: Support the Ops Concept Satisfy the Requirements Be faithful to the Prototype Be buildable within the budgets and schedules in the Plan All major risks resolved or covered by risk management plan Key stakeholders committed to support full life cycle

University of Southern California Center for Systems and Software Engineering ARB Review Success Criteria 29 TRR / OCR Show value Product works as expected (or not with noted exceptions) Product will help users do job Show quality development e.g. relevant parts of your IOC documentation Process Show sustainability e.g. support requirements/plans Transition plan & status Show confidence that product is/will be ready to be used e.g. Transition plan & status See also value Determine whether client needs anything further to ensure successful Transition and Operation Changes in priorities for remaining features? Changes being made to operational procedures? More materials needed for training? Changes in system data or environment we should prepare for? Anyone else who should experience CCD? CCD

University of Southern California Center for Systems and Software Engineering Team Preparation for ARB Reviews Week-1 Within-team Dry run of presentations and demo Further dry runs as necessary ARB Week ARB Presentation and discussion Follow-up team discussions, client discussions Week+1 Monday: FC packages due Monday: DC packages due 30

University of Southern California Center for Systems and Software Engineering FCR ARB Session Outline Architected Agile Team (x,y): (presentation time, total time) (5, 5) Remote Team Member(s) (jointly) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions (10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; desired capabilities and goals (10,10) Prototype. Most significant capabilities [buying information](especially those with high risk if gotten wrong) (5, 10)Requirements. Most significant requirements (5, 10)Architecture. Top-level physical and logical architecture; status of NDI/reuse choices (5, 10)Life Cycle plan. Life cycle strategy; focus on Foundations phase; key stakeholder responsibilities; project plan, resource estimation (5, 10)Feasibility Evidence. Business case (beginnings, including benefits analysis); NDI analysis results; major risks; (5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix (10)Things done right; issues to address (Instructional staff) Plan on 2 minutes per briefing chart, except title Each chart MUST have information specific to your project 31

University of Southern California Center for Systems and Software Engineering FCR ARB Session Outline NDI/ NCS Team (2 semesters) (x,y): (presentation time, total time) (5, 5) Remote Team Member(s) (jointly) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions (10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; core capabilities, constraints and goals (5, 5) WinWin Agreements. Agreed Win conditions in each category (10,10) Prototype. Most significant capabilities, NDI/NCS integration (5, 10)Architecture. Top-level physical and logical architecture; (5, 10)Life Cycle plan. Life cycle strategy; focus on Foundations phase; key stakeholder responsibilities; project plan, resource estimation (10, 15)Feasibility Evidence. NDI/NCS alternatives, NDI/NCS evaluation & analysis results; Business case (beginnings, including benefits analysis); major risks; Capability and LOS feasibility evidence (5, 5)QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix (10)Things done right; issues to address (Instructional staff) Plan on 2 minutes per briefing chart, except title Each chart MUST have information specific to your project 32

University of Southern California Center for Systems and Software Engineering DCR ARB Session Outline NDI/ NCS Team (1 semester) (x,y): (presentation time, total time) (5, 5) Remote Team Member(s) (jointly) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions (10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; core capabilities, constraints and goals (5, 5) WinWin Agreements. Agreed Win conditions in each category (10,10) Prototype/ Product Demo. Most significant capabilities, NDI/NCS integration (5, 5)Architecture. Top-level physical and logical architecture; (if applicable) (5, 10)Life Cycle plan. Life cycle strategy; focus on Development phase & transition increment; key stakeholder responsibilities; project plan; resource estimation (10, 15)Feasibility Evidence. NDI/NCS alternatives, NDI/NCS evaluation & analysis results; Business case (beginnings, including benefits analysis); major risks; Capability and LOS feasibility evidence (5, 5)QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix (5, 5)Acceptance Test Plan and Cases (10)Things done right; issues to address (Instructional staff) Plan on 2 minutes per briefing chart, except title Each chart MUST have information specific to your project 33

University of Southern California Center for Systems and Software Engineering Team’s strong points & weak points List at least one item for each of the following –List your team’s strong points Operational view Technical view –List your team’s weak points Operational view Technical view –Identify specific technical concerns & possible solutions –Identify operational risks & possible mitigation Sources of observations –Team activities, package evaluation, winwin negotiation, and etc. 34

University of Southern California Center for Systems and Software Engineering Defect Identification Review For each document section, UML model, and etc. identify the following –type of review you used (peer review, agile artifact review, and etc. ) –Other form of defect identification, e.g., grading, client feedback, etc. Current Defect Injection and Removal Matrix –Current, total defect information from your progress report 35

University of Southern California Center for Systems and Software Engineering ARB Session Outline DCR Similar format to FCR, different focus: –Less time for OCD, Prototype –More time for Architecture, Plans –Details TBD based on FCR ARB experience General rule on focus: emphasize your projects high risk areas –At FCR (in most cases) this will involve establishing the operational concept (including system analysis) –At DCR (in most cases) this will involve the system design and development plan (especially schedule) 36

University of Southern California Center for Systems and Software Engineering Results To Date 37 * - Poor performance - Poor team management - Poor communication (within team and with client)

University of Southern California Center for Systems and Software Engineering ARB Packages If you would like to have your ARB presentation and FC/DC package printed for you at the ARB meeting, – the documents (in a zip file) to 24 hours in advance, unless your ARB is on Monday, in which case, the documents 5 hours in advance. –Otherwise, bring 5 copies of your ARB presentation and 2 copies of your FC package to the meeting. 38

University of Southern California Center for Systems and Software Engineering Demos in ARB For those teams doing a live demo in the ARB meeting, please include screenshots of your demo in your presentation for your IV&Vers to see the demo. 39

University of Southern California Center for Systems and Software Engineering ARB Presentation slides Upload your ARB presentation slides (before your ARB) on your team website for off-campus students. 40

University of Southern California Center for Systems and Software Engineering Web-ex & Teleconf Off-campus student who can not attend the ARB in-person, will be connecting through web-ex

University of Southern California Center for Systems and Software Engineering Past FCR Experiences and General Guidelines 42

University of Southern California Center for Systems and Software Engineering Outline FCR ARB 2008 Feedback Summary Examples of Good and Bad Practices seen at ARBs ARB Chartsmanship & Presentation 43

University of Southern California Center for Systems and Software Engineering Overall FCR Feedback – Fall 2008 Generally done well: presentations, time management, client rapport Reconcile FCR content with pass/fail criteria Define key terms, acronyms in glossary If you’re deferring or skipping a normally- included artifact, explain why (e.g. COTS internals unavailable) 44

University of Southern California Center for Systems and Software Engineering OCD Feedback – Fall 2008 Generally done well: Organizational goals, Operational concept, System boundary and organizational environment. Benefits chain needs rework –Added stakeholders: users, clients, developers, IV&Vers, database administrators, maintainers, interoperators, suppliers –Assumptions are about environment not about outcome Organization Goals are Benefits Chain End Outcomes, not project Initiative contributions Business Workflow – not technical workflow Prototype and System are NOT the same 45

University of Southern California Center for Systems and Software Engineering Prototype Feedback – Fall 2008 Generally done well: GUI Prototypes, Good understanding of client’s needs Prototype all high-risk elements, not just GUI’s –COTS interoperability, performance, scalability Use user-friendly terms (John Doe, 22 Elm St.) not abstractions (N.,A or test test test) Identify end users and try to get feedback from end users Focus on important and high priority requirements (initially) 46

University of Southern California Center for Systems and Software Engineering SSRD Feedback – Fall 2008 Generally done well: Project and Capability requirements, OCD-SSRD traceability Prioritize all the requirements Propagate LOS goals from OCD into SSRD or drop LOS requirements from SSRD (and SSAD) Distinguish between client imposed requirements (SSRD) and developer choice solution (SSAD) Make sure all requirements are testable Qualify “24/7” Availability with noted exceptions 47

University of Southern California Center for Systems and Software Engineering SSAD Feedback – Fall 2008 Generally done well: Overall views Follow UML conventions (arrows, annotations, etc.) Follow the SSAD guideline Read the exit criteria for the milestone carefully 48

University of Southern California Center for Systems and Software Engineering LCP Feedback – Fall 2008 Generally done well: Overall strategy, roles and responsibilities Fill in 577b TBDs Full elaboration phase plan COCOMO drivers differ as per the module Add IV&Vers interactions to the Gantt Chart Add maintainer’s responsibilities 49

University of Southern California Center for Systems and Software Engineering FED Feedback – Fall 2008 Generally done well: Business case framework, risk analysis Specify LOS feasibility plans Include training, operations, maintenance opportunity costs/ effort Drop developers hours as cost Try to quantify benefits, show return on investment Change ROI to yearly vs. by semester Distinguish one-time from annual costs in business case Benefits start in 2007 Requirements traceability: Replace the generic requirements traceability table. Show traceability between items not sections Co-ordinate risk in LCP and FED Elaborate process rationale 50

University of Southern California Center for Systems and Software Engineering Examples of good and bad practices seen at ARBs In LCP: –Error in LCP document in how you present your schedule of activities –Need to make sure it is easy to read and understandable 51

University of Southern California Center for Systems and Software Engineering Bad example: Gantt Chart – Construction Phase 1 st Iteration 52

University of Southern California Center for Systems and Software Engineering Good example - Schedule for Test Activities (Iteration 1) 53 ActivityDate Due Finalize design of required database entities and relations3/5/08 Define test packages (plans, cases and criteria)3/7/08 System flow and training guide3/8/08 Language training3/9/08 Implement Transaction Reconciler -> Implement CD transaction reconciliation functionality3/10/08 -> CD Unit Testing3/11/08 -> Implement RJ transaction reconciliation functionality3/14/08 -> RJ Unit Testing and Regression Testing3/15/08 Implement Report Generator3/17/08 RG Unit Testing and Regression Testing3/18/08 Implement Notification Generator3/20/08 NG Unit Testing and Regression Testing3/21/08 Implement Manual Exception Adjuster3/23/08 MEA Unit Testing and Regression Testing3/24/08 Integration Testing3/27/08 Core Capability Drive-thruTBD

University of Southern California Center for Systems and Software Engineering Bad example: Breakeven Analysis 54 ROI at year 0 should always start at -1

University of Southern California Center for Systems and Software Engineering Good example: Business Case - ROI Benefit: –Manpower: Two employees reconciling only 80% of what they originally were reconciling: $45,132 *.8 = $35, /year –Accounting Saved: $45,132 - $35, = $10, –Benefit = $10, Cost: –Maintenance Cost: $1,375 / year –Development Costs to ICT = $ $ = $ –Transition Costs = $ –Cost = $ ($1,375 /year) ROI = Benefit – Cost / Cost = (($10, / year) – ($ ($1,375 /year)) / ($1,375 /year) = (($ / year) ) / ( ($1,375 / year)) = 0.76 for first year = every year after 55

University of Southern California Center for Systems and Software Engineering Good example: Business Case Breakeven Analysis Current System Employee Hours: –2 employees * ((52 weeks * 50% of time reconciling) * 40 hours/week) = 2080 hrs Proposed System Employee Hours: –2080 hrs * 80% of work = 1664 hrs –2080 hrs – 1664 hrs = 416 hrs savings yearly Cost in time: –52 hours for maintenance / year –95 ICT total participation hours –95 hrs + 52 hrs / year (X)– 416 hrs /year (X)= 0 –95 – 364 X = 0 –X =.26 years = Break even time after project delivery 56

University of Southern California Center for Systems and Software Engineering Good way to present your risks : Approach –Risk 1: Personnel Turn-over New team members - spending time reading and understanding the project. A returning team member - explaining and providing information for new team members. –Risk 2: Team Size Reducing Re-negotiating and re-prioritizing priorities of system requirements with the client –Risk 3: Client Availability Asking for an appointment with the client early 57

University of Southern California Center for Systems and Software Engineering ARB Chartsmanship & Presentation Do not repeat the ICM Guidelines Do not sweat the small stuff Use audience-based terminology NEVER read a slide’s contents –Paraphrase or hit only the high points –Practice, so it flows well, BEFORE your dry run Assume 2 minutes presentation time per chart –After timed dry run practice Don’t repeat previous speakers’ material –OK to refer to it Do dry runs with outsider audience 58