Architecture Review Boards Foundations Commitment Review Including Past Experiences October 22, 2012

Outline AT&T/Lucent ARB Concept -Overview -Results -Recommendations USC CS577 ARB Concept -Participants -Procedures

[CSCI 577a FCR] [CSCI 577a DCR and 577b RDCR]

Outline AT&T/Lucent ARB Concept -Overview -Results -Recommendations USC CS577 ARB Concept -Participants -Procedures

The Incremental Commitment Spiral Model (ICSM) 4 Key Principles: Stakeholder value-based system definition and evolution Incremental commitment and accountability Concurrent system and software definition and development Evidence and risk-based decision making First of all, ICSM or the Incremental Commitment Spiral Model, which is a process model. One of the most important ingredient of the ICSM is the opportunity and risk that will determine course of action in the development.

ICSM for 24-week e-services projects 26

USC CS577 ARB Participants Project Team -Everybody presents something Reviewers -Clients -Instructors and TA’s -Industry participants 80 minute time slots

ARB/milestones for two-semester team FCR ARB: October 29th to November 2nd Based on preliminary FC package Focus on FCR success criteria DCR ARB: December 3rd to 7th Based on draft DC package Focus on DCR success criteria

ARB/milestones for one-semester AA team FCR/DCR ARB: October 29th to November 2nd Based on DC package Focus on DCR success criteria CCD: November 19th Core Capability Drive-through Client(s) will have hands-on experience on your core capabilities TRR: November 26st Informal review with TA; readiness for transition OCR: November 28th to December 2nd Based on AsBuilt package

ARB Review Success Criteria FCR For at least one architecture, a system built to arch. will: Support the Ops Concept Satisfy the Requirements Be faithful to the Prototype(s) Be buildable within the budgets and schedules in the Plan Show viable business case Most major risks identified and resolved or covered by risk management plan Key stakeholders committed to support Foundations (nee Architecting or Elaboration) Phase (to DCR) DCR For the selected architecture, a system built to the arch. will: Be buildable within the budgets and schedules in the Plan All major risks resolved or covered by risk management plan Key stakeholders committed to support full life cycle

Commitment Review Success Criteria CCD TRR / OCR Determine whether client needs anything further to ensure successful Transition and Operation Changes in priorities for remaining features? Changes being made to operational procedures? More materials needed for training? Changes in system data or environment we should prepare for? Anyone else who should experience CCD? Show value Product works as expected (or not with noted exceptions) Product will help users do job Show quality development e.g. relevant parts of your IOC documentation Process Show sustainability e.g. support requirements/plans Transition plan & status Show confidence that product is/will be ready to be used e.g. Transition plan & status See also value

Team Preparation for ARB Reviews Week-1 Within-team Dry run of presentations and demo Further dry runs as necessary ARB Week ARB Presentation and discussion Follow-up team discussions, client discussions Week+1 Monday: FC packages due Monday: DC packages due

FCR ARB Session Outline Architected Agile Team (x,y): (presentation time, total time) (5, 5) Remote Team Member(s) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions; S/P Engineer observations (10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; desired capabilities and goals (10,10) Prototype. Most significant capabilities [buying information](especially those with high risk if gotten wrong) (5, 10) Requirements. Most significant requirements (5, 10) Architecture. Top-level physical and logical architecture; status of NDI/reuse choices (5, 10) Life Cycle plan. Life cycle strategy; focus on Foundations phase; key stakeholder responsibilities; project plan, resource estimation (5, 10) Feasibility Evidence. Business case (beginnings, including benefits analysis); NDI analysis results; major risks; (5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix (10) Things done right; issues to address (Instructional staff) Plan on 2 minutes per briefing chart, except title Each chart MUST have information specific to your project

FCR ARB Session Outline NDI/ NCS Team (2 semesters) (x,y): (presentation time, total time) (5, 5) Remote Team Member(s) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions; S/P Engineer observations (10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; core capabilities, constraints and goals (5 , 5) WinWin Agreements. Agreed Win conditions in each category (10,10) Prototype. Most significant capabilities, NDI/NCS integration (5, 10) Architecture. Top-level physical and logical architecture; (5, 10) Life Cycle plan. Life cycle strategy; focus on Foundations phase; key stakeholder responsibilities; project plan, resource estimation (10, 15) Feasibility Evidence. NDI/NCS alternatives, NDI/NCS evaluation & analysis results; Business case (beginnings, including benefits analysis); major risks; Capability and LOS feasibility evidence (5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix (10) Things done right; issues to address (Instructional staff) Plan on 2 minutes per briefing chart, except title Each chart MUST have information specific to your project

DCR ARB Session Outline NDI/ NCS Team (1 semester) (x,y): (presentation time, total time) (5, 5) Remote Team Member(s) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions; S/P Engineer observations (10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; core capabilities, constraints and goals (5 , 5) WinWin Agreements. Agreed Win conditions in each category (10,10) Prototype/ Product Demo. Most significant capabilities, NDI/NCS integration (5, 5) Architecture. Top-level physical and logical architecture; (if applicable) (5, 10) Life Cycle plan. Life cycle strategy; focus on Development phase & transition increment; key stakeholder responsibilities; project plan; resource estimation (10, 15) Feasibility Evidence. NDI/NCS alternatives, NDI/NCS evaluation & analysis results; Business case (beginnings, including benefits analysis); major risks; Capability and LOS feasibility evidence (5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix (5, 5) Acceptance Test Plan and Cases (10) Things done right; issues to address (Instructional staff) Plan on 2 minutes per briefing chart, except title Each chart MUST have information specific to your project

Specfics for DEN students Team’s strong points & weak points List at least one item for each of the following List your team’s strong points Operational view Technical view List your team’s weak points Identify specific technical concerns & possible solutions Identify operational risks & possible mitigation Sources of observations Team activities, package evaluation, WinWin negotiation, and etc.

Specfics for DEN students System/Project Engineer at ARB WinWin Shaping Status: Open WinCs: Identified Agreed WinCs with Issues & without Issues Overall Project Evaluation consideration All SCS(s) CRACKness Complexity Precedentedness (for team!) Communication & use of communication tools between on-campus team, S/PE and client Skills/needs match Knowledge/experience mis-matches

QFP – Defect Identification Review For each document section, UML model, and etc. identify the following type of review you used (peer review, agile artifact review, and etc. ) Other form of defect identification, e.g., grading, client feedback, etc. Current Defect Injection and Removal Matrix Current, total defect information from your progress report

ARB Session Outline DCR Similar format to FCR, different focus: Less time for OCD, Prototype More time for Architecture, Plans Details TBD based on FCR ARB experience General rule on focus: emphasize your project's high risk areas At FCR (in most cases) this will involve establishing the operational concept (including system analysis) At DCR (in most cases) this will involve the system design and development plan (especially schedule)

Results To Date * Reasons: - Poor performance Poor team management Poor communication (within team and with client)

ARB Packages If you would like to have your ARB presentation and FC/DC package printed for you at the ARB meeting, email the documents (in a zip file) to csci577@usc.edu 24 hours in advance, unless your ARB is on Monday, in which case, email the documents 5 hours in advance. Otherwise, bring 4 copies of your ARB presentation and 2 copies of your FC package to the meeting.

Demos in ARB For those teams doing a live demo in the ARB meeting, please include screenshots of your demo in your presentation for your IV&Vers to see the demo in case video connection is a problem for reviewers to make notes on

ARB Presentation slides Upload your ARB presentation slides (before your ARB) on your team website for off-campus students

Webex & Teleconf Off-campus students who can not attend the ARB in-person, will be connecting through Webex

Past FCR Experiences and General Guidelines

Outline FCR ARB 2011 Feedback Summary Examples of Good and Bad Practices seen at ARBs ARB Chartsmanship & Presentation

Overall FCR Feedback – Fall 2011 Generally done well: presentations, time management, client rapport Reconcile FCR content with ARB Success Criteria Define key terms, acronyms in SID glossary If you’re deferring or skipping a normally-included artifact, explain why (e.g. COTS internals unavailable), and note in SID-document tailoring section Occassional order changes in presentation without telling us in a modified agenda at the beginning Role of primary DEN/remote student often mis-stated Only one IS System/Project Engineer (S/PE) Includes IIV&V (also done by second DEN/remote student) Includes shaping, and re-shaping throughout semester(s)

Overall FCR Feedback 2011 - 2 When asked a question: Give the answer in brief, this will help your time management and the Review Board will get the desired information Do NOT answer back while Review Board attempts to provide guidance Many had poor time management that indicated that presentation(s) had NOT been practiced Occassional pointing at laptop screen, not projected image (even better over Webex, use mouse) Very occasionally, slides with NO value added 48 48

OCD Feedback (1) System boundary diagram Generally done well: Organizational goals, Operational concept, System boundary and organizational environment. Some benefits chains need rework: Added stakeholders: users, clients, developers, IIV&Vers, database administrators, maintainers, interoperators, suppliers Assumptions are about environment not about outcome Involvement/use of system before system is built Some organization goal(s) are Benefits Chain end outcomes System boundary diagram If you are using the component in/for your system, remove it from environment, e.g. PHP, .NET framework. 49 49

OCD Feedback (2) Organization Goals Are Benefits Chain End Outcomes (or maybe a subset) Are NOT project Initiative contributions Identify Levels of Service properly 100% availability, 100% reliability - not feasible! Make sure you can measure LOS goals Prototypes and System are NOT the same (usually) Business Workflow Use activity-type diagram Illustrate business activities Not technical/system activity May not even “see” system explicitly 50 50

Prototype Feedback Generally done well: GUI Prototypes, Good understanding of client’s needs Prototype all high-risk elements, not just GUI’s COTS interoperability, performance, scalability Use user/client-friendly terms “John Doe, 22 Elm St.” not generic substitutions like “Name1, Addr1” Use as an opportunity to gather more information and/or examples Identify end users and try to get feedback from end users Focus on important and high priority requirements (initially) 51 51

SSRD Feedback Generally done well: Project and Capability requirements, OCD-SSRD traceability Prioritize all the requirements Propagate LOS goals from OCD into SSRD or drop LOS requirements from SSRD (and SSAD) Distinguish between client imposed requirements (SSRD) and developer choice solution (SSAD) Make sure all requirements are testable Qualify “24/7 Availability" with exceptions Update the new requirements in WinBook tool There is no such thing as an “implicit requirement” 52 52

SSAD Feedback Generally done well: Overall views Follow UML conventions (arrows, annotations, etc.) Generalization of actors Uncommon mistakes in use-case diagrams Two actors-one use case (means BOTH present) Arrow direction for <extends> or <includes> Devil is in the detail; simple is the best Only two teams had an adequate start on Information & Arctifacts Diagram Read the exit criteria for the milestone carefully 53 53

SSAD Feedback Generally done well: Overall views Follow UML conventions (arrows, annotations, etc.) Generalization of actors Uncommon mistakes in use-case diagrams Two actors-one use case (means BOTH present) Arrow direction for <extends> or <includes> Devil is in the detail; simple is the best Only two teams had an adequate start on Information & Arctifacts Diagram Read the exit criteria for the milestone carefully 54 54

Team 3 Signup New User

Team 3 Information & Artifacts

Team 14 Information & Artifacts

LCP Feedback - 1 Generally done well: overall strategy, roles and responsibilities Too many 577b TBDs Identify required skills for NN new team member(s) (577b; if needed to meet "team size") Show (concentrate on) your future plan; not the past Full Foundations [nee Elaboration] phase plan Don’t plan ONLY for documentation Include Modeling Include Prototyping; coding; executable architecture 58 58

LCP Feedback - 2 COCOMO drivers Often differ per the module (type) PMATs rationale was often wrong: CS577 projects' process maturity should be between 2 and 3 NOBODY did the detailed check list in the SwCEwCII book! Some driver rationales were "ridiculous" Add DEN Student interactions to Gantt Chart IIV&V System/Project Engineer (includes Shaping) Add maintainer’s responsibilities 59 59

FED Feedback Generally done well: Business case framework, risk analysis Specify LOS feasibility plans Include training, operations, maintenance, opportunity costs/effort Few had developers hours as cost (which is correct) Try to quantify benefits, show return on investment Change ROI to reflect on-going costs (possibly savings) Distinguish one-time from annual costs in business case Benefits start in mid 2010 (go at 6 months granularity); Costs start mid 2009 Elaborate process rationale Complete section 6 – COTS Analysis 60 60

SID Feedback Requirements traceability Update Glossary! OCD  WinC  SSRD  SSAD Update every time there is a change Update Glossary! Glossary MUST have all system/project specific terms Non-standard (unusual uses) 61 61

QFP Generally done well Some missing traceability injection-removal matrix Some seemed to try to "snow us with data", not present just a quick summary Did NOT specify type of "Peer Review" Pass around or Buddy Check (NOT desirable: no record of concerns) Agile Artifact Review Agile Internal/Informal (Role Based) Peer Review Will be expected to say lots more to say at DCR! 62 62

Things to improve Presentation – communication skill One word wrong could lead to billion $ loss. Practice in front of others Be concise and precise Consistencies among each artifact Team work vs. integrated individual works Prepare your client: Tell them what an ARB is (use agenda, success criteria) Tell them what to expect from ARB Time management Get in and set-up ASAP Have documents & client present 63 63