doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 1 Project: IEEE Working Group for Wireless Personal Area Networks (WPANs) Submission Title: [SG3a Down Selection Process Discussion] Date Submitted: [12Nov02] Source: [Mary DuVal] Company [Texas Instruments] Address [P.O. Box , MS 847 Plano, TX USA 75086] Voice:[+1 (972) ], FAX: [+1 (972) ], Source: [Ian Gifford] Company [Consultant] Address [23 Kelshill Road, Chelmsford, MA USA 01863] Voice:[ ], FAX: [ ], Re: [02/385r0] Abstract:[SG3a Call For Proposal timelines and processes contribution.] Purpose:[The purpose of this submission is to baseline the current SG3a thinking on the CFP, Evaluation, Scoring, and Voting Processes to kickoff a Nov02 Plenary SG discussion topic on the possible CFP and Down Selection Processes. The next steps are to have the SG agree to a single process and a timeline.] Notice:This document has been prepared to assist the IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release:The contributor acknowledges and accepts that this contribution becomes the property of IEEE and may be made publicly available by

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 2 IEEE Working Group for Wireless Personal Area Networks (WPANs) SG3a Down Selection Process Discussion

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 3 Contents SG3a Work To Date Overview of the process Selection Process –Evaluation –Down Selection

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 4 Call for Applications, Proposals, and Down Selection Process (1) a summary of the 8 application presentations from the Study Group 3a call for applications and definition of the fundamental requirements implied by applications (document: IEEE P /104 Alternate PHY Technical Requirements – latest revision) (2) a list of expected performance and attribute criteria IEEE P /105 Alternate PHY Selection Criteria – latest revision The CFP Processes need to be elaborated, see next slide.

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 5 The CFP Process, what do we want? The figure to the right depicts the current state of the CFP Process thinking. We need to determine the processes we want to use. And then create the documents necessary to accomplish the call.

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 6 Proposed CFP Timeline Nov02 Session #21 –We should be able to release a CFP here (if the down selection process is chosen - the rest of the effort should be close) Jan03 Session #22 –Absolute latest release of CFP!!!!!!!!!!!! Mar03 Session #23 –Proposals should be heard, or this will be lightly attended and momentum will fade. May03 Session #24 –If too many proposal for one session, this should be the overflow. –We should spend part of this session discussing proposals presented, answering technical questions/concerns (i.e. put the diehard engineers in a room and let them verbally duke it out). –Spend time on conference calls discussing proposals to let everyone get comfortable (before and after this session). Jul03 Session #25 –Voting must occur here - 1st opportunity for automatic quorum after presentations Sep03 Session #26 –Start drafting process - proposals for draft improvements should be entertained at this time (i.e. now that we know the realm of the game, let the games begin) Nov03 Session #27...

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 7 SG3a Future Planning 2003 JJASONDJFMAMJJ D JFMAM The CFA opened on 11Dec01 and closed on 21Jan02. The CFP… A SON You are here CFP 2002 CFA D PAR Draft

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 8 Selection Process My suggestion is to discuss this process in two stages –Evaluation of proposal by group and –Actual down selection procedure History: –Separate Evaluation/down selection voting: TG3, TG4 –Evaluation process down selects (combined): –Down selection voting only (criteria summary created for information only): TGg (802.11)

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 9 Selection Process Observations/Suggestions Decoupling the evaluation from the down selection allows both needed parts to play in the process –Technical Investigation (and challenging) –Politics (in the actual voting) Separation allows –Technical community to attempt to fairly evaluate each proposal –Then voters to vote based on their needs/politics Technical evaluation allows –Observers to learn more about each proposal –Helps some on-the-fence voters to understand the value each proposal can bring to their future product plans If we don't factor in some politics, the market may go elsewhere to battle it out (it might even still go elsewhere, based on events) –Either way, it has to have room to play Based on group conversations, need exists for an evaluation period

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 10 Selection Process (Decision Point) Straw Poll: –Separate Evaluation/Down Selection Voting: –Evaluation is the Down Selection Voting (combined) : –Down Selection Voting only: –Abstain: The down selection subcommittee suggests separating the down selection into 2 parts (evaluation and down selection voting)

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 11 Evaluation Process Evaluation is really 2 discussions (or phases) –Criteria Importance Level Mandatory/Optional ABC –A: Mandatory requirement –B: Important desired requirement –C: A nice to have requirement Weighted values (0 – 10) –Scoring Pass/Fail Pugh Matrix –Better (+), Same, Worse (-) than a Baseline Solution Rating (0 – 5)

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 12 Evaluation Process Observations/Suggestions Evaluation must promote dialogue between members without taking forever to accomplish Criteria Importance Level –The down selection subcommittee suggests using the ABC rating method –ABC rating allows the proposals to be evaluated and discussed, without getting too bogged down in the extreme details –This area is where proposers have a tendency to twist the ratings in favor of their proposals, instead of in the best interest of the application needs –The application needs can be determined in the selection of ABC types and leave the arguing of meeting intent to the evaluation stage Scoring –Pugh Matrix requires determination of a baseline solution and levels of each criteria that allows proposals to be considered worse or better This will take at least another meeting (probably through January), possibly delaying the effort

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 13 Evaluation Process Options Matrix Scoring Criteria Importance Level Mandatory/ Optional ABC RatingWeighted Values Pass/Fail  Pugh Matrix  Rating (0-5) 

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 14 Any other suggested methods? –Phase 1 (Criteria Importance Level) –Phase 2 (Scoring) Evaluation Process

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 15 Evaluation Process (Decision Point) Criteria Importance Level ScoringStraw Poll Count 1Mandatory/OptionalPass/Fail 2Mandatory/OptionalRating (0-5) 3ABC RatingPass/Fail 4ABC RatingRating (0-5) 5Weighted ValuesPugh Matrix 6Weighted ValuesRating (0-5) 7Abstain

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 16 Down Selection Voting Procedure History: Processes used in the past in groups, such as TG3/TG4/TGg, have been methods of elimination voting. The methods used for these eliminations have been as follows: –Ranking vote, lowest rank is voted off the list (TG3) –Vote for desired proposal, lowest number of votes is off the list (TG4) –Two-step approach (TGg): Low hurdle vote - each voter casts a yes, no, or abstain vote for each proposal, proposals that did not obtain 25% were eliminated Vote for desired proposal, lowest number of votes is off the list

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 17 Down Selection Voting Procedure Observations/Suggestions Down selection should promote proposal merger conversations as selection proceeds –Like proposals are encouraged to join forces to allow voters to align based on key issues –Details will be revisited and hashed out during the drafting process TG4 started from TG3 process and improved on it

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 18 Down Selection Voting Procedure (Decision Point) Any other suggested methods? Straw Poll: –Ranking vote (lowest rank voted off): –Vote for desired proposal (lowest # of votes is off): –2 staged vote (eliminate low support proposals, vote for desired proposal):

doc.: IEEE /470r2 Submission November 2002 Mary DuVal, TI and Ian Gifford, ConsultantSlide 19 Next Steps Create Evaluation Components –Criteria Importance Levels Create Annex for inclusion in IEEE P /105 Alternate PHY Selection Criteria Work on designations next (Tuesday, Wednesday) –Scoring Document scoring method in Annex for inclusion in IEEE P /105 Alternate PHY Selection Criteria –Subcommittee will supply text for review Create an evaluation form to be used by voters during evaluation stage –This can be completed in January (if needed) Down Selection Procedures –Down selection subcommittee will create flow diagram for discussion in a session on Thursday based on results of included straw poll Down Selection committee looking for any interested volunteers – see Mary or Ian to join (Tuesday night, 8 pm – Garden Isle 4)