0 For Government Use Only Central Performance Metric: Scenario and Tasks Scenario ……..... ….. ….. … Test Questions ……..... ….. ….. … T1. Find time/space slots on Monday for tutorials T1 - T10, given speakers, room availability and attendance estimates. Conf Plan Worksheet Pg1 Venue: ______ Max Capacity: _____ Min Capacity: ______ Attendance est: _______ No. of rooms: ______ Location: ______ Miles from Campus: ____ Answers & Scoring Methods Conf Plan Worksheet Pg2 Total Cost: _____ Refreshment: _____ Rent: _____ AV Rental: _____ Equip Rental: _____ Lunch Catering: _____ Dinner Catering: _____ Reimbursements: _____ Conf Plan Worksheet Pg3 RADAR Evaluation Framework

1 For Government Use Only Scoring Quality of conference schedule –Score completeness of plan + how well plan satisfies applicable constraints –Score at multiple granularities –Quality of individual subevent schedules –Quality of schedule as a whole –Prioritized subevents –Priorities may derive from general principles of conference planning, specific priorities of conference sponsors, other? –Weighted constraints –Constraints may derive from physical possibility, principles of conference planning, event-specific needs, location requirements, organizational requirements, participant requirements –Performance of background tasks, e.g. –Adequate briefing to relevant parties –Accurate and timely adjustment of websites –Securing of necessary confirmations from vendors, participants –Completion of additional planning tasks

2 For Government Use Only Constraint Types Some constraints remain constant throughout the evaluation, some are changed as the crisis is introduced or as it evolves. Examples of constraints include the following: Laws of Time and Space –No physical resources in different places at the same time. –No resources required to travel the speed of light – or more - between locations (i.e., 12pm in Miami, 1pm in Pittsburgh) –Persons can’t be in the same place at the same time

3 For Government Use Only Constraint Types Resource Constraints –The budget for X is Y dollars. –x security guards/floor are required per floor. –Food/drink requirements –Room requirements: size, AV requirements, configuration. –Physical proximity of: Rest of conference Restrooms Vendor Constraints –Vendor has x items of type Y, resupplied weekly/ daily –Vendor has staff/resource in supply N security guards available at time. –Parking spaces

4 For Government Use Only Constraint Types Scheduling Constraints (applies to events or event types) –Finish to Start. Event B starts after Event A. –Start to Start. Event A starts when Event B starts. –Finish to Finish. Event A must end when Event B ends. –Lag Time. Event A must start when Event B ends + N hours. –As Soon as Possible. Schedule Event A in the first available window after Event B. –Start/Finish No Earlier/Later than Day X. Schedule Event A to start (or finish) at any time after/before 8am/ 5 p.m. on Day X. –Softer schedule constraints: There is likely to be audience overlap at Event X, Y and Z (i.e., don’t schedule them at the same time)

5 For Government Use Only Constraint Types Human Rules –Consider overlap possibilities for events. –Coordination of start and end times for parallel events –Don’t split sessions around lunch or other short breaks, if possible –Only schedule events at ½ hour (or any arbitrary user-defined) intervals. (Events can’t start at 2:06 p.m.) –Schedule events with large audience overlap in close spatio-temporal proximity. –Don’t schedule banquet on last day of conference.

6 For Government Use Only Completed components of schedule worksheet include: –Time, participant, location, resources scheduled. –For each component, score reflects completeness and constraint satisfaction for that event type. –Component score is weighted by: Importance in overall event (benefit) Degree of constraint satisfaction. Difficulty of constraint satisfaction. Schedule Score is then determined by the Weighted Proportional Completeness: –Take the ratio of the weighted completeness of the schedule to the maximum possible weighted completeness score if all events are fully scheduled. Schedule Scoring

7 For Government Use Only Scoring of schedule will depend on: –Ensuring that the constraint space is appropriately challenging. –Identifying the maximal possible score This requires coordination with CMU. We want to ensure that the problem space is not too constrained to prevent possible solutions or so underconstrained and use satisfiability of the constraints space to measure and ensure degree of scheduling difficulty. Schedule Scoring

8 For Government Use Only Other Scoring Website Scoring –We assume that a website template exists and can be followed. So, we ignore issues of design and organization. Score website for: –Completeness –Consistency with worksheet schedule –Correctness Other items: notification of release

9 For Government Use Only Other Scoring Briefing Scoring –Score briefing for: Completeness Consistency with worksheet schedule Correctness –Organization of Material –Appropriateness of Material with respect to given audience.

10 For Government Use Only Other Scoring Other issues: –Background planning, arranging with vendors: This is indirectly scored via the schedule plan. Audit trails used to ensure that contact and background tasks were performed as necessary.