UBICOMP ‘01 A Hybrid Evaluation Approach for Ubiquitous Computing Environments Mark Burnett, Chris Rainsford DSTO, Australia

Types of Evaluation Typical Characteristics QuantitativeQualitative Target ProblemWell defined, agreed metrics. Loosely defined, new, unknown metrics ProsWell defined outcomes, easy to compare candidates Allows exploration, encourages diversity, human subjectivity allowed. ConsNarrow focus, does not accommodate human subjectivity. Not well defined, hard to compare candidates, inconclusive. Evaluation Outcomes Improved performanceDiscovery of new approaches, identification of strong points TREC What’s the equivalent of ROBOCUP for a ubicomp environment? PKDD

A Hybrid Evaluation Framework Environment: Ubiquitous/pervasive  Invisible  Connected  Context-aware suggests hybrid of qualitative and quantitative approaches How to evaluate a ubiquitous computing environment? choose a well-defined environment, for example a smart room choose some well-defined tasks for a hybrid evaluation What to evaluate in a ubiquitous computing environment? software infrastructure and services in support of a task participants can build their own smart room (devices+infrastructure+s/w)