MIDAS Multi-device Integrated Dynamic Activity Spaces Unmil P. Karadkar Ph.D. Candidate Dept. of Computer Science Texas A&M University

Theory – Which aspects of information content do humans find valuable? Techniques – How to adapt content while optimizing for human perception? – How to enable heterogeneous device co-use? Frameworks – What are the attributes of multi-device systems? – How options are available to system designers for each attribute? System – Design a flexible architecture to embody multi-device system behaviors – Proof-of-concept implementation and analysis Practice – Harness device multiplicity to accomplish significant tasks Research Questions

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

14”17” 40”- 60” 21-24” 3.5” Custom 13”- 17” 75” 10” 4.5” 6” 7” –Display characteristics –Network bandwidth –Processing power –Interaction modalities

Societal Changes User demographics Nature of activity Access location Affordability – Free phones with contract – $150 e-readers Multiplicity – iPhone, iPad, MacBookPro Heterogeneous contexts of information access

MIDAS M ulti-device I ntegrated D ynamic A ctivity S paces M I D A S

1.Deliver information elements to multiple devices simultaneously – Content – Interaction 2.Rely on widely available infrastructure – Wired and wireless computer networks – Mobile phones, e-readers, tablets, desktop computers – Sensors embedded in the environment 3.Support content transformation – Within form (image scaling, text summarization) – Across forms (text to speech, audio extraction from video) Design Goals

4.Strive to retain content integrity from a human perspective – Select transformations that a user perceives to be closest to the original form 5.Direct content elements to devices that can present them optimally 6.Reconfigure presentation dynamically in response to changes in device availability – New devices become available – Current devices become unavailable Design Goals

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

Background Individual devices device-based content adaptation Ubiquitous Computing Infrastructure-enabled device co-use Service separation –Yahoo!, CNN, Weather Channel, UT Mobile Image distillation –Digestor (Fox 1996) Text and Web form summarization –Pythia (Bickmore 1997) –Power Browser (Buyukkokten 2002) –Proteus (Anderson 2001) Web page content extraction –Baseline SVPs + device-based PVPs (Chua 2007) Location-aware services –Computer for the 21 st century (Weiser1991) –Java Ring (Olsen 2001) –IntelliBadge TM (Cox 2003) Mobile devices in interactive spaces –Personal Server (Want 2002) –InfoStick (Kohtake 1999) –Elope (Pering 2005) Opportunistic device co-use

Background—Device co-use Pebbles (Myers 1997) – Desktop and PDA co-use – Extend Windows applications to PDAs – PDAs as input and control devices WebSplitter (Han 2000) – Split Web content across a user’s devices – Support for multiple views ANMoLE (Haneef 2004) – Dynamic content reformulation (splitting video) – Integration of phone and computer networks

Background—Device co-use Ubiquitous Display System (Aizawa 2002) – Large public displays and mobile phones – Control from mobile phone Multibrowsing (Johanson 2001) – Desktops, notebooks, PDAs, and large displays – Specially encoded links for content routing Interface Distribution Daemon (Luyten 2005) – Web interface as well as stand-alone applications – Multiple content allocation modes – Usability analysis - Completeness and Continuity

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

Concurrency – Simultaneous multi-device content presentation Control – Interaction support Comity – Permissible diversity in device characteristics Completeness – Presentation of relevant information elements Coverage – Distribution of content to optimize device usage Conversion – Content transformation to suit target device characteristics 12C Framework

Composition – Relationship of components to the interface Coherence – Consistency in presentation of content elements Coordination – Reallocation of elements when devices change Continuity – Support for users to interpret and evaluate system state Constancy – Reuse of content instances that have been shown before Confidence – Security and trustworthiness in communication 12C Framework

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

Development Platform Web or something else?

WWW – Dominant infrastructure – World-wide audience Stateless server – Client-initiated communication – Content embedded within structure, – Changing devices means restarting the activity Standardized browsers – Little support for custom behavior WorldWide Web

context-aware Trellis (caT) Petri net-based hypertext system (Furuta, Stotts ) – Formal, graph-based model – Browsing semantics (user privileges, time of day, location) Decouple specification from information content – Content not subsumed within structure –, – Separation allows devices to select suitable content Decouple content from presentation – Browsers present content autonomously Stateful server – Support for multiple clients – Easy propagation of user actions and content to multiple devices

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

Device 3 Browser 4 Browser 3 Browser 2 Device 2 Browser 4 Browser 3 Browser 2 Device 1 Browser 4 Browser 3 Browser 2 MIDAS Browser Architecture Specification Information Service caT Users

Device 3 Browser 4 Browser 3 Browser 2 Application Coordinator Device 2 Browser 4 Browser 3 Browser 2 Application Coordinator Device 1 Browser 4 Browser 3 Browser 2 MIDAS Browser Architecture Device Manager Browser Coordinator Resource Realizer Specification Information Service Users

Device 3 Browser 4 Browser 3 Browser 2 Application Coordinator Device 2 Browser 4 Browser 3 Browser 2 Application Coordinator Device 1 Browser 4 Browser 3 Browser 2 MIDAS Architecture instanceproperties content handle user actions content handles Specification Information Service Users instance handle, actions status, user actions instance handle content Browser user actions content Resource Realizer Resource Repository Browser Coordinator Device Manager content handles instance properties (instances, devices)

Presentation only Input only Interactive Presentation only Input only Interactive Device Manager Instance properties Device Selection Scheme Device properties, interaction mode Instance Scorer Content-Device Map Content Handles, properties Information Service Resource Realizer Browser Coordinator Fully Replicated Interaction Replicated Instance Optimal MIDAS Optimal Fully Replicated Interaction Replicated Instance Optimal MIDAS Optimal

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

Content Transformation Images – Scaling (automatic) – Color reduction (automatic) Text – Text extraction from formatted documents (automatic) HTML, Word, PDF – Summarization (semi-automatic) – Formatted data such as tables (semi-automatic) – Visual or audio rendering (automatic) Video – Audio extraction, down-sampling (automatic) – Video summarization (semi-automatic)

Resource Manager Authoring support Device dependence for content instances Automatic transformation of images (ImageMagick) Author validation

Perception-based Image Transformation Images – Changes to attributes of photographs – Fastest growing information content on the Web Goal of user study – Understand human perception of nearness Scaling of images Color reduction (and gray scaling) – Obtain actionable metrics to design rules for providing content in its most suitable form

Study Design Two stages – Stage 1: Image classification – Stage 2: Similarity of image pairs Subject demographics – Five subjects per stage – Graduate students and staff – Different disciplines

Stage 1 Subjects classified 100 photographs – Taken by four photographers – Everyday and travel pictures – Prints of digital photos to make piles No guidelines regarding classification – Subjects created between 12 and 40 categories I coalesced categories based on image content – Four final categories

Image Types People Structures Nature Text

Stage 2 Subjects viewed image pairs on identical displays – No overlap or switching windows Each pair differed in size or color – Variables manipulated independently 20 image pairs for each variable Answered three questions – Similarity of images (9-point scale) – Suitability of automatic substitution (yes/no) – Acceptance of informed substitution (9-point scale)

Scaling

Color Reduction

Data Analysis 160x x x x x768 2 (1-bit) 4 (2-bit) 16 (4-bit) 256 (8-bit) 16 Million (24-bit) Size Colors

Effect of Information Content Loss DistanceScoreContent Loss % % % % % DistanceScoreContent Loss % % % % Scores on a 9 pt. Scale 320 X 240 X 24 vs. 320 X 240 X X 768 X 24 vs. 160 X 120 X 24 (no loss) (98 %) (no loss) (96 %) Scaling Color depth change

Similarity by image type Images of nature and structures scale well Human faces Legibility of textual elements Algorithms for face and text detection in images exist Distance naturestructurespeopletext Scaling

DistanceAutomatic substitution AcceptabilityInformed Substitution Acceptability 144%7.6656% %7.4251% %7.1350% %7.6779%4.18 Scaling– Image Substitution No clear trends in terms of distance Subjects allowed automatic substitution – Corresponded with higher replacement suitability

DistanceAutomatic substitution AcceptabilityInformed Substitution Acceptability 044%7.5056% %7.4364% %8.0090% %8.7571% %8.0071%3.80 Color Reduction – Image Substitution Acceptance of automatic substitution is lower than that for scaled images The suitability ratings are correspondingly higher

Perception-informed Rules Principles – Prefer scaling over color reduction – Avoid scaling of images that contain people and text – Gray scaling is the best option when used by itself Normalized score: Original image = 0 (best score) +W color (9 - Score dist, color ) 9 x Image Rating = W size (9 - Score dist, size ) 9 xW type x W togray = 1.0 (grayscale) W togray = 0.0 (otherwise) W type +W togray (9 - Score dist, color ) 9 x W type (people, text) = 0.53 W type (nature, structures) = 0.47

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

Browsing an Online Art Exhibit Petri net structure –Not a user’s view Co-browsing from –Mobile phone –240 X 320 pixels –3.5” display –24 bit-color –1 GB storage –Desktop computer –1600 X 1200 pixels –21” display –24-bit color –40 GB storage

Browsing an Online Art Exhibit Desktop computer –1600 X 1200 pixels –21” display –24-bit color –40 GB storage –Display-only Mobile phone –240 X 320 pixels –3.5” display –24 bit-color –1 GB storage

Browsing an Online Art Exhibit Desktop computerMobile phone

Browsing an Online Art Exhibit Web interface on Desktop computer

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

1.Deliver information elements to multiple devices simultaneously – Content – Interaction 2.Rely on widely available infrastructure – Wired and wireless computer networks – Mobile phones, e-readers, tablets, desktop computers – Sensors embedded in the environment 3.Support content transformation – Within form (image scaling, text summarization) – Across forms (text to speech, audio extraction from video) Analysis - Design Goals ✔ ✔ ✔

4.Strive to retain content integrity from a human perspective – Select transformations that a user perceives to be closest to the original form 5.Direct content elements to devices that can present them optimally 6.Reconfigure presentation dynamically in response to changes in device availability – New devices become available – Current devices become unavailable Analysis - Design Goals ✔ ✔ ✔

Analysis - 12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Present information elements on multiple devices Inclusion of devices in MIDAS is optional WebSplitter, ANMoLE A user can choose the interaction mode for each device Inclusion of public devices is not supported Ubiquitous Display System Content is directed to devices automatically Multibrowsing, Ubiquitous Display System

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Interaction support Flexible – dependent on device choices Most architectures support centralized or distributed It is possible to have no input devices at all May change during a session

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Permissible diversity in device characteristics MIDAS places few restrictions on devices Pebbles, Ubiquitous display system Device must run the Browser Coordinator No requirement of proximity to a user Remote devices may be included - printers

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Presentation of relevant information elements Ensures completeness by presenting all resource ids Resource Manager aids content authors Support includes textual description as fallback IDD assertion is based on fixed content instances ANMoLE and MIDAS convert content

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Distribution of content to optimize device usage Flexible – dependent on Device Manager mode Fully replicated - redundancy Interaction replicated Instance optimal - coverage for individual instances MIDAS optimal – system-wide coverage Coverage may not apply to some systems Ubiquitous Display System, Multibrowsing

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Content transformation to suit target device characteristics Resource Manager Authoring-time syntactic conversion with user validation Device Presentation-time implicit conversion

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Relationship of components to the interface Generative approach Instances and actions together form the interface Many Web-based systems take a degenerative approach Content filtering

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Consistency in the presentation of content elements Content presented on all devices reflects the current state Device Manager mode dependent Replication modes may present inconsistent information if the content is not vetted by the author

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Reallocation of elements when devices change In response to Device changes User actions Environmental properties changes (inherited from caT) Potential for optimization Not all events threaten the integrity of a presentation

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Reuse of instances that have been shown before Not supported (yet) MIDAS always picks the most optimal instance Systems that use fixed instances support constancy better Web-based presentations

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Support for users to interpret and evaluate system state Supported with user-initiated state changes Device changes User actions Presentation changes due to environmental properties Causes may be subtle – off-peak hours in another time zone

12C Framework ConcurrencyComityControl ConversionCoverage Completeness Coordination CoherenceComposition ConfidenceConstancyContinuity Security and trustworthiness in communication Reliance on the underlying communication infrastructure Connection-oriented sockets Security or encryption is not supported

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

Contributions Theory – 12C Framework for characterizing multi-device systems – Human perception of nearness in image adaptation Nature of content loss is important Not all images scale equally gracefully Gray scaled images are perceived to be the closest Techniques – Perception-informed metrics for image adaptation System – MIDAS architecture and prototype Flexible infrastructure Minimal requirements for including diverse devices Expression of various server- and client-side behaviors Coordination in response to users, environment, and devices

Motivation Related Work 12C Framework Approach MIDAS Architecture Content Transformation Demonstration Analysis Contributions Future Work Outline

Future Work Infrastructure – WebSockets API (HTML5), CSS3 (device-specific templates) – Clients for Android, iOS, and Windows – Inclusion of large-screen displays — GoogleTV(?) Interaction and Usability – Realistic setting, contemporary devices – Attention, cognition, privacy – Controlled studies, data from long-term use – Exploration of collaborative use Content transformation – Other image adaptations – Suitability of lessons learned from presentation for disabled populations – Cross-form adaptation – Normalization of scores across different content forms

The most profound technologies are those that disappear. - Weiser 1991.

(3, ∞) (green) context-aware Trellis Super set of HTML – Structure – Browsing semantics – Access control – Timed browsing Separation of – Hypertext structure – Information content – Browsing behavior