Display-agnostic Hypermedia Unmil P. Karadkar, Richard Furuta, Selen Ustun, YoungJoo Park, Jin-Cheon Na*, Vivek Gupta, Tolga Ciftci, Yungah Park Center for the Study of Digital Libraries Department of Computer Science Texas A&M University *Division of Information Studies School of Communication and Information Nanyang Technological University

Talk Overview Trellis and caT (context-aware Trellis)Trellis and caT (context-aware Trellis) MIDASMIDAS –Multi-device Integrated Dynamic Activity Spaces –Objectives –Architecture –Browser enhancements Display-agnostic hypermediaDisplay-agnostic hypermedia Resource ManagementResource Management Future workFuture work

Trellis Hypertext specificationHypertext specification –Based on a formal Petri net model –Includes the browsing semantics –Separates structure from its information content The browsing experience depends uponThe browsing experience depends upon –User identity –Initial state of the hypertext –Actions of other users browsing the hypertext

context-aware Trellis (caT) Extends TrellisExtends Trellis –Inherits its characteristics –Incorporates context-awareness –Includes support for hierarchical structures User Context affects the browsing experienceUser Context affects the browsing experience –Location –Time –Characteristics and preferences

caT and Trellis Server-centricServer-centric –Server maintains the state of browsing –Directs browsers to display information content for the current state BrowsersBrowsers –Receive pointers to the content to be displayed –If possible, display the content –Type-specific browsers, multiple browsers may be active simultaneously Information RenderingInformation Rendering –Onus for invoking appropriate browsers is on the reader –Information that cannot be presented by active browsers may not be rendered at all

MIDAS Multi-device Integrated Dynamic Activity SpacesMulti-device Integrated Dynamic Activity Spaces Extends caT - Inherits its characteristicsExtends caT - Inherits its characteristics ObjectivesObjectives –Ensure that all information content associated with the current browsing state is presented (completeness) –Ensure consistency of the presented information (coherence) –Coordinate the information rendered over all available browsers –Support browsing from multiple browsers on a set of diverse client devices –Allow clients and browsers to dynamically join or leave a browsing session – implications for maintaining coherence

Display-agnosticism MIDAS users may browse hypertexts fromMIDAS users may browse hypertexts from –Various devices Desktop, notebook computers PDAs, cell phones, palm top computers –Locations with different characteristics Homes, offices (private, protected) Airport terminals, hotel lobbies (public, insecure) Cars (semi-private) MIDAS must cope with diversity inMIDAS must cope with diversity in –device characteristics –Environmental constraints –User preferences Separation of information content from its renderingSeparation of information content from its rendering

Display-agnostic hypermedia Lend themselves to multiple forms of presentationLend themselves to multiple forms of presentation –Single structural representation –Alter the presentation for best rendering on individual devices –Conform to user preferences and characteristics –Within constraints of characteristics of available information SeparateSeparate –Structure from information content –Information content from presentation Address diversity of the information access environmentAddress diversity of the information access environment

Architecture – Trellis and caT Users Hypertext Authoring Tool Hypertext Specification Information Service Device 3 Device 2 Device 1 Browser 4 Browser 3 Browser 2 Authors Browser 1

Device 3 Device 2 Device 1 Browser 4 Browser 3 Browser 2 MIDAS MIDAS - Architecture Authors Users Device Manager user and device profiles, current device load resource handles, author preferences user actions Browser 1 select device(s) status, user actions Browser Coordinator Browser registry user actions information content Hypertext Authoring Tool Hypertext Specification Information Service resource handles, constraints, preferences information content, resource properties Resource Realizer Resource Repository resource instance(s) resource handle, constraints Resource Author Authors

Attributes That Affect Browsing DeviceDevice –Display (resolution, colors) –Storage space –Processor speed –Bandwidth UserUser –Expertise –Media, device and other preferences –Access privileges EnvironmentEnvironment –Privacy –Security –Interference Information elementsInformation elements –Media (mime) type –File size –Version –Creator –Creation timestamp –Location –Textual description –Required display space –Display colors –Required bandwidth –Language –Priority –Privacy High-level properties –Optimization quality time Cost –Interaction vs. autonomy

Browser-driven Information Rendering Different browsers render a caT specification differently, displaying only the elements they can render

Text browsing

Spatial Browser Composite browserComposite browser –Combines various information elements for display on a single canvas –Authors may Specify the display location of individual elements Leave the organization of elements to the browser Reinterprets hypertexts for an alternate presentationReinterprets hypertexts for an alternate presentation

Spatial Browsing

Audio-video Browser Audio-only feedback to usersAudio-only feedback to users Users interact via KeyboardUsers interact via Keyboard Serves asServes as –The primary browsing mechanism for visually impaired users –Alternate or supporting browsing mode for normal sighted users Dual mode browserDual mode browser –Hypermedia browsing –Access to individual information elements Extends caT’s browsing support to audio-video mediaExtends caT’s browsing support to audio-video media –Converts text to audio for rendering

Audio Browsing

Multi-device Browsing Support for variation in device characteristicsSupport for variation in device characteristics –Display size, network bandwidth –Availability of browsers for each device Information parallelismInformation parallelism –Media types (text, images, audio, video) –Variation in resolution Image scaling Text summarization Audio/video quality Authoring hypertexts and creating multiple instances of each resource is a tedious task!Authoring hypertexts and creating multiple instances of each resource is a tedious task!

Resource Management Provide feedback to support a variety of devicesProvide feedback to support a variety of devices –Feedback regarding device categories currently supported –Suggest media formats needed to support additional device types Aid resource creatorsAid resource creators – Automatic conversion of formats Audio to text, text to audio Image scaling and color-depth adjustments Extraction of text from formatted documents Text summarization

Future Work Unified hypertext authoring and resource management interfaceUnified hypertext authoring and resource management interface What is the closest match for a 640X480 image?What is the closest match for a 640X480 image? –80X60 thumbnail or a textual description of the image contents? –Similarity metrics for delivering the most appropriate instances of a hypertext’s information content How do users deal with multiple devices?How do users deal with multiple devices? –Implications for partitioning information across devices Integration of similarity metrics and partitioning algorithms into the Device manager and Browser CoordinatorsIntegration of similarity metrics and partitioning algorithms into the Device manager and Browser Coordinators

For more information The caT project homepageThe caT project homepage

Composition of Hypertext Systems ModelModel SpecificationSpecification ContentContent StructureStructure PresentationPresentation BehaviorBehavior ServerServer BrowsersBrowsers ExperienceExperience

Hypertext Systems Specification Model ContentStructure Behavior Presentation ServerBrowser Experience Author Reader

Diversity in Information Access Devices The information environment is diversifyingThe information environment is diversifying –Information devices Mobile devices, Desktop computers, and Large screen devices Form factor, network bandwidth, storage, processing power –Properties of the environment Private (homes, offices), semi-private (cars), and public (malls) Most hypertexts are authored for a single class of devices – desktop computers or mobile devicesMost hypertexts are authored for a single class of devices – desktop computers or mobile devices –Separate structure as well as services for each class The Weather Channel, CNN, Yahoo! –Do not account for diversity of device characteristics within their class

Architectural Enhancements Multi-device Integrated Dynamic Activity Spaces

Trellis and caT Architecture Users Hypertext Authoring Tool Hypertext Specification Information Service Device 3 Device 2 Device 1 Browser 4 Browser 3 Browser 2 Authors The Web Browser 1

context-aware Trellis Petri net-based hypertext systemPetri net-based hypertext system Stateful serverStateful server –Maintains state (unlike HTTP servers) –Users may browse simultaneously from multiple browsers More than one computer –Browsers mirror the state of browsing –State is maintained in the absence of any clients Type-specific browsers support variousType-specific browsers support various –Media types (text, images, audio, video) –Browsing metaphors (combine media types – Web browser, spatial browser)

context-aware Trellis Browsing experience depends onBrowsing experience depends on –User identity (privileges) –User environment –Actions of other users John Desktop Browser A Browser B Notebook 1 Browser C Bob Notebook 2 Browser D Browser E caT Server (hypertext specification, state)

context-aware Trellis caT servercaT server –Separates structure from information content –Directs browsers to render this content BrowsersBrowsers –Render the content to the best of their abilities Image browser will not display textual information at all –Accept user actions and forward these to the server Link activation

Enhancements to caT Extend caT’s browser-baseExtend caT’s browser-base Architectural enhancementsArchitectural enhancements –MIDAS

Browser multiplicity Earlier caT’s browsersEarlier caT’s browsers –Text, image, and Web browsers New browsers supportNew browsers support –Additional media types (audio-video browser) –Different presentation metaphors (spatial browser)