Composing Devices Prasun Dewan Department of Computer Science University of North Carolina

2 Issues raised by UPnP n Scaling problem of multicast-based discovery u Auto Shut off problem n Simple-minded search n Lack of access and concurrency control n No programmer-defined types n High programming and maintenance costs n Manual, procedural service composition n Device-independent UI

3 Declarative Specifications n Defined by declarations n Higher-level than procedural u HTML vs. Toolkits F User addresses what rather than how. n Not as general. Domain specific u HTML vs. Toolkits F Form-based user interfaces F No graphics, incremental feedback. n Composition domains for declarative specifications?

4 Database based composition Queries over multiple devices u For each rainfall sensor, average rainfall u For each sensor in Tompkin county, current rainfall u For next 5 hrs, every 30 minutes, rainfall in Tompkin county

5 Query Kinds Device Queries n Historical u For each rainfall sensor, average rainfall n Snapshot u For each sensor in Tompkin county, current rainfall n Long-running u For next 5 hrs, every 30 minutes, rainfall in Tompkin county u New kind of query n Defining device databases and queries?

6 Cougar Device Model n Embedded vs. Attached n Stationary vs. Mobile n Strongly vs. Intermittently connected n Local area vs. wide area n Device database work focuses on stationary devices n Sensors vs. arbitrary devices?

7 Cougar Device Operations n Operation Model u acquire, store, and process data u may trigger action in physical world u return result n Synchronous operation u returns result immediately n Asynchronous operation u result(s) later F abnormal rainfall F as event n Intermittently connected device u only asynchronous operations possible u device not guaranteed to be connected when operation invoked

8 Defining Device Database Device DBMS vs. Traditional Relational DBMS n Device vs. Data collection u Computed vs. stored values n Distributed information sources u Data needed not available locally u May not even be available remotely for intermittent n Long-running queries. u Not modelled by traditional DBMS Solution n Base device relations u One record for each device n Virtual Relations u Records partitioned over distributed nodes u Includes results of device operations n Extended query language over virtual relations

9 Base Relations Collection of devices of a particular type. n one record per device n attributes u device id u X coordinate u Y coordinate IDXY

10 Virtual Relations Per device function u Attribute for F each function argument F result F global timestamp of result F device id u New record added for each new result u Append-only relation u Each device contributes to part of relation a1a1 Valamam TSID f( a 1,…, a m ): T

11 Example RFSensor n One function u getRainfallLevel() : int n Base relation u RFSensors n Virtual relation u VRFSensorsGetRainfallLe vel ValueTS Device ID RFSensors VRFSensorsGetRainfallLevel Device IDXY

12 For next four hours, retrieve every 30 seconds, rainfall level of each sensor if it is greater than 50mm. Query Q: SELECT VR.value FROM RFSensors R, VRFSensorsGetRainfallLevel VR WHERE R.ID = VR.ID AND VR.value > 50 AND AND $every(30) Run Q for 4 hours 200 devices R cardinality = 200 VR cardinality = 480 Long running queries joinselection AND R.X =... IN... Why Join?

13 Execution Strategies RMaterialized VR Q No local knowledge RVR VR.value > 50R.ID = VR.ID Val every 3O min Local rate info, remote Q RVR if R.ID = VR.ID VR.value > 50 Val every 3O min Local Join, Remote Selection RVR R.ID = VR.ID if VR.value > 50 Val every 3O min Local Selection, Remote Join

14 Performance Metrics n Traditional u Throughput u Response time F Long query running time implementation independent n Sensor specific u Resource usage F network, power n Reaction time u Production to consumption time

15 Power Usage Components n CPU n Memory access n Sending message n Sending Nbytes Cost in joules = W cpu *CPU + W ram *RAM + W msg* Msg + W bytes *NBytes

16 Centralized Warehouse No local knowledge n Single location monitors all sensors n Queries are sent to site n Works for historical queries n Wastes resources for long running queries u Irrelevant sites u Higher rate than necessary n What to monitor for long running queries u Camera direction? n Centralization u Workload u Bottleneck

17 Distributed Device Database n All sensors together form a distributed device database system n Individual nodes u sense on demand u do part of query processing n Better resource utilization n Know what to monitor n Historical? u based on some long-running query

18 Remote Query Evaluation n All sensors send data n No relations sent to sensors RVR VR.value > 50R.ID = VR.ID Val every 3O min Local rate info, remote Q

19 Local Join RVR if R.ID = VR.ID VR.value > 50 Val every 3O min Local Join, Remote Selection n Only joined sensors send value n Whole relation send to device u communication overhead

20 Local Selection n Only selected sensors send data n Whole relation not send to device u best approach RVR R.ID = VR.ID if VR.value > 50 Val every 3O min Local Selection, Remote Join

21 Composition Techniques n Query approach u Data of sensors aggregated together n Alternatives u Operations of devices aggregated u Results of one device affect operations of another

22 Automatic Composition Domains n ICrafter and Local Work u Device pipes F Camera/Scanner to printer u Proxy based connection F compression u Adapter based connection F transcoding u Composite commands F DoAll Turn on all VCRs F Do Related Turn on projector Dim lights

23 ICrafter Approach System selects composer based on interfaces of components and registered interface patterns DataProducer DataConsumer PowerSwitch DataProducer, DataConsumerProducerConsumerPipe Interface Pattern“Generic” Composer PowerSwitchPowerSwitchAll

24 Selecting Composer n Pattern described using “regular expressions” u single producer consumer u multiple producer consumers n Client requests composer for a set of devices n Returns composer for each subset of devices with matching pattern n Composer provides user-interface for invoking composite commands u Switch on all lights u Drag producer to consumer or vice versa

25 ICrafter: A Service Framework for Ubiquitous Computing Environments Shankar Ponnekanti, Brian Lee, Armando Fox, Pat Hanrahan, Terry Winograd (Stanford Univ.)

26 Aggregation n May seem individual UI’s can be combined. However ….

27 UI(S1+S2) != UI(S1)+UI(S2) n Good for individual operations n Clumsy for compound operation: snap-and- display

28 UI(S1+S2) != UI(S1) + UI(S2) n Transfer = snap and display in one click

29 Where We Stand n Suppose n services, m appliances and w workspaces u O( F(n) × m × w ) UI’s u F(n) is some combinatorial function of n n Existing ad-hoc interaction systems: Jini, UPnP, Hodes et al (Mobicom97, USITS99), Roman et al (WMCSA00) n Mostly focus on appliance heterogeneity

30 Generators: Handling Aggregation n Generators for multiple services u Eg. {Camera, Display}, {Camera, Display, Display} n Generators for service patterns u Eg. {Camera, Display+}, {Projector * } n Generators for service interface patterns u Eg. {DataProducer, DataDisplayer+}

31 Simplified Example n Request for {Camera, Display} n Matches Camera generator, Display generator, and {DataProducer, DataDisplayer} generator

32 Simplified Example n Aggregated using panels

33 Outline n What? n Why? n How?

34 Bootstrapping

35 Future Work n Authentication and synchronization n Aggregation: more intelligent generator set selection: u Eg. Eliminate a {DataProducer, DataDisplayer} generator if a {Camera, Display} generator exists

36 Interactive Workspaces Project Info n Some software already available n Major release (including ICrafter) due this month n For software and other info:

37 Problems of ICrafter n Must manually connect devices u select from a list u drag and drop u Automatic continuous like Unix pipes? F Equivalent of standard input and output? n Interface/Class-based composition u rather than programming pattern F Getter/setter

38 Problems of Interface-based Approach n Type of Data in Interfaces? u Generic Object F Truly generic composer possible F One composer for all producer/consumer pairs Rather than camera/printer, temperature sensor/aggregator pairs F Errors possible Temperature piped to printer u Programmer-defined F Type checking possible F Need programmer-defined composers.

39 Problems of Interface-based Approach n Number of DoAll Composers u At least one for each interface n Number of Interfaces Implemented? u One of for every operation F power, play F can share composers for different devices can turn on all cameras and VCRs F too many interfaces and associated composers u One of each service F limited sharing of composers F need one for each combination of device types

40 Summary of Problems of Interface- based Approach n Explosion of # of composers u In producer/consumer connections F for each consumer/producer data type u In do-all composition F if interface = operation for every operation F if interface = service for every service combination with overlapping operations n No automation really supported in composing n Automation only in looking up and “composing”composers n No support for proxy and adapters.

41 Alternate: Pattern-based approach n Pattern-based vs. interface- based protocols n Pattern-based u must follow conventions

42 DoAll Composition capture() on() capture() n Do all: on() n Do all: capture() on()

43 Method-based doAll algorithm n Find operation headers of all component devices n For each operation header create a doAll operation that invokes device-specific implementation of header on each device providing it. n Accidentally combining unrelated commands with same signature u foo(), compute(), doIt(), mutate() n Not combining related commands with different signatures u power() powerVCR() and on() n non combinable commands() u serialize()

44 Overriding Default doAll() doAll on(): vcr.powerVCR(), tv.powerTV() and camera.turnOn() Access methods on = power.Device.() serialize = serialize() Disable doALL serialize() Access methods on = turnOn() serialize = serialize() Disable doALL serialize()

45 doAll Access-Method Algorithm n Find access-method headers of all component devices n For each header create a doAll operation that invokes an implementation of header on each device providing it that does not disable its compositions

46 Producer-Consumer Composition n Much harder problem u how to regularize producers and consumers? n Enumerate and Generalize Examples u Continuous vs. discrete through composer F Unix pipes of continuous data vs. ICrafter connection of discrete data u Continuous direct connection F push-pull based u continuous proxy based connection F push-pull based u continuous adapter based connection F push-pull based u Direct connection vs. connection through composer

47 interface PictureProducer { Image capture();... } interface PictureConsumer { void print (Image p); … } Discrete: Single producer, single consumer Connect (camera, printer) new operation: printAndCapture

48 interface PictureProducer { Image capture();... } interface PictureConsumer { void print (Image p); … } Discrete: Multiple producer, single consumer Connect (cameras, printer) new operation: captureAllAndPrint

49 interface PictureProducer { Image capture();... } interface PictureConsumer { void print (Image p); … } Discrete: Single producer, multiple consumers Connect (camera, printers) new operation: captureAndPrintAll

50 interface PictureProducer { Image capture(); void setPictureDestination(PictureConsumer c); void sendToPictureDestination();... } interface PictureConsumer { void print (Image p); … } Continuous Push-based Connection Connect (camera, printer): camera.setPictureDestination(printer);

51 interface PictureProducer { void capture(); void addPictureDestination(PictureConsumer c); void sendToPictureDestinations();... } interface PictureConsumer { void print (…); … } Pushing to multiple destinations Connect (camera, printer): camera.addPictureDestination(printer);

52 interface PictureConsumer { void setPictureSource(PictureProducer c); void getFromPictureSource();... } interface PictureProducer { void capture (…); … } Direct Pull-based Continuous Connection connect(camera, printer): printer.setPictureSource(camera);

53 interface PictureConsumer { void addPictureSource(PictureProducer c); void getFromSources();... } interface PictureProducer { void capture (…); … } Pulling from multiple sources connect(camera, printer): printer.addPictureSource(camera);

54 interface PictureConsumer { void print();... } interface PictureProducer { void addPictureDestination(PictureConsumer c) void capture (…); } Proxy based connection interface PictureCompresser extends PictureConsumer { void addPictureDestination(PictureConsumer c); void compress();... } registerProxy(ACompressor) connect(camera, printer): compresser  new ACompresser(); compresser.addPictureDestination(printer); camera.addPictureDestination(printer);

55 interface PictureConsumer { void addPictureSource(PictureProducer c); void getFromSources();... } interface PictureProducer { void capture (…); … } Proxy-based pull connection interface PictureCompresser extends PictureProducer { void addPictureSource(PictureProducer c); void compress();... } registerProxy(ACompressor) connect(camera, printer): compresser  new ACompresser(); compresser.addPictureSource(camera); printer.addPictureSource(camera);

56 interface PictureConsumer { void print();... } interface PictureProducer { void addPictureDestination(PictureConsumer c) void capture (…); } Proxy vs. Adapter: Proxy interface PictureCompresser extends PictureConsumer { void addPictureDestination(PictureConsumer c); void compress();... } registerProxy(ACompressor) connect(camera, printer): compresser  new ACompresser(); compresser.addPictureDestination(printer); camera.addPictureDestination(printer);

57 interface BWPictureConsumer { void print();... } interface ColorPictureProducer { void addPictureDestination(ColorPictureConsumer c) void capture (…); } Proxy vs. Adapter: Adapter interface PictureTranscoder extends ColorPictureConsumer { void addPictureDestination(BWPictureConsumer c); void transcode();... } connect(camera, printer): transcoder  new APictureTranscoder(); camera.addPictureDestination(transcoder); transcoder.addPictureDestination(printer);

58 Implementation of Continuous Connection n Motivated by composition of components of Collaboration Infrastructure called Collaboration Bus n Direct, Proxy based, and Adapter based continuous connections n Integration of registration/discovery and composition n Based on notion of facets (like pins on a chip) n Munson and Sunku

59 Facets public void add ( ) {...} [public remove ( ) {...}] [public Enumeration enumerate () {...}] public void set ( ) {...} [public get () {...}] public ; FacetName (FacetType) object Assign (source, facet, destination) source.addFacet(destination) source.setFacet(destination) source.facet = destination

60 Bi Directional Connect connect (camera, pictureDestination, printer, pictureSource) assign (camera, pictureDestination, printer) assign( printer, pictureSource, camera); PictureDestination (PictureConsumer) camera printer PictureSource (PictureProducer)

61 Uni Directional Connect connect (camera, null, printer, PictureSource) assign (printer, pictureSource, camera) camera printer PictureSource (PictureProducer)

62 Uni Directional Connect connect (camera, pictureDestination, printer, null) camera.setPictureDestination(printer) PictureDestination (PictureConsumer) camera printer

63 Multiple Communication Primitives n May wish to pass reference to u Object u Socket u Multicast channels n Pattern u getReferenceFor () u If this method exists return value used instead of object defining the facet

64 Multiple Communication Primitives connect (camera, PictureDestination, printer, PictureSource) assign (camera, PictureDestination, printer.getReferenceForPictureSource()) assign( printer, PictureSource, camera.getReferenceForPictureDestination()); PictureDestination (PictureConsumer) camera printer PictureSource (PictureProducer)

65 Integrating connection and discovery Connect (printer, PictureDestination) u Integrate discovery and connection u Discover an object whose type can be assigned to facet F Discovered objects must have been registered F Registries distributed and connected to each other u Must now find peer facet  public Facet get DestinationFacet(). u Make bi-directional connect if such a facet exists, else unidirectional connect

66 Integrating registration and connection Register(camera) u For each auto connect facet, f F Connect (camera, f) u Facet is auto connect if F member of AutoConnectFacetNames property u Or if class of object F Implements AutoConnectAll interface

67 Proxy-based Connect PictureDestination (PictureConsumer) camera printer PictureSource (PictureProducer) PictureDestination ( PictureCosnumer ) Proxy PictureSource (PictureProducer)

68 Proxy-based connect registerProxy(p, “Proxy/facet1/type1/facet2/type2”) connect (s, fs, d, fd) p = findProxy(“Proxy/fd/typeOf(s)]/fs/typeOf(d)”) If (p != null) connect (s, fs, p, fd), connect (p, fs, d, fd) else if not (direct_connect(s, fs, d, fd)) adapterConnect(s, fs, d, fd) ;

69 Adapting the Type PictureDestination (ColorPictureConsumer) camera printer PictureSource (BWPictureProducer) PictureDestination ( BWPictureConsumer ) Adapter PictureSource (ColorPictureProducer)

70 Adapting the Facet ColorPictureDestination (ColorPictureConsumer) camera printer BWPictureSource (BWPictureProducer) BWPictureDestination ( BWPictureConsumer ) Adapter ColorPictureSource (ColorPictureProducer) connect (camera, ColorPictureDestination, printer, BWPictureSource) Cannot derive adapter facet names from connect.

71 Unidirectinal: Adapting the Facet ColorPictureDestination (ColorPictureProducer) camera printer BWPictureDestination ( BWPictureProducer ) Adapter connect (camera, ColorPictureDestination, printer, BWPictureSource) Can derive adapter from connect, but not opposite facet

72 Adapting the Type registerProxy(p, “Proxy/facet1/type1/facet2/type2”) adapterConnect (s, fs, d, fd) p = findProxy(“Proxy/fd/*/fs/*”) connect (s, fs, p, fd), connect (p, fs, d, fd)

73 Adapting the Facet registerProxy(p, “Proxy/facet1/type1/facet2/type2/pfacet1/pfacet2”) adapterConnect (s, fs, d, fd) p = findProxy(“Proxy/fd/*/fs/*/pf1/pf2”) connect (s, fs, p, pf1), connect (p, pf2, d, fd)

74 Creating proxies registerProxy(p, “Proxy/facet1/type1/facet2/type2/pfacet1/pfacet2”) registerProxy(pClass, “Proxy/facet1/type1/facet2/type2/pfacet1/pfacet2”) pClass instantiated automatically registerProxy(pFactory, “Proxy/facet1/type1/facet2/type2/pfacet1/pfacet2”) special method in pFactory to create proxy

75 Hybrid of Discrete and Continuous n Example u set time of newly turned on vcr to that of clock F composite command: replicate clock time F system calls getter method for clock property and then setter method for vcr property when vcr service registered n Can generalize to arbitrary properties u synchronize TV channels in a bar F user enters synchronize channel mode F composer listens for channel properties and synchronizes them F Roussev’s coupler research valid here

76 Domain-specific Device Composition n Example u Compose multiple devices, possibly of different users, so that they work together to show a multimedia web presentation. n Websplitter u Based on XML

77 WebSplitter: A Partial-View XML Framework For Multi- Device Collaborative Web Browsing Professor Rick Han Department of Computer Science University of Colorado at Boulder

78 Outline n A High-Level View of Pervasive Computing n WebSplitter Motivations u Per-Device Partial Views u Per-User Partial Views n WebSplitter’s XML Framework n Demo Prototype

79 Motivation: Enabling A Pervasive Computing World

80 WebSplitter Motivation #1: Generate Partial Views Per Device n Scenario: Web presentation in lecture hall controlled by a wireless PDA Each device receives partial view device’s capability intended function Leverage other dev’s Composite view Lecturer

81 WebSplitter Motivation #2: Partial-View Collaborative Presentations n Scenario: Online Web presentation to audience – what you see is ~ what I see Lecturer Participant 2 Participant 1

82 Partial Views Per User n WebSplitting generates subsets of a Web page for each user, i.e. partial views Partial views limit access to components –To hide information –To prevent misdirection of a presentation Partial view scenarios: –Teacher giving presentation to students (distributed) Customized learning based on individual skill levels, i.e. (teacher, pupil1, pupil2, …), not merely (teacher, students) –Adversarial collaboration, e.g. lawyers

83 WebSplitter: Joint Multi-User Multi-Device Presentations n Scenario: Web presentation to online audience and a lecture hall via a remote control PDA Lecturer Participant 2 Participant 1

84 Multiple users Multiple devices WebSplitter’s Architecture WebSplitter Middleware Policy File

85 XML Framework: How does WebSplitter use XML? n Web pages are authored in XML u Tags correspond to components on Web page u Easy to manipulate tags & groups of tags – nested hierarchy F use XSL to filter/insert/delete tags n “Policy file” implements partial views u defines tag mapping policies: which users and which devices have access to which XML tags u Written in XML u External annotation file containing authorization

86 Example: Splitting an XML Web page The online Shopping company The most reliable and the best prices online WebSplitter The online Shopping company The most reliable and the best prices online Title, Description, and GIF Logo HTML to PocketPC WAV Music and GIF Logo XML to laptop

87 XML Policy File: Filtering Tags by User Groups n Define user privilege groups u n Each privilege group receives a subset of tags u presentation, head, title, nav_bar, slide, picture presentation, slide, picture

88 XML Policy File: Filtering Tags by Devices n Within a group, map subsets of tags to devices u u Standardizing names <cmdb:group name = ”lecturer” … presentation, head, title, nav_bar, slides, picture presentation, head, title, nav_bar, notes <cmdb:group name = ”students” …… ………

89 XML Policy File: Filtering Tags By User & By Device n Define user privilege groups n Each privilege group receives a subset of tags n Within a group, map subsets of tags to devices <cmdb:group name = ”lecturer” … presentation, head, title, nav_bar, slides, picture presentation, head, title, nav_bar, notes <cmdb:group name = ”students” …… ………

90 Implementation: WebSplitter Demo Bluetooth Bluetooth LAN Access Point EudoraWeb Laptop2 WebSplitter proxy Ethernet LAN Laptop1 Web Server XML Source & Policy File 2 privilege groups in policy file

91 WebSplitter Prototype Demonstrates: n Splitting to different browsers on different devices u XML source Web page u XML policy file F 2 privilege groups u Client-side component on Palm, WinCE, Netscape u Static service discovery

92 Issues raised by WebSplitter n Integrate with Producer-Consumer Notion u Splitting an output ~ connection to multiple consumers u Splitting an input ~ connection to multiple producers n Automatic vs. Declarative

93 Composition Techniques n Fully automatic u doAll command u pipes u false positives/negatives possible n Semi-automatic u disabling/enabling doAll composition u Specifying proxies n Declarative u WebSplitter XML file

94 Semi-Automatic vs. Declarative n Fine line between semi-automatic and declarative n Simple parameters (forms) rather than specification language. n Semi-automatic overrides fully automatic