Outdoor Distributed Computing* Cristian Borcea Department of Computer Science New Jersey Institute of Technology * Work done at Rutgers prior to September 2004

2 Computers Embedded Everywhere

3 Distributed object tracking over a large geographical area Cars collaborating for a safer and more fluid traffic Outdoor Distributed Applications How to write distributed programs over outdoor networks of embedded systems? How to write distributed programs over outdoor networks of embedded systems?

4 Traditional (Indoor) Distributed Computing Commonly, computation is distributed for performance or fault tolerance Commonly, computation is distributed for performance or fault tolerance Relatively easy to program Relatively easy to program –Nodes are computationally equivalent –Networking is robust and has acceptable delays –Configuration is stable

5 Outdoor Distributed Computing Computation is physically distributed Computation is physically distributed –Nodes distributed across the physical space –Location-aware computing Hard to program Hard to program –Nodes are functionally heterogeneous and potentially mobile –Ad hoc wireless networks of embedded systems –Volatile configurations

6 Outdoor Programming Example Left Hill Right Hill Mobile robot with camera Motion Sensor Intrusion detection across the hills using motion sensors and autonomous mobile robots with cameras Intrusion detection across the hills using motion sensors and autonomous mobile robots with cameras Number and location of systems are unknown Number and location of systems are unknown Configuration is not stable over time Configuration is not stable over time –Intrusions can appear randomly –Robots can move

7 Traditional Distributed Computing Does Not Work Well Outdoors End-to-end data transfers may hardly complete End-to-end data transfers may hardly complete Fixed address naming and routing (e.g., IP) are too rigid Fixed address naming and routing (e.g., IP) are too rigid Difficult to deploy new applications in existing networks Difficult to deploy new applications in existing networks Outdoor distributed computing requires novel programming models and system architectures Outdoor distributed computing requires novel programming models and system architectures

8Outline Motivation Motivation Spatial Programming Model Spatial Programming Model Smart Messages System Architecture Smart Messages System Architecture Implementation Details Implementation Details Experimental Results Experimental Results Conclusions Conclusions

9 Traditional (Indoor) Programming Programs access data through variables Programs access data through variables Variables mapped to physical memory locations Variables mapped to physical memory locations Page Table and OS guarantee reference consistency Page Table and OS guarantee reference consistency Access time has an (acceptable) upper bound Access time has an (acceptable) upper bound Program Virtual Address Space Page Table + OS Physical Memory Variable access

10 From Indoor to Outdoor Computing Virtual Address Space Space Region Variables Spatial References Variables mapped to physical memory Spatial references mapped to systems embedded in the physical space Reference consistency Bounded access time ? ?

11 Spatial Programming (SP) at a Glance Provides a virtual name space over outdoor networks of embedded systems Provides a virtual name space over outdoor networks of embedded systems Embedded systems named by their locations and properties Embedded systems named by their locations and properties Runtime system takes care of name resolution, reference consistency, and networking aspects Runtime system takes care of name resolution, reference consistency, and networking aspects Implementation on top of Smart Messages: SP applications execute, sequentially, on each system referenced in their code Implementation on top of Smart Messages: SP applications execute, sequentially, on each system referenced in their code

12 radius Hill = new Space({lat, long}, radius); {lat,long} Space Regions Virtual representation of a physical space Virtual representation of a physical space Similar to a virtual address space in a conventional Similar to a virtual address space in a conventional computer system computer system Defined statically or dynamically Defined statically or dynamically

13 Spatial References Defined as {space:property} pairs Defined as {space:property} pairs Virtual names for embedded systems Virtual names for embedded systems Similar to variables in conventional programming Similar to variables in conventional programming Indexes used to distinguish among similar systems Indexes used to distinguish among similar systems in the same space region in the same space region {Hill:robot[0]} {Hill:robot[1]} {Hill:motion[0]} Hill r2 r7 m5

14 From Indoor to Outdoor Computing Virtual Address Space Space Region Variables Spatial References Variables mapped to physical memory Spatial references mapped to systems embedded in the physical space Reference consistency Bounded access time ? ?

15 Reference Consistency At first access, a spatial reference is mapped to an embedded system located in the specified space At first access, a spatial reference is mapped to an embedded system located in the specified space Mappings maintained in per-application Mapping Table (MT) – similar to a page table Mappings maintained in per-application Mapping Table (MT) – similar to a page table Subsequent accesses to the same spatial reference will reach the same system (using MT) as long as it is located in the same space region Subsequent accesses to the same spatial reference will reach the same system (using MT) as long as it is located in the same space region {space, property, index} {unique_address, location}

16 Reference Consistency Example Left Hill Right Hill {Left_Hill:robot[0]}.move = ON; r2 r7 r5 {Left_Hill:robot[0]}.move = OFF;

17 Space Casting {Left_Hill:robot[0]} Left Hill Right Hill {Right_Hill:(Left_Hill:robot[0])}r7

18 From Indoor to Outdoor Computing Virtual Address Space Space Region Variables Spatial References Variables mapped to physical memory Spatial references mapped to systems embedded in the physical space Reference consistency Bounded access time Mapping Table ?

19 Bounding the Access Time How to bound the time to access a spatial reference? How to bound the time to access a spatial reference? –Systems may move, go out of space, or disappear Solution: associate an explicit timeout with each spatial reference access Solution: associate an explicit timeout with each spatial reference access try{ {Hill:robot[0], timeout}.move = ON; {Hill:robot[0], timeout}.move = ON; }catch(TimeoutException e){ // the programmer decides the next action // the programmer decides the next action}

20 for(i=0; i<1000; i++) for(i=0; i<1000; i++) try{ try{ if ({Left_Hill:motion[i], timeout}.detect > Max_motion) if ({Left_Hill:motion[i], timeout}.detect > Max_motion) Max_motion = {Left_Hill:motion[i], timeout}.detect; Max_motion = {Left_Hill:motion[i], timeout}.detect; Max_id = i; Max_id = i; }catch(TimeoutException e) }catch(TimeoutException e) break; break; intrusionSpace = rangeOf({Left_Hill:motion[Max_id].location}, Range); intrusionSpace = rangeOf({Left_Hill:motion[Max_id].location}, Range); {intrusionSpace:robot[0]}.camera = ON; {intrusionSpace:robot[0]}.camera = ON; {intrusionSpace:robot[0]}.focus = {Left_Hill:motion[Max_id].location}; {intrusionSpace:robot[0]}.focus = {Left_Hill:motion[Max_id].location}; Spatial Programming Example - Find the sensor that detected the “strongest” motion on Left Hill - Turn on a camera in the proximity of this sensor Left Hill Right Hill

21Outline Motivation Motivation Spatial Programming Model Spatial Programming Model Smart Messages System Architecture Smart Messages System Architecture Implementation Details Implementation Details Experimental Results Experimental Results Conclusions Conclusions

22 “Dumb” Messages vs. “Smart” Messages Data migration Lunch: Lunch:AppetizerEntreeDessert Execution migration

23 Smart Messages at a Glance User-defined distributed applications User-defined distributed applications Composed of code bricks, data bricks, and execution control state Composed of code bricks, data bricks, and execution control state Execute on nodes of interest named by properties Execute on nodes of interest named by properties Migrate between nodes of interest Migrate between nodes of interest Self-route at every node in the path during migrations Self-route at every node in the path during migrations

24 Cooperative Node Architecture Network SMPlatform AdmissionManagerAdmissionManager Network SMSM Operating System & I/O CodeCacheCodeCache VirtualMachineVirtualMachine TagSpaceTagSpace SM Ready Queue AuthorizationAuthorization

25 SM Execution at a Node Takes place over a virtual machine Takes place over a virtual machine Non-preemptive, but time bounded Non-preemptive, but time bounded Ends with a migration, or terminates Ends with a migration, or terminates During execution, SMs can During execution, SMs can –Spawn new SMs –Create smaller SMs out of their code and data bricks –Access the tag space –Block on a tag to be updated (update-based synchronization)

26 Tag Space Collection of application tags and I/O tags Collection of application tags and I/O tags –Essentially, tags are (name, value) pairs Application tags: persistent memory across SM executions Application tags: persistent memory across SM executions I/O tags: access to operating system and I/O subsystem I/O tags: access to operating system and I/O subsystem Tags used for Tags used for –Content-based naming migrate(tag) –Inter-SM communication write(tag, data), read(tag) –Synchronization block(tag, timeout) –I/O access read(temperature)

27 Protection Domains for Tag Space Owner: SM that creates the tag Owner: SM that creates the tag Family: all SMs having a common ancestor with the SM that owns the tag Family: all SMs having a common ancestor with the SM that owns the tag Origin: all SMs created on the same node as the family originator of the tag owner Origin: all SMs created on the same node as the family originator of the tag owner Code: all SMs carrying a specific code brick Code: all SMs carrying a specific code brick Others: all the other SMs Others: all the other SMs Family Code Origin Others Owner

28 Access Control Example (Code-based protection domain) Node5 Node3 Node4 SM2 C2 Cr Node1 Node2 Code Bricks Tag Owner = SM1 [Hash(Cr), RW] SM1 C1 Cr SM1 SM3 C3 C4 Access permission granted for SM2 Access permission denied for SM3 Cr Same routing used by SM1 and SM2

29 SM Admission Ensures progress for all SMs in the network Ensures progress for all SMs in the network Prevents SMs from migrating to nodes that cannot provide enough resources Prevents SMs from migrating to nodes that cannot provide enough resources SMs specify lower bounds for resource requirements (e.g., memory, bandwidth) SMs specify lower bounds for resource requirements (e.g., memory, bandwidth) SMs accepted if the node can satisfy these requirements SMs accepted if the node can satisfy these requirements More resources can be granted according to admission policy More resources can be granted according to admission policy –If not granted, SMs are allowed to migrate

30 Application Example n=0 while (n<NumTaxis) migrate(Taxi); migrate(Taxi); if (readTag(Available)) if (readTag(Available)) writeTag(Available, false); writeTag(Available, false); writeTag(Location, myLocation); writeTag(Location, myLocation); n++; n++; n=0 Taxi Taxi n=0n=0n=1n=1n=2 need 2 taxis data brick application code brick routing SM

31 migrate() migrate() – migrates application to next node of interest – names nodes by tags – implements routing algorithm sys_migrate() sys_migrate() – one hop migration – used by migrate to implement routing SM Migration migrate(Taxi) sys_migrate(2) sys_migrate(3) sys_migrate(4) TaxiTaxi 1 234

32 Network Routing Example 12i RouteToTaxi = 2 Taxi RouteToTaxi = ?RouteToTaxi = j migrate(Taxi){ while(readTag(Taxi) == null) if (readTag(RouteToTaxi)) sys_migrate(readTag(RouteToTaxi)); else create_SM(DiscoverySM, Taxi); createTag(RouteToTaxi, lifetime, null); block_SM(RouteToTaxi, timeout); }

33 SM Self-Routing SMs carry the routing and execute it at each node SMs carry the routing and execute it at each node Routing information stored in tag space Routing information stored in tag space SMs control their routing SMs control their routing –Select routing algorithm (migrate primitive) Multiple library implementations Multiple library implementations Implement a new one Implement a new one –Change routing algorithm during execution in response to Adverse network conditions Adverse network conditions Application’s requirements Application’s requirements

34 Self-Routing Simulation Results starting node node of interest other node Geographic + On-Demand Routing performs better than simple on-demand routing 25%-40% decrease in completion time 62%-92% reduction in amount of bytes sent in the network - Less energy and bandwidth consumption

35Outline Motivation Motivation Spatial Programming Model Spatial Programming Model Smart Messages System Architecture Smart Messages System Architecture Implementation Details Implementation Details Experimental Results Experimental Results Conclusions Conclusions

36 Spatial Programming Implementation Using Smart Messages SP application translates into an SM SP application translates into an SM –Spatial reference access translates into an SM migration to the mapped node –Embedded system properties: Tags SM self-routes using geographical routing and content-based routing SM self-routes using geographical routing and content-based routing Reference consistency Reference consistency − Unique addresses (stored in mapping table) are unique tags created at nodes − SM carries the mapping table

37 Max_motion = {Left_Hill:motion[1], timeout}.detect; Max_motion = {Left_Hill:motion[1], timeout}.detect; SP to SM Translation: Example Spatial Reference Access Smart Message ret = migrate_geo(location, timeout); if (ret == LocationUnreachable) ret = migrate_tag(yU78GH5, timeout); ret = migrate_tag(yU78GH5, timeout); if ((ret == OK) && (location == Left_Hill)) return readTag(detect); return readTag(detect); else throw TimeoutException {Left_Hill,motion,1}{yU78GH5,location} MappingTable CodeBrick Left Hill Right Hill

38 SM Prototype Implementation Modified version of Sun’s Java K Virtual Machine Modified version of Sun’s Java K Virtual Machine –Small memory footprint (160KB) SM and tag space primitives implemented inside virtual machine as native methods (efficiency) SM and tag space primitives implemented inside virtual machine as native methods (efficiency) I/O tags: GPS location, neighbor discovery, image capture, light sensor, system status I/O tags: GPS location, neighbor discovery, image capture, light sensor, system status Ad hoc networks of HP iPAQ PDAs running Linux

39 Lightweight Migration Traditional process migration difficult Traditional process migration difficult –Strong coupling between execution entity and host –Needs to take care of operating system state (e.g., open sockets, file descriptors) Tag space decouples the SM execution state from the operating system state Tag space decouples the SM execution state from the operating system state SM migration transfers only SM migration transfers only –Data bricks explicitly specified by programmer –Minimal execution control state required to resume the SM at the next instruction (e.g., instruction pointer, operand stack pointer)

40Outline Motivation Motivation Spatial Programming Model Spatial Programming Model Smart Messages System Architecture Smart Messages System Architecture Implementation Details Implementation Details Experimental Results Experimental Results Conclusions Conclusions

41 Micro-benchmark Results Cost of data serialization Cost of single hop migration

42 Experimental Results for Simple Routing Algorithms Completion Time Routing algorithmCode not cached (ms)Code cached (ms) Geographic On-demand user node node of interest intermediate node 8 HP iPAQs running Linux and using IEEE for wireless communication

43 SP Application: Intrusion Detection 10 HP iPAQs with cards and GPS devices user node light sensor camera node regular node monitoredspace

44Conclusions Spatial Programming makes outdoor distributed computing simple Spatial Programming makes outdoor distributed computing simple –Volatility, mobility, configuration dynamics, ad hoc networking are hidden from programmer Implementation on top of Smart Messages Implementation on top of Smart Messages –Easy to deploy new applications in the network –Quick adaptation to highly dynamic network configurations Acknowledgments: Liviu Iftode, Ulrich Kremer, Chalermek Intanagonwiwat, Porlin Kang Acknowledgments: Liviu Iftode, Ulrich Kremer, Chalermek Intanagonwiwat, Porlin Kang

45 Other projects based on Smart Messages Portable Smart Messages: work over unmodified JVMs using bytecode instrumentation (done by Nishkam) Portable Smart Messages: work over unmodified JVMs using bytecode instrumentation (done by Nishkam) – –Tested on iPAQs, Ericsson P900, and Nokia Communicator Context-aware Migratory Services in Ad Hoc Networks (Oriana will present it after this talk) Spatial Views: A location-aware, declarative programming language (Uli Kremer will present it on Friday)

46 Two of my current projects Intelligent distributed transportation systems – –Sponsored by NSF, collaboration with Rutgers University (Pravin will talk about it in the afternoon) – –Design a vehicular network architecture and build a prototype for distributed computing over vehicular networks SmartCampus – –Sponsored by NSF, joint work with IS & ECE departments at NJIT – –Investigate mobile computing applications, protocols, middleware over a campus-wide, location-aware, mobile community test-bed (100s of users with mobile devices)

47 Thank you!