1 VON: A Scalable Peer-to-Peer Network for Virtual Environments Shun-Yun Hu ( 胡舜元 ) CSIE, National Central University, Taiwan 2007/02/15

2 Outline Introduction Voronoi-based Overlay Network (VON) Evaluation Application for physical simulation Conclusion

3 What is Networked Virtual Environment? Virtual Reality + Internet 3D worlds with people, agents, objects, terrain Military simulations (’80)  Massively Multiplayer Online Games (mid-‘90) Trends: larger scale, more realistic simulation

4

5 NVE: A Shared Space

6 The Scalability Problem Many nodes on a 2D plane ( > 1,000) Message exchange with those within Area of Interest (AOI) How does each node receive the relevant messages? Area of Interest

7 A simple solution (point-to-point) But…too much irrelevant message N * (N-1) connections ≈ O(N 2 )  Not scalable! Source: [Funkhouser95]

8 A better solution (client-server) Message filtering at server to reduce traffic N connections = O(N)  server is bottleneck Source: [Funkhouser95]

9 Current solution (server-cluster) Still limited by servers. Expensive to deploy & maintain. Source: [Funkhouser95]

10 Scalability Analysis Scalability constrains Computing resource(CPU) Network resource(Bandwidth) Non-scalable system vs. Scalable system x: number of entities y: resource consumption at the limiting system component Resource limit

11 What Next? Strategies Increase resource  More servers Decrease consumption  Message filtering ArchitecturesScale Point-to-point (LAN)tens10^1 Client-serverhundreds10^2 Server-clusterthousands10^3 ?millions 10^6 … Peer-to-Peer

12 What is Peer-to-Peer (P2P)? [Stoica et al. 2003] Distributed systems without any centralized control or hierarchical organization Runs software with equivalent functionality Examples File-sharing: Napster, Gnutella, Kazza, eDonkey Distributed Search: Chord, CAN, Tapestry, Pastry VoIP:Skype

13 Peer-to-Peer Overlay A P2P overlay network source: [Keller & Simon 2003]

14 Promise & Challenge of P2P Promises Growing resource, decentralized  Scalable Commodity hardware  Affordable Challenges Topology maintenance  dynamic join/leave Efficient content retrieval  no global knowledge

15 Outline Introduction Voronoi-based Overlay Network (VON) Evaluation Application for physical simulation Conclusion

16 Design Goals Observation: for virtual environment applications, the contents we want are messages from AOI neighbors Content discovery is a neighbor discovery problem Solve the Neighbor Discovery Problem in a fully- distributed, message-efficient manner. Specific goals: Scalable  Limit per-node message traffic Responsive  Direct connection with AOI neighbors

17 Voronoi Diagram 2D Plane partitioned into regions by sites, each region contains all the points closest to its site Can be used to find k-nearest neighbor easily Neighbors Site Region

18 Design Concepts Identify enclosing and boundary neighbors Each node constructs a Voronoi of its neighbors Enclosing neighbors are minimally maintained Mutual collaboration in neighbor discovery CircleArea of Interest (AOI) Whiteself Yellowenclosing neighbor (E.N.) L. Blueboundary neighbor (B.N.) PinkE.N. & B.N. GreenAOI neighbor L. Greenunknown neighbor Use Voronoi to solve the neighbor discovery problem

19 Procedure (JOIN) 1)Joining node sends coordinates to any existing node Join request is forwarded to acceptor 2)Acceptor sends back its own neighbor list joining node connects with other nodes on the list Acceptor’s region Joining node

20 Procedure (MOVE) 1)Positions sent to all neighbors, mark messages to B.N. B.N. checks for overlaps between mover’s AOI and its E.N. 2)Connect to new nodes upon notification by B.N. Disconnect any non-overlapped neighbor Boundary neighbors New neighbors Non-overlapped neighbors

21 Procedure (LEAVE) 1)Simply disconnect 2)Others then update their Voronoi new B.N. is discovered via existing B.N. Leaving node (also a B.N.) New boundary neighbor

22 Dynamic AOI Crowding within AOI can overload a particular node It’s better if AOI-radius can be adjusted in real time

23 Adjustment Conditions AOI-radius decrease Number of connections > connection limit AOI-radius increase Maximum connections not exceeded Current AOI-radius < preferred AOI-radius Mutual awareness rule Do not disconnect a neighbor who sees me

24 Demonstration Simulation demo Random movements (100 nodes, 1200x700 world) Local vs. global view Dynamic AOI adjustment

25 Outline Introduction Voronoi-based Overlay Network (VON) Evaluation Application for physical simulation Conclusion

26 Simulation Method C++ implementation of VON (open source VAST library) World size: 1200 x 1200 (AOI: 100) Trials from 200 – 2000 nodes Connection limit: time-steps (~ 300 simulated seconds, assuming 10 updates/seconds) Behavior model Random movement:random destination Constant velocity: 5 units/step Movement duration: random (until destination is reached)

27 Scalability: Avg. Transmission / sec

28 Scalability: Max. Transmission / sec

29 Scalability: Avg. Neighbor Size

30 Reliability: Effects of Packet Loss

31 Analysis of Design Scalability Bounded resource consumption  dynamic AOI Consistency (Topology) Topology is fully connected  enclosing neighbors Reliability Self-organizing  distributed neighbor discovery Responsiveness Lowest latency  direct connection, no relay

32 Outline Introduction Voronoi-based Overlay Network (VON) Simulation results Application for physical simulation Conclusion

33 A look at simulations Important tools in scientific research Larger scale and higher resolution are constantly sought However, computational resource can be limited An Untapped Potential 300 Million PCs on the Internet (2000 est.) Up to 80% to 90% of CPU is wasted Large supply of computing resource, growing rapidly

34 Examples (UC Berkeley – space radio analysis) 5.3 M world-wide participants 2.2 M years of single-processor CPU 54 teraflop machine (top 3 in 2005: 70.72, 51.87, 35.86) (Stanford – protein’s 3D structure) 30,000 volunteers 1 M days of single-processor CPU Published 23 papers in: Science, Nature, Nature Structural Biology, PNAS, JMB, etc.

35 The Grand Question Can we build the ultimate simulator for large-scale simulation utilizing millions of computers world-wide? Potential applications: Nuclear reaction Star clusters Atomic-scale modeling in material science Weather, earthquakes Biology (protein, ecosystem, brain,...)

36 Current Limitations Current methodology Centralized server + many clients Client requests “work unit” to process Communication is minimized Clients do not communicate Issues: Only suitable for “embarrassingly parallel” simulations Sophisticated server-side algorithm and management required An alternative: peer-to-peer (P2P) computing [Hori et al. 2001]

37 A Simulation Scenario How can we utilize P2P for simulation-purpose? Answer: depends on what you want to simulate We observe that many simulations… are spatially-oriented (i.e. based on coordinate systems) run in discrete time-steps exhibit localized interaction (i.e. short-range interaction) example: molecular dynamics (MD) simulation Protein folding?

38 Outline Introduction Voronoi-based Overlay Network (VON) Simulation results Application for physical simulation Conclusion

39 Summary NVE scalability is achievable with P2P architecture and is a neighbor discovery problem A promising solution: Voronoi-based P2P Overlay Leverage knowledge of each peer to maintain topology Properties Scalable: fully-distributed, dynamic AOI Efficient: low irrelevant messages, zero relay Simple: simple protocol and procedure

40 Potential Applications Online games Position updates in current MMOGs, Voice-chats Military Enable large-scale, affordable military training simulation 3D Web Provide multi-user interactivity to static 3D world Scientific simulations Distribute spatial simulation requiring frequent synchronization

41 Acknowledgements Dr. Jui-Fa Chen ( 陳瑞發老師 ) Tsu-Han Chen( 鄭子涵 ) Members of the Alpha Lab, TKU CS Dr. Chin-Kun Hu( 胡進錕老師 ) Guan-Ming Liao( 廖冠名 ) LSCP, Institute of Physics, Academia Sinica Joaquin Keller(France Tele. R&D, Solipsis) Jon Watte(there.com) Kuan-Ta Chen( 陳寬達, NTU)

42 Q&A VON: A Scalable Peer-to-Peer Network for Virtual Environments IEEE Network, vol. 20, no. 4, Jul./Aug Thank you! (

43 Issues for Creating NVE Consistency (events/states) Responsiveness multiplayer Security Scalability Persistency massively multiplayer Reliability (Fault-tolerance)

44 Issues for Creating P2P NVE Consistency (events/states) Responsiveness multiplayer Security Scalability Persistency massively multiplayer Reliability (Fault-tolerance) Consistency (topology)  P2P NVE

45 Server-cluster issues Insufficient total resource Hardware provisioning  over-provision! High user density (crowding) User limits  limits scale & realism! Excessive inter-server communications Less load balancing  difficult balance!

46 Related Work (1): DHT-based: SimMUD [Knutsson et al. 2004] (UPenn) Pastry + Scribe Regions Coordinators (super-nodes) Fixed-size region Relay overhead

47 Related Work (2): Neighbor-list Exchange [Kawahara et al. 2004] (Univ. of Tokyo) Fully-distributed Nearest-neighbors List exchange High transmission Overlay partition

48 Related Work (3): Mutual Notification: Solipsis [Keller & Simon 2003] (France Telecomm R&D) Links with AOI neighbor Mutual cooperation Inside convex hull Potentially slow discovery Inconsistent topology

49 Consistency Metrics Topology Consistency [Kawahara, 2004] observed AOI neighbors actual AOI neighbors Drift Distance [Diot, 1999] Distance between observed position and actual position (average over all nodes)

50 Consistency: Topology Consistency

51 Consistency: Drift Distance

52 Problems of Voronoi Approach Message traffic Circular round-up of nodes Redundant message sending (inherent to fully-distributed design) Incomplete neighbor discovery Can happen with inconsistent / incorrect neighbor list Fast moving node Limited AOI Direct connections

53 P2P NVE Comparisons DHT- based Neighbor-list exchange SolipsisVON Consistency (topology) DHT & Supernode (consistent) Neighbor list- exchange (partitioning) Neighbor notify&query (undiscovery) Neighbor notify (consistent) Responsive- ness two to many One hop ScalabilityO(n) on supernode Constant in crowding Constant if fixed density Constant in crowding ConLatency too high Overlay partitioning Occasional undiscovery Circular node line-up

54 Future Perspectives Short-term Distributed event/state consistency Customizable AOI (Heterogeneity in P2P) Recovery from overlay partition and fast-moving nodes Long-term Persistency issue(P2P-based database) Security issue (protection from malicious nodes) 3D content distribution (3D streaming on P2P) Massive, persistent 3D environment sharable by all!