1 Scalable Peer-to-Peer Virtual Environments Shun-Yun Hu ( 胡舜元 ) CSIE, National Central University, Taiwan 2008/05/08

2

4

Massively Multiplayer Online Games MMOGs are growing quickly Multi-billion dollar industry 10 million subscribers for World of Warcraft 600,000 concurrent users, but 3,000 per world Can we scale to millions in the same world?

Imagine you start with a globe

Zoom in…

To Phoenix..

and Hyatt..

Right now it’s flat…

But in the near future…

Virtual Environments (VEs): A Shared Space

Issues for Creating VEs Consistency Interactivity multiplayer Security Scalability Persistency massively multiplayer Reliability

15 Model for virtual worlds Many nodes on a 2D plane Message exchange with those within Area of Interest (AOI)‏ How does each node receive the relevant messages? Area of Interest

16 A simple solution (point-to-point)‏ But…too many irrelevant messages N * (N-1) connections ≈ O(N 2 )  Not scalable! Source: [Funkhouser95]

17 A better solution (client-server)‏ Message filtering at server to reduce traffic N connections = O(N)  server is bottleneck Source: [Funkhouser95]

18 Current solution (server-cluster)‏ Still limited by servers. Expensive to deploy & maintain. Source: [Funkhouser95]

The Problem Client-server: resources limited by provisioning Resource limit [Funkhouser95]

The Solution Peer-to-Peer: resources grow with demand Resource limit [Keller & Simon 2003]

Voronoi-based Overlay Network (VON)‏

Design Goals Observation: for virtual environment applications, the contents we want are messages from AOI neighbors Content discovery is a neighbor discovery problem Specific goals: Scalable  Limit per-node message traffic Responsive  Direct connection with AOI neighbors

23 If you talk with your AOI Neighbors directly… But how to discover new neighbors?

24 Voronoi Diagram 2D Plane partitioned into regions by sites, each region contains all the points closest to its site Can be used to find k-nearest neighbor easily Neighbors Site Region

25 Design Concepts Identify enclosing and boundary neighbors Each node constructs a Voronoi of its neighbors Enclosing neighbors are minimally maintained Mutual collaboration in neighbor discovery boundary neighbor (B.N.)‏L. Blue unknown neighborL. Green normal AOI neighborGreen E.N. & B.N.Pink enclosing neighbor (E.N.)‏Yellow selfWhite Area of Interest (AOI)‏Circle Use Voronoi to solve the neighbor discovery problem

26 Procedure (JOIN)‏ 1)Joining node sends coordinates to any existing node Join request is forwarded to acceptor 2)Acceptor sends back its own neighbor list joining node connects with other nodes on the list Acceptor’s region Joining node

27 Procedure (MOVE)‏ 1)Positions sent to all neighbors, mark messages to B.N. B.N. checks for overlaps between mover’s AOI and its E.N. 2)Connect to new nodes upon notification by B.N. Disconnect any non-overlapped neighbor Boundary neighbors New neighbors Non-overlapped neighbors

28 Procedure (LEAVE)‏ 1)Simply disconnect 2)Others then update their Voronoi new B.N. is discovered via existing B.N. Leaving node (also a B.N.)‏ New boundary neighbor

29 Demonstration Simulation demo Random movements (100 nodes, 1200x700 world)‏ Local vs. global view Dynamic AOI adjustment

Simulation Method C++ implementation of VON (open source VAST library) World size: 1200 x 1200 (AOI: 100) Trials from 200 – 2000 nodes Connection limit: time-steps (~ 300 simulated seconds, assuming 10 updates/seconds) Behavior model Random movement:random destination Constant velocity: 5 units/step Movement duration: random (until destination is reached)

Scalability: Avg. Transmission / sec

Scalability: Max. Transmission / sec

Scalability: Avg. Neighbor Size

Reliability: Effects of Packet Loss

Voronoi State Management (VSM)‏

36 Voronoi State Management VON deals only with positions, but we want to manage states stored in spatial objects (with x, y). Let game states be managed by all clients Each client has two roles: peers & arbitrators i.e. Voronoi partitioning Three problems: O(n 2 ) connections at hotspots Some cells have large sizes Constant ownership transfer

37 VSM: solution ideas Connection overload→ Aggregators clustering Large cell-size → Virtual peers incremental transfer Constant transfers→ Explicit ownership transfer

38 VSM: Consistency control Managing arbitrator receives and processes events Events are forwarded if necessary Updates sent to affected arbitrators, then peers

39 VSM: Load balancing Traditional: high-capacity nodes first, then adjust VSM: low-capacity nodes first, then cluster Overload: ask for aggregator, submit control Underload: disintegrate, release control

40 VSM: Load balancing (2)‏ Sphere of control adjustable More than one aggregator → choose nearest

41 VSM: Fault tolerance Regular arbitrator: Pick backup arbitrator, backup states Backup transfers ownership to enclosing arbitrators Aggregators: Pick backup aggregators Take over original if failed Choose new backup

Peer-to-Peer 3D Streaming

43 Background MMOGs today need CD installation (too slow!)‏ But 3D data is huge Content streaming is needed 80% - 90% content is 3D (e.g., 3D streaming)

44 3D streaming Object streaming [Hoppe 1996] base refinements Scene streaming [Teler & Lischinski 2001] multiple objects object selection & prioritization

45 3D streaming vs. media streaming Video / audio media streaming is very matured User access patterns are different for 3D content Highly interactive  Latency-sensitive Behaviour-dependent  Non-sequential Analogy Constant & frequent switching of multiple channels

46 Benefits of peer-to-peer Scalable Growing amount of total resources Affordable Commodity PCs Feasible Better client hardware (CPU, broadband networks)‏ Availability of user-hosted machines

Challenges for P2P 3D streaming Appropriate peer grouping Matching interests / needs Matching capabilities Dynamic group management Interest groups are dynamic(non-sequential) Real-time constraints(latency-sensitive) Minimal server involvement Visibility determination (object selection) Request prioritization (piece selection)

48 Observation Limited & predictable area of interest (AOI)‏ Overlapped visibility = shared content

49 overlapped visibility = shared content

50 Download content from AOI neighbors star: selftriangles: neighbors circle: AOIrectangles: objects

51 Neighbor discovery via VON Boundary neighbors New neighbors Non-overlapped neighbors [Hu et al. 06] Voronoi diagrams identify boundary neighbors for neighbor discovery

52 Prototype experiment Progressive models in a scene (by NTU) Peer-to-peer AOI neighbor requests(by NCU) Found matching client upload / download

53 Simulation setup Environment 1000x1000 world, 100ms / step, 3000 steps client: 1 Mbps / 256 Kbps, server: 10 Mbps (both)‏ Objects Random object placement (500 objects)‏ Object size based on prototype User behavior Random & clustering movement (1.5 * ln(n) hotspots)‏

54 Server bandwidth usage

55 Client bandwidth usage

56 Fill ratio

57 Base latency

Impacts of P2P VEs… No server as bottleneck  scalable Commodity hardware  affordable 2D web  3D web Earth-scale virtual worlds (millions/billions of people)‏

59 Q&A VON: A Scalable Peer-to-Peer Network for Virtual Environments IEEE Network, vol. 20, no. 4, Jul./Aug FLoD: A Framework for Peer-to-Peer 3D Streaming IEEE INFOCOM, Apr Thank you!

60 Prototype experiment Data 3D scene converted from a game demo Setup 100 Mbps LAN 10 participants, 48 logins captured in 40 min. Results Found matching client upload & download Avg. server request ratio (SRR): 36%

61 The flow of FLoD Prefetching:not considered Prioritization:visual importance & view-dependency Peer & piece selection:query-response, random peer, sequential piece

63 Client bandwidth usage (cluster)‏

64 Effect of user density

65 Effect of upload bandwidth

66 Cache utilization

67 Collaborations With Publications Guan-Ming Liao(Actainment Co., Taiwan)‏ Dr. Jui-Fa Chen, Tsu-Han Chen(TKU, Taiwan)‏ Dr. Bing-Yu Chen, Ting-Hao Huang(NTU, Taiwan)‏ Dr. Jehn-Ruey Jiang(NCU, Taiwan)‏ 3 graduated & 3 current masters(NCU, Taiwan)‏ On-going Dr. Pedro Morillo TenaVON(Universitat de València, Spain)‏ Dr. Wei Tsang OoiFLoD(NUS, Sinagpore)‏ Dr. Maha AbdallahHeaven(Université Paris 6, France)‏ Dr. Gregor SchieleMMVE(Univ. of Mannheim, Germany)‏

68 Voronoi-based Overlay Network Boundary neighbors New neighbors Non-overlapped neighbors VON: A Scalable Peer-to-Peer Network for Virtual Environments IEEE Network, vol. 20, no. 4, Jul./Aug Voronoi diagrams identify boundary neighbors for neighbor discovery

69 FLoD (with CM Lab, CSIE, NTU)‏ FLoD: A Framework for Peer-to-Peer 3D Streaming IEEE INFOCOM, Apr Drastic server bandwidth reduction First P2P 3D streaming prototype