1 TI1100A – Internet-Based Applications – A. Iosup Alexandru Iosup Parallel and Distributed Systems Group Delft University of Technology Our team: Undergrad Adrian Lascateu, Alexandru Dimitriu (UPB, Romania), Saleem Anwar (Vrije Universiteit, the Netherlands), …, Grad Vlad Nae (U. Innsbruck, Austria), Nezih Yigitbasi (TU Delft, the Netherlands), Staff Dick Epema, Johan Pouwelse, Henk Sips (TU Delft), Thomas Fahringer, Radu Prodan (U. Innsbruck), Nicolae Tapus, Mihaela Balint, Vlad Posea (UPB), Guillaume Pierre (Vrije U.). Internet-Based Distributed Systems

Internet: A Distributed System TI1100A – Internet-Based Applications – A. Iosup 2

3 Internet-Based Applications Source:

TI1100A – Internet-Based Applications – A. Iosup 4 Agenda 1.Internet: An Ongoing Story 2.Internet-Based Applications (part 1) Peer-to-Peer File-Sharing: Gnutella, KaZaa, BitTorrent 3.Internet-Based Applications (part 2) Massively Social Gaming: WoW, RuneScape, FarmVille

TI1100A – Internet-Based Applications – A. Iosup 5 The Internet: How it Works Routing Domain Name Servers IP address to name = Google.com Since 1984 No central authority ICANN Coordinates assignment of unique identifiers: domain names, IP addresses, … wlan.ewi.tudelft.nlgoogle.com

TI1100A – Internet-Based Applications – A. Iosup 6 The Internet: Early History ARPANET Leonard Kleinrock develops the Queueing Theory (theoretical properties of the Internet) 4 computers at UC Santa Barbara UC Los Angeles Stanford Research Inst. U of Utah 1972 ARPANET public TCP/IP at Stanford 1982 ARPANET + TCP/IP = the (early) Internet Drawing by Alex McKenzie, Dec 1969

TI1100A – Internet-Based Applications – A. Iosup 7 The Internet Today net, ca, us com, org mil, gov, edu asia de, uk, it, fr, pl br, kr, nl unknown Source:

Dutch Research Internet SURFnet connects ~200 locations (~750K users): - universities - academic hospitals - Polytechnic schools - research centers - 6,000km connections ~ Dutch Railway system DAS-4DAS-4 Sources: Cees te Laat and Henri Bal

EU Research Internet

TI1100A – Internet-Based Applications – A. Iosup 10 ABILENE: Backbone Research Network Test: Land Speed Record ~ 7 Gb/s in single TCP stream from Geneva to Caltech Source: MonALISA monitoring framework, 2005

TI1100A – Internet-Based Applications – A. Iosup 11 Agenda 1.Internet: An Ongoing Story 2.Internet-Based Applications (part 1) Peer-to-Peer File-Sharing: Gnutella, KaZaa, BitTorrent 3.Internet-Based Applications (part 2) Massively Social Gaming: WoW, RuneScape, FarmVille

Peer-to-Peer: Aggregate use of distributed resources Resources at the "edge" of the internet PCs are the dark matter of the internet Cheap or near-zero cost In aggregate, these resources are valuable Difficult to aggregate using traditional models P2P applications = novel ways of aggregating resources

Potential P2P Characteristics Cost structure Setup Operate Scalability (vs Client/Server) Availability (efficiency + reliability) Fault-tolerance (recovers from errors) Self-organization (deals with dynamic activities) Other, application-specific benefits Control over network Quality of Service

P2P file sharing Popular application 75 % of EU Broadband users use P2P every month, Jupiter Media 35M EU have downloaded music from file sharing service, Forrester Research Various generations Napster, Gnutella / Kazaa, eMule, BitTorrent Get files Movies / TV shows (DivX) Music (MP3) Games / Applications (ISO)

File Sharing Evolution -1990V1.0 Floppy & Tape 1990sV2.0 FTP & Web servers 1999V3.0 Napster 2001V4.0 Gnutella / Kazaa 2003V5.0 Bittorrent 2005V6.0 Azureus V7.0 Tribler at TU Delft

Internet P2P backbone Bandwidth 2004

Internet P2P backbone Bandwidth 2006

Generic model of P2P file sharing searchdownload Rating & Moderation off-line inject idle

TI1100A – Internet-Based Applications – A. Iosup 19 P2P History: Napster

P2P History: Napster, Central File List napster.com users File Xfer: HTTP

P2P History: Napster May 1999: Shawn Fanning founds Napster Dec 1999: first lawsuit Mar 2000: 25% university traffic 2000: est. 60M users Feb 2001: US Circuit Court of Appeals: Napster knew users violating copyright laws Jul 2001: # simultaneous online users: Napster 160K, Gnutella: 40K, Morpheus: 300K Jan 2004: 66 % of Internet bandwidth (in selected networks)

TI1100A – Internet-Based Applications – A. Iosup 22 P2P History: Gnutella

P2P History: Gnutella, Flooding Requests and results File Xfer: HTTP

TI1100A – Internet-Based Applications – A. Iosup 24 P2P History: Gnutella 2000: Justin Frankel and Tom Pepper (Nullsoft—Winamp) Mar 14, 2000: Announced on Slashdot Mar 15, 2000: AOL bans Gnutella 2001: Limewire Basic open-source 2002: Gnucleus open-source Jun 2005: 1.81M active users Jan 2006: 3M active users Ultrapeers + Leaves (not all P2P) Still alive! Source: Rasti, Stutzbach, Rejaie

Fake files

TI1100A – Internet-Based Applications – A. Iosup 26 P2P History: KaZaa

Kazaa problems Every download == upload – Bandwidth is a resource – 80% of Gnutella people do not share Fake files – Popular films/MP3s/Games – Renames of content Limited search capability – Supernodes do not scale – Napster-like indexing – Low-quality metadata

Taking Down KaZaa: Fake Files Deliberate attack Patented Junk files Virus (search for.pif.exe.bat etc.)‏ Everyone can do it Unmoderated My Shared Folder Multiple machines Problems: – Hard to find – Download and delete – Irritates users

TI1100A – Internet-Based Applications – A. Iosup 29

Bittorrent & Suprnova.org Suprnova.org Search inject Rating & Moderation Bittorrent – Bram Cohen, 2004 Download searchdownload Rating & Moderation off-line upload idle

BitTorrent overview peer tracker peer Suprnova.org LotR.torrent

Bartering for bytes seed leecher medium download speed low download speed high download speed Download speed: upload-bandwidth constrained Peer selection policy: with highest bandwidth first Piece selection policy: rarest first

TI1100A – Internet-Based Applications – A. Iosup 35 BitTorrent: Is It Legal? Yes, for freely-distributable content Most often not, for copyrighted content (even for dead swarms) Note: Watch the legal battles on this topic!

TI1100A – Internet-Based Applications – A. Iosup 36 Tribler: BitTorrent Compatible

Tribler P2P Research

Bittorrent in the real-world 100 DAS2 nodes (1-Ghz Pentium-IIIs, 1 GB RAM)‏ 8-month traces of more than 2,000 global components Complete lifetime of a popular file (90,000 peers)‏ Uptime measurement of 55,000 peers 150 GB of collected data

Peer availability 3 hours 1 day 3 days

Overall system activity Number of active users in the system is strongly influenced by the availability of the global components in BitTorrent/Suprnova

TI1100A – Internet-Based Applications – A. Iosup 42 Summary on P2P File-Sharing Use the resources at the “edge” of the Internet Many iterations Napster, Gnutella, KaZaa, BitTorrent, … Tribler is a BitTorrent-compatible P2P File-Sharing client Developed at TU Delft Dutch and EU grants Dutch Govt. Innovation Award Various research awards P2P File-Sharing is at the center of legal debates over sharing of content over the Internet (get involved!)

TI1100A – Internet-Based Applications – A. Iosup 43 Agenda 1.Internet: An Ongoing Story 2.Internet-Based Applications (part 1) Peer-to-Peer File-Sharing: Gnutella, KaZaa, BitTorrent 3.Internet-Based Applications (part 2) Massively Social Gaming: WoW, RuneScape, FarmVille

Why Social Gaming? Massivizing Social Games: Distributed Computing Challenges and High Quality Time – A. Iosup 44

Massivizing Social Games: Distributed Computing Challenges and High Quality Time – A. Iosup 45 Online Gaming used to be multimedia, is now DC Online Gaming used to be networking, is now all DC Online Gaming used to be v-worlds, is now many apps Online Gaming used to be art, may now be computing Massivizing Social Gaming = Rich Challenge (of Distr.Sys.)

Massivizing Social Games: Distributed Computing Challenges and High Quality Time – A. Iosup 46 What’s in a name? MSG, MMOG, MMO, … 1.Virtual World Sim Explore, do, learn, socialize, compete + 2.Game Data Player stats and relationships, others + 3.Game Content Graphics, maps, puzzles, quests, culture Massively Social Gaming = (online) games with massive numbers of players (100K+), for which social interaction helps the gaming experience Over 250,000,000 active players

Massivizing Social Games: Distributed Computing Challenges and High Quality Time – A. Iosup 47 Bungie, Computing then Serving 1.4PB/yr. Halo 3 is one of the many successful games Halo 3 players get, in 1.4PB Detailed player profiles Detailed usage stats Ranking CERN produces ~15PB/year (10x larger) (Not) faster than the speed of light, the Higgs boson (?)

Massivizing Social Games: Distributed Computing Challenges and High Quality Time – A. Iosup 48 World of Warcraft, a Traditional HPC User (since 2003) 10 data centers 13,250 server blades, 75,000+ cores 1.3PB storage 68 sysadmins (1/1,000 cores)

Massivizing Social Games: High Performance Computing and High Quality Time – A. Iosup 49 Zynga, an Amazon WS User Sources: CNN, Zynga. Source: InsideSocialGames.com “Zynga made more than $600M in 2010 from selling in-game virtual goods.” S. Greengard, CACM, Apr 2011 Selling in-game virtual goods: “Zynga made est. $270M in 2009 from.”

Massivizing Social Games: Distributed Computing Challenges and High Quality Time – A. Iosup 50 MSGs are a Popular, Growing Market 25,000,000+ subscribed players (from 250,000,000+ active) Over 10,000 MSGs in operation Subscription market size $7.5B+/year, Zynga $600M+/year Sources: MMOGChart, own research.Sources: ESA, MPAA, RIAA.

TI1100A – Internet-Based Applications – A. Iosup 51 Agenda 1.Internet: An Ongoing Story 2.Internet-Based Applications (part 1) Peer-to-Peer File-Sharing: Gnutella, KaZaa, BitTorrent 3.Internet-Based Applications (part 2) Massively Social Gaming: WoW, RuneScape, FarmVille 1.What’s in a Name? 2.Platform Scalability Challenge 3.Gaming Analytics Challenge 4.Content Generation Challenge

TI1100A – Internet-Based Applications – A. Iosup 52 Research Challenge: Solve the Platform Problem of MMOGs The Platform Problem of MMOGs Scaling quickly to millions of players, efficient hosting - 1M in 4 days, 10M in 2 months - Up-front and operational costs - Response time & Scalability

15 november Rich Research Opportunities: How to Build the Platform? Understand the application Deployment models Scheduling Performance engineering Reliability engineering Scalability and Elasticity Etc. (including Usability, Security, Utility Models, and Programming Models) PDS Group

TI1100A – Internet-Based Applications – A. Iosup 54 Load impact on game-play Responsive game-play Unresponsive game-play August 24, 2015 [Source: Nae, Iosup, and Prodan, ACM SC 2008]

TI1100A – Internet-Based Applications – A. Iosup 55 Models: Zoning: huge game-world division into geographical sub-zones – each zone is handled by different machines Mirroring: the same game-world handled by different machines, each one handling a subset of the contained entities (synchronized states) Instancing: multiple instances of the same zone with independent states. (World of Warcraft, Runescape,..) Game parallelization models [Source: Nae, Iosup, and Prodan, ACM SC 2008]

TI1100A – Internet-Based Applications – A. Iosup 56 Online games hosting model Generic Online Games (non-MM) Static: dedicated isolated single servers MMOGs Static: dedicated clusters - using parallelization techniques Problems with these approaches: 1.Large amount of over-provisioning 2.Non-efficient coverage of the world for the provided service [Source: Nae, Iosup, and Prodan, ACM SC 2008]

Cloud Computing VENI Massivizing Online Games using Cloud Computing

TI1100A – Internet-Based Applications – A. Iosup 58 Using data centers for dynamic resource allocation Main advantages: 1.Significantly lower over-provisioning 2.Efficient coverage of the world is possible Proposed hosting model: dynamic Massive join Massive leave [Source: Nae, Iosup, and Prodan, ACM SC 2008]

TI1100A – Internet-Based Applications – A. Iosup 59 Static vs. Dynamic Allocation What is the penalty for static vs. dynamic allocation? 250% 25% [Source: Nae, Iosup, and Prodan, ACM SC 2008]

August 24, Remaining Challenge in Perf. Eng.: To the Real IaaS Cloud “The path to abundance” On-demand capacity Cheap for short-term tasks Great for web apps (EIP, web crawl, DB ops, I/O) “The killer cyclone” Not so great performance for scientific applications (compute- or data-intensive) Not so stable performance Tropical Cyclone Nargis (NASA, ISSS, 04/29/08) VS August 24, Iosup et al., Performance Analysis of Cloud Computing Services for Many Tasks Scientific Computing, IEEE TPDS, 2011, Iosup et al., On the Performance Variability of Production Cloud Services, CCGrid 2011, pds.twi.tudelft.nl/reports/2010/PDS pdfpds.twi.tudelft.nl/reports/2010/PDS pdf

Remaining Challenge in Scheduling: Provisioning and Allocation Policies August 24, Where? When? How many? Time Load ProvisioningAllocation From where? Which type? etc. When? etc. Q: How to select policies? (Source: A. Antoniou, MSc Defense, TU Delft, 2012.)

Remaining Challenge in Deployment: Zynga zCloud: Hybrid Self-Hosted/EC2 After Zynga had large scale More efficient self-hosted servers Run at high utilization Use EC2 for unexpected demand August 24, (Sources: and

TI1100A – Internet-Based Applications – A. Iosup 63 Agenda 1.Internet: An Ongoing Story 2.Internet-Based Applications (part 1) Peer-to-Peer File-Sharing: Gnutella, KaZaa, BitTorrent 3.Internet-Based Applications (part 2) Massively Social Gaming: WoW, RuneScape, FarmVille 1.What’s in a Name? 2.Platform Scalability Challenge 3.Gaming Analytics Challenge 4.Content Generation Challenge

TI1100A – Internet-Based Applications – A. Iosup 64 Research Challenge: Solve the Analytics Problem of MMOGs The Analytics Problem of MMOGs Analyzing the behavior of millions of players, on-time - Data mining, data access rights, cost v. accuracy, … - Reduce upfront costs - Low response time & Scalable

Massivizing Social Games: Distributed Computing Challenges and High Quality Time – A. Iosup Social Everything! Social Network=undirected graph, relationship=edge Community=sub-graph, density of edges between its nodes higher than density of edges outside sub-graph (Analytics Challenge) Improve gaming experience Ranking / Rating Matchmaking / Recommendations Play Style/Tutoring Self-Organizing Gaming Communities Player Behavior

Massivizing Social Games: High Performance Computing and High Quality Time – A. Iosup 66 The CAMEO Framework 1.Address community needs Can analyze skill level, experience points, rank Can assess community size dynamically 2.Using on-demand technology: Cloud Comp. Dynamic cloud resource allocation, Elastic IP 3.Data management and storage: Cloud Comp. Crawl + Store data in the cloud (best performance) 4.Performance, scalability, robustness: Cloud Comp. A. Iosup, CAMEO: Continuous Analytics for Massively Multiplayer Online Games on Cloud Resources. ROIA, Euro-Par 2009 Workshops, LNCS 6043, (2010)

The Game Trace Archive (upcoming) Share gaming traces and best-practices on using them Support simulations and real-world experiments Massivizing Social Games: Distributed Computing Challenges and High Quality Time – A. Iosup 67 NamePeriodSize (GB) Node (M) Edge (M) Category KGS2002/ / Chess Game FICS1997/ / Chess Game BBO2009/ / Card Game XFire2008/ / OMGN Dota League2006/ / RTS DotaLicious2010/ / RTS Dota Garena2009/ / RTS WoWAH2006/ /1030.1N/AMMORPG

TI1100A – Internet-Based Applications – A. Iosup 68 Cost of Continuous RuneScape Analytics Put a price on MMOG analytics (here, $425/month, or less than $ /user/month) Trade-off accuracy vs. cost, runtime is constant

Massivizing Social Games: High Performance Computing and High Quality Time – A. Iosup Sample Analytics Results Analysis of Meta-Gaming Network “When you play a number of games, not as ends unto themselves but as parts of a larger game, you are participating in a metagame.” (Dr. Richard Garfield, 2000) XFire: since 2008 (3+ years), 500K of 20M players * A. Iosup, POGGI: Puzzle-Based Online Games on Grid Infrastructures EuroPar 2009 (Best Paper Award) S. Shen, and A. Iosup, The XFire Online Meta-Gaming Network: Observation and High-Level Analysis, MMVE 2011 PhD

TI1100A – Internet-Based Applications – A. Iosup 70 Sample Game Analytics Results Skill Level Distribution in RuneScape RuneScape: 135M+ open accounts (world record) Dataset: 3M players (largest measurement, to date) 1,817,211 over level 100 Max skill 2,280 Number of mid- and high-level players is significant New Content Generation Challenge High Level Mid Level

TI1100A – Internet-Based Applications – A. Iosup 71 Agenda 1.Internet: An Ongoing Story 2.Internet-Based Applications (part 1) Peer-to-Peer File-Sharing: Gnutella, KaZaa, BitTorrent 3.Internet-Based Applications (part 2) Massively Social Gaming: WoW, RuneScape, FarmVille 1.What’s in a Name? 2.Platform Scalability Challenge 3.Gaming Analytics Challenge 4.Content Generation Challenge

TI1100A – Internet-Based Applications – A. Iosup 72 Research Challenge: Solve the Content Problem of MMOGs The Content Problem of MMOGs Generating content on time for millions of players - Player-customized: Balanced, Diverse, Fresh - Up-front and operational costs - Response time & Scalability

73 (Procedural) Game Content (Generation) Game Bits Texture, Sound, Vegetation, Buildings, Behavior, Fire/Water/Stone/Clouds Game Space Height Maps, Bodies of Water, Placement Maps, … Game Systems Eco, Road Nets, Urban Envs, … Game Scenarios Puzzle, Quest/Story, … Game Design Rules, Mechanics, … Hendricks, Meijer, vd Velden, Iosup, Procedural Content Generation for Games: A Survey, ACM TOMCCAP, 2012 Derived Content NewsGen, Storification

Massivizing Social Games: High Performance Computing and High Quality Time – A. Iosup 74 The POGGI Content Generation Framework Only the puzzle concept, and the instance generation and solving algorithms, are produced at development time * A. Iosup, POGGI: Puzzle-Based Online Games on Grid Infrastructures, EuroPar 2009 (Best Paper Award)

75 Puzzle-Specific Considerations Generating Player-Customized Content Puzzle difficulty Solution size Solution alternatives Variation of moves Skill moves Player ability Keep population statistics and generate enough content for most likely cases Match player ability with puzzle difficulty Take into account puzzle freshness 4 21

TI1100A – Internet-Based Applications – A. Iosup 76 Summary Current Technology The Future Happy players Happy cloud operators Million-user, multi-bn market Content, World Sim, Analytics Massive Social Gaming Upfront payment Cost and scalability problems Makes players unhappy Our Vision Scalability & Automation Economy of scale with clouds Ongoing Work Content: POGGI Framework Platform: Analytics: CAMEO Framework Publications Gaming and Clouds 2008: ACM SC, TR Perf 2009: ROIA, CCGrid, NetGames, EuroPar (Best Paper Award), CloudComp, TR variability 2010: IEEE TPDS, Elsevier CCPE 2011: Book Chapter CAMEO Graduation Forecast 2010/2011: 1PhD, 2Msc, 4BSc

The Parallel and Distributed Systems Group at TU Delft August 31, Johan Pouwelse P2P systems Henk Sips multicore P2P systems Dick Epema grids/clouds e-Science P2P systems Alexandru Iosup online gaming grids/clouds P2P systems Home page Publications see PDS publication database at publications.st.ewi.tudelft.nlpublications.st.ewi.tudelft.nl

TI1100A – Internet-Based Applications – A. Iosup 78 Thank you for your attention! Questions? Suggestions? Observations? Alexandru Iosup (or google “iosup”) Parallel and Distributed Systems Group Delft University of Technology More Info: Do not hesitate to contact me…

TI1100A – Internet-Based Applications – A. Iosup 79 Bridge Base Online (BBO): 1M+ players, top free site Dataset: 100K players 9K group Social relationships from bridge pairing Large (~10K) online social groups can coordinate Identified player behavior community builder, community member, random player, faithful player Sample Game Analytics Results BBO Activity and Social Network Interaction group-socnet Coordinated large-scale social group

TI1100A – Internet-Based Applications – A. Iosup 80 Why Not Let Players Generate Puzzles? How to control production pipeline? After all, game developers sell content not technology. How to select content? Ranking problems, diversity problems. How to avoid game exploits? Virtual currency = Real currency User-generated content is clearly an interesting research area, but that’s another story. Source: mmobux.com, Aug 2009