Massively Multiplayer Online Games & DIS Commercial/Defense Convergence?

Massively Multiplayer Online Games MMOGs are similar to some DoD projects; can DoD use them as COTS? After all, they routinely scale to many thousands of simultaneous users World of Warcraft, Eve Online, and others routinely have >10K simultaneous users and are up 7/24/365. Can DoD exploit this to achieve something similar for not much money?

MMOGs Not all is wonderful in MMOGs Cost to develop a title can run $25M+ High commercial failure rate of games; success limited to a few titles, but the successful ones make a lot of money Software frameworks they are based on are fast-moving, proprietary, and not standards-based Multi-year development cycles From a business standpoint, the publisher makes money on subscriptions, not title sales price. Support infrastructure (servers, bandwidth, customer support) means higher fixed & marginal costs per customer, but a steady stream of subscription income

MMOGs & DoD Can DoD make use of MMOG frameworks? The wrong answer: let’s throw away all our software and rewrite it in Second Life A better answer: can we use existing DoD M&S protocols to and leverage the server-side MMOG frameworks? Ideally this would give us the ability to scale while keeping our old models So: pick an MMOG framework, see if we can hook up our models over DIS or HLA We want a low cost for failure; odds are good that an MMOG application will not be successful, for reasons unrelated to technology, so make the consequences of failure low

Picking an MMOG MMOGs are following a classic software arc, from closed and proprietary to standards- based and open source, with parallel proprietary software. Often the market moves from defacto to formal standards MMOG frameworks are just beginning to enter the standards/open source stage; probably several years away from consolidating Second Life is the clear market leader so far, but also defines its own network protocols

Darkstar Sun’s Project Darkstar is a Research Labs project on MMOGs (projectdarkstar.com) Open source, free Not a game engine; not a complete server-side framework, either Analogous to a java servlet application server The goal: scale to hundreds of thousands of simultaneous users via scale-out architecture; to add more users, you add more servers at the data center Java-based on the server side, lightweight client side in Java, C++, Python, Obj-C, etc. Client side only implements communications framework; the server side does the heavy lifting

Darkstar Why Darkstar? Free, open source Agnostic about protocols; defines a method for passing messages, but defining the content of the messages is up to you (so we can use DIS) Potentially disruptive at low price point in labor, software, and equipment It’s cheap, easy, and quick; if it works, great, and we can have a significant short-term impact; if not, we don’t have a lot invested in it

Darkstar Architecture Darkstar insights: Entity updates are often embarrassingly parallel Keep objects in a single persistent data store/database Scale by throwing more hosts at problem Use database transactions to maintain object consistency

Darkstar Architecture Game Server Darkstar Stack Game Server Darkstar Stack Game Server Darkstar Stack Persistent Object Store Client …

Darkstar Architecture Clients connect to a server via channels Servers can handle entity logic & AI, and update the persistent state of the entity A persistent object store (database) handles entity state changes via transactions (all or nothing) Servers can handle persistent tasks that change entity state An object manager keeps state consistent between servers

Entity Updates Tasks are threads scheduled on the server that typically run for very short periods (~100ms). There may be hundreds of tasks running at once on a server, and many servers. If a task updates an object it establishes a lock on that object. Other tasks cannot modify the object while locked, and are rescheduled if locked out This simplifies a lot of problems with server-side logic: multiple thread updates, deadlock, etc Makes it appear to be a single-threaded server update logic

Entity Updates Time Task A Obj A Write Lock Task B Obj A Task B Obj A Write Lock Update Transaction Reschedule

Channels “Channels” are an abstraction for passing messages Similar semantics to HLA: reliable, unreliable, unreliable ordered, etc. Channels only pass byte arrays--the contents of the messages are up to you No multicast (though you could write one) If contents are opaque, we might as well pick something like DIS

Darkstar Advantages: Makes it look like one big single threaded server Scalable (maybe!) on CPU Persistent; can pull the power plug and when rebooted the server continues as before (including tasks) Easy to integrate DIS Shardless overall design; not necessary to segment servers by geographic area

Darkstar Drawbacks Scalability not demonstrated yet! Scales across multiple CPUs and cores on one node, but not across multiple hosts Roadmap suggests January 2010 as earliest date for this But…you can run a decent number of entities on a single server, and a few hundred entities is big enough for a significant class of applications

Darkstar/DIS Architectures We can make use of Darkstar in multiple configurations, depending on objectives. Pure server: all entities hosted on server Mixed: Darkstar host acts as peer in conventional DIS exercise Router/DDM: all entities on clients, DDM runs on server

Darkstar : Pure Server Client Game Server Darkstar Stack Client DIS messages sent across Darkstar channel All entities hosted on server

Pure Server All entities are hosted on the server Clients view updates from constructive entities on the server Users can control entities on server (with higher latency) Server can perform DDM

Darkstar: Mixed Game Server Darkstar Stack Gateway DIS Network Darkstar Channel Sockets DIS Peer DIS Peer Darkstar is another DIS peer

Darkstar: Mixed In this configuration a Darkstar simply acts as a host for constructive entities Even though the DIS format is used between the gateway and the server, it still needs to be encapsulated inside the channel abstraction; the gateway simply packages and unpackages the DIS No DDM Provides a handy way to host entities

Darkstar: Router/DDM Client Game Server Darkstar Stack Client DIS messages sent across Darkstar channel

Darkstar: Router/DDM All entities hosted on clients, Darkstar server receives DIS messages over channel, and can act as a DDM server to route messages to a subset of clients Darkstar acts as a handy place to route data and do DDM while hosting no entities itself

Performance Darkstar does not yet scale across multiple nodes; earliest date for this is early 2010 But it does scale across multiple CPUs and cores in one box Single, dual-core CPU: can host approx mid- hundreds of entities, max Dual CPU, dual core mid-range server (Dell 2950): around a thousand entites, max

Conclusions Darkstar is useful as a peer in existing DIS exercises; can host several hundred entities in a relatively convenient way, persistently and 7/24/365 Interesting if scaling is demonstrated MMOG space will probably coalesce around de facto standards, later codified by standards organizations (IETF/MMOX). SISO should keep an eye on this