2. FLogging: Metric Collection on the High Seas Joe Ludwig Director of Development Flying Lab Software

3. What is FLogging?  Server-side  Metric collection  Game events with parameters  Server performance events

4. Formal + Log = FLog

5. Design Goals  Designer goals  Lots of different views on data  Possible to derive context from data  Programmer goals  Easy to add new flog types  No global recompiles

6. Data Collection

7. Adding a new flog type  Add any new field types in XML  Add the new flog type in XML  Upgrade database to generate SQL  Tables  Procedures  Views  Add call to record flog

8. Flog types defined in XML Flogger_field_types.xml: level TINYINT

9. Flog types defined in XML Flogger_types.xml: level_gained level true false

10. Add call to record flog void flsAdvancementManager::setLevel(flsCharacter* pCharacter, igChar iNewLevel) { pCharacter->assignNewLevel(iNewLevel); if(pCharacter->isPlayer()) { flsFlogger::flogf("level_gained", pCharacter->getLocalId(), "%d", iNewLevel); }

11. Database Format - Tables  Flog table – one entry per flog  Value tables – One entry per flog that uses that field type  Flog_type table – One entry per type

12. flog table FieldType IdBIGINT IDENTITY Flog_typeINT Flog_timeDATETIME Cluster_idINT Character_idINT PuidBIGINT

13. value table FieldType IdBIGINT ValueTINYINT Type of value column is controlled by flogger_field_types.xml

14. flog_type table FieldType Flog_typeINT NameCHAR(32) hideByDefaultTINYINT

15. Database Format – Stored Procedures  Flog type procedures  Flog type expiration procedures  Flog_heartbeat procedure

16. Flog type procedures CREATE PROCEDURE Flog_level_gained @iObjectId INT, @level_param TINYINT AS BEGIN DECLARE @pid bigint, @cid int; SELECT @pid = puid, @cid = clusterid FROM #flog_vars; INSERT INTO flog (puid, cluster_id, flog_type, object_id) VALUES (@pid, @cid, 3, @iObjectId); DECLARE @lastid BIGINT; SET @lastid = SCOPE_IDENTITY(); INSERT INTO flog_level (id, value) VALUES (@lastid, @level_param); END

17. Flog age procedures CREATE PROCEDURE Flog_Age_skill_used AS DELETE flog FROM flog, flog_item_id WHERE DATEDIFF(DAY, flog.flog_time, CURRENT_TIMESTAMP)>=2 AND flog.id=flog_item_id.id

18. Flog Heartbeat procedure CREATE PROCEDURE Flog_Maintenance AS BEGIN EXEC flog_Age_maneuver_skill_used; EXEC flog_Age_skill_used; EXEC flog_Age_volley_fired; END

19. Database Format – Type View CREATE VIEW Flog_Data_level_gained AS SELECT flog.id, flog_level.value AS [level] FROM flog INNER JOIN flog_level ON flog.id=flog_level.id WHERE flog.flog_type=3

20. What we flog  Login/Logout events  Zoning  Economic events  Advancement events  Missions  Conquest events  Chat  GM commands

22. How much data?  4.5 million new flog records per week  Top 5 flog types account for 50%

26. Example flog output

28. Money Gained ReasonAmount money_gained_mission31,310,500 money_gained_trade11,038,331 money_gained_kill8,029,993 money_gained_auction6,826,344 money_gained_shop4,123,899 money_gained_create_society13,000 money_gained_respec1,263 money_gained_mail350

29. Money Lost ReasonAmount money_lost_trade11,038,331 money_lost_recipe10,980,920 money_lost_auction9,956,249 money_lost_shop7,587,867 money_lost_ActivateStructureDeed4,760,800 money_lost_admin3,338,500 money_lost_recipe_tax633,113 money_lost_PayStructureUpkeep510,225 money_lost_auction_listing_fee159,867 money_lost_create_society89,000 money_lost_mail14,550 money_lost_AuctionRelist1,566

30. Exploit Detection

31. Reports using context  Context is available in the data  Extracting it can be slow  Example: Deaths per level

32. Current level function CREATE FUNCTION level_at_flog (@iObject INT, @iCluster INT, @iFlogId BIGINT) RETURNS INT AS BEGIN DECLARE @iLevel INT SELECT TOP 1 @iLevel = flog_level.value FROM flog INNER JOIN flog_level ON flog.id=flog_level.id WHERE (flog.id < @iFlogID) AND flog.object_id=@iObject AND flog.cluster_id = @iCluster AND flog.flog_type=3 ORDER BY flog.id DESC IF @iLevel IS NULL SET @iLevel=1 RETURN @iLevel END

33. Death by level table CREATE TABLE death_by_level (id BIGINT, level INT, object_id INT, cluster_id INT)

34. Death by level table INSERT INTO death_by_level (id, level, object_id, cluster_id) SELECT flog.id, dbo.level_at_flog(flog.object_id, flog.cluster_id, flog.id) AS [Level], flog.object_id, flog.cluster_id FROM flog WHERE flog.flog_type=65

35. Death by level table idLevelobject_idcluster_id 130804710 1994073450 2316997410 2421624430 3485688740 3821457750 3848618860 3905208860 4023539220 4233839330

36. Death by level report SELECT level, COUNT(id) AS [Deaths] FROM death_by_level GROUP BY level ORDER BY level

39. More data from context  Death by zone  Death by mission  XP by zone  Skills purchased by level  Play time per level

40. Room to improve  More and better reports  Automatic data rollups  Parameterized queries  Multiple associated fields in one table  Performance

41. Questions? Slides available at http://programmerjoe.comhttp://programmerjoe.com Joe Ludwig joe@flyinglab.com