Where Intelligence Lives & Intelligence Management Geoff Hulten

Overview Where intelligence lives Intelligence management

Places Intelligence Could Live Client Service Back-end Hybrid

What does it matter where intelligence lives? Latency in Updating Quality is evolving quickly Problem is evolving quickly Risk of costly mistakes Latency in Execution Slowing the experience Slowing the action The right answer changes too fast Cost of operation Cost of distributing intelligence Cost of executing intelligence Offline operation Work without Internet? Keep it out of Abuser’s hands…

Where Intelligence Lives Lives in Service Lives on Client 1 MB Model Daily Update 100k Users 10 Calls/Day 1 mb x 1 Intelligence Creation Intelligence Creation Server 1 mb x 100,000 Server Telemetry 100kb x 10 X 100,000 Let's go through a simple comparison between hosting intelligence in the client and in a service. Let's say you have a 1MB model file and to balance out your Intelligent System you determine you need to update this model once a day. If you want to run the intelligence on a server you'd have just 1MB per day to transfer the model files between servers. But then you'd need to deal with all the intelligence executions. Say you have 100,000 customers. Maybe each customer interacts with the intelligence 10 times per day, and each intelligence call includes 100KB of data (maybe an image), and you have to use CPU on your server to process everything... 100,001 MB plus compute. If you want to run on the client you have to transfer new model files to them every day,100GB a day. Then do a bit of work to gather telemetry. But executing the intelligence is cheaper and has lower latency. About a wash in terms of cost. But there are clearly many options with some big implications. And once you get everything balanced perfectly, someone is going to come to you and say "Hey, our system is making some bad mistakes, so we have to update the intelligence faster, once per hour instead of once per day -- sound good?“ If your implementation is flexible, you can accommodate and you'll have more options for achieving the Intelligent System's objectives. If not? Well, you’ll have to figure out how to achieve balance some other way, like by making the experience less forceful. Clients Clients Total: 100,001 mb + compute Total: 100,000 mb + Telemetry

Places Intelligence can Live Where it Lives Latency in Updating Latency in Execution Cost of Operation Offline? Static in Product Poor Excellent Cheap Yes Client Side Variable Based on update rate Server-Centric Good Internet Roundtrip Can be high No Back-end Partial Hybrid ??

Examples of Where Intelligence Lives Kinect Anti-Phishing Online Shopping Self-Driving Car Sprinkler Controller Composition Assistant Latency in Updating Not Important Latency in Execution Critical Cost of Operation Not Key Factor Offline Operation Important Solution (?) Client Centric Latency in Updating Important Latency in Execution Critical Cost of Operation Important (?) Offline Operation Solution (?) Client Centric Latency in Updating Critical Latency in Execution Medium Important Cost of Operation Important Offline Operation Not Important Solution (?) Hybrid (ALL) Latency in Updating Slow Okay Latency in Execution Not Important Cost of Operation Important (?) Offline Operation Critical Solution (?) Backend (Cache) Latency in Updating Medium Important Latency in Execution Very Important Cost of Operation Offline Operation Not Important Solution (?) Server / Backend Latency in Updating Slow Okay Latency in Execution Important Cost of Operation Important (?) Offline Operation Solution (?) Server / Client

Intelligence Management Simple Intelligence Management Process of getting new models deployed safely, repeatably, at scale. Complexity Many models Living multiple places Comes from many sources Interdependencies Frequency Hourly for three years… ~26,000 times Long-lived Human Systems Staff Turnover Skill set of model deployers >> Copy $NewModelPath $DeployedModelPath >> RestartService.exe Risk of: High cost Error prone Hard to understand Paralysis

Effective Intelligence Management Sanity check the intelligence Simplify Deployment Workflow Support controlled light-up Dealing with Mistakes

Sanity Checking Intelligence Intelligence creators should test everything… But they might not… Automate and manage verification workflow Compatibility with Runtime Model correctly encoded Metadata (thresholds) present Feature extraction code in sync Models all in sync Runtime constraints RAM footprint Prediction perf (across contexts) Training vs Runtime prediction parity Obvious mistakes Verify offline accuracy on independent validation set Mistake distribution similar (not more costly) Business critical contexts Critical sub-populations

Deploying and Lighting Up (Online Evaluation) Single Deployment All users see all updates ‘at once’ Simple Relies on great offline tests Risk of costly/hard-to-find mistakes. Silent Intelligence Run two versions at once Ensure online is same as offline Gives time to see ‘new’ contexts Latency. No interactions. Controlled Rollout Several live at once, transition slowly Lets you observe user interactions Overhead to build and manage Adds latency. Flighting Intelligence (A/B test) Deploy options, track till one better Connects accuracy to true objective Latency. Hard to confirm small gains.

Overriding Problems Override mistakes Rollback problems Heuristics or simple rules Creating and deploying them Managing them over time Rollback problems Deploy new version quickly Store multiple versions in runtime Big Red Switch

Summary of Intelligence Management Where intelligence lives Static Client-side Server-centric Backend Hybrid Managing Intelligence Sanity checking Deploying Lighting it up Dealing with mistakes