1. Closing the Loop: Training data from Experience
Geoff Hulten

2. Closed Loop Example: Search Engine
Search for:
ML Course
Your Results:
Some Boring Course Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Some Spam ML Ad Sed eiusmod tempor incididunt ut labore et dolore magna aliqua.
Another Boring Course Duis aute irure dolor in reprehenderit.
UW’s CSEP 564 Ut enim ad minim veniam, quis nostrud exercitation.
Some Awesome ML Course Lorem ipsum dolor sit amet, consectetur adipiscing elit.
UW’s CSEP 564 Ut enim ad minim veniam, quis nostrud exercitation.
Some Spam ML Ad Sed eiusmod tempor incididunt ut labore et dolore magna aliqua.
Another Awesome Course Duis aute irure dolor in reprehenderit.
P(Relevant | Query, User, Page)
Simple Heuristic: TF/IDF
Training Data
Machine Learning…
Click the one
they like
Higher Sentiment / More Engagement
High Sentiment / Engagement
Virtuous Cycle
Extremely Simplified…

3. Properties of Good Data
Without user present
With user present FP TP TN FN
Context, Action, and Outcome
Sensors
What the user / system did
Override, feedback, power reading
Unbiased
Strong sentiment influences data
Mistakes influence data
Reporting experience influence data
Good Coverage
Homes, geo-locations
Kitchens, factories, spaceships
Summer, winter, user types
Avoids Feedback Loops
Prominence leads to interaction
Absence suppresses interaction
Real Interactions
Users using the product as indented
Not in the lab, not in a survey
Expert labelers not as good
Scale
User base
Frequency
Complex concepts need a lot of data

4. Example: Self Driving Car
Context, Action, Outcome
Coverage
Real Interactions
Unbiased
Feedback Loops
Scale
Product viable
Intelligence minimal
Telemetry in place
Intelligence asks user if it is right
User answers are great data
Randomize to get coverage
Safe Automation
Highest quality models
Cheaper mistakes
Full Automation
Mistakes extremely rare
Value of data diminished
Once you know your problem can benefit from an intelligent system, you need to decide how much of it you want to try to solve with one.
One interesting property of intelligent systems is this: they perform worst on the day you ship them. Once you close the loop between users and the intelligence, your system will get better and better over time.
That means you might want to start with an easy objective, and rebalance toward harder objectives as your system improves.
Let's say you are trying to build an autonomous car. This is certainly an intrinsically hard, big, open-ended and time changing type of problem. You could work on this until you get it totally perfect, and then ship it.
<SLIDE>
Or you could start with an easier sub-problem. Say forward collision avoidance. Something is stopped in front of the car? Apply the brakes.
And here's the thing. You could build the exact same car for collision avoidance that you would build for the autonomous driving, all the controls, all the sensors, everything. Just instead of setting an objective of full automation, which is super hard, you set an objective of reducing collisions, which is more manageable.
But avoiding collisions is valuable to users on its own right, so some people will buy your car and use it. And when they do, you'll be able to collect information from all the car's sensors and all the user interactions to get data to build better and better intelligence.
Then, when you are ready, you might set yourself a slightly harder objective, say lane following. That provides even more value to users -- the virtuous cycle.
And after a while of lane following, your intelligence might get so good you can move on to completely autonomous driving. The process might take months. It might take years. But it will almost always be cheaper than trying to build an autonomous car without a closed loop between users and intelligence.
It's important to set an objective that you can achieve with the intelligent system you can build today -- and it's great when you can grow the intelligent system to solve more and more interesting objectives over time.
Legacy System
Compete on other dimensions
Collision Warning
Lane Leave Warnings
Collision Avoidance
Lane Following
Progressively less interaction
Dumb
Passive
Getting Forceful
Increasingly Forceful

5. Ways to Get Data from Users
Implicit
User using the product as intended
Sometimes hard to interpret
Carefully crafted UX can enable
Escalations
User s/calls support
Good for finding unanticipated problems
Bad for machine learning
Reply
Delete
Unopened x
Read
Ratings
1-5 stars; thumbs up/down
Recommender systems (not )
Can be quite sparse
Inbox:
Dinner this weekend? Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Here are your automatic payment... Sed eiusmod tempor incididunt ut labore et dolore magna aliqua.
Congradulations {Ghulten} you’ve... Duis aute irure dolor in reprehenderit.
What did you think about... Ut enim ad minim veniam, quis nostrud exercitation.
Is this message:
Junk OK
Report Junk
Reports
User flags a mistake in the systems
Highly biased to visible/costly mistakes
Classifications
Ask users to provide specific labels
Good coverage / bias
Requires users’ good will
Contact Us

6. Advanced topics
Combining / sampling data
Maintaining independence

7. Summary: Closed Loop Pattern
Better Models
Virtuous Cycle between users and Intelligence
Models worst on day one, improve over time
Needs experience tuned to get training data
More forceful as quality increases
Used by many, many practical systems
Intelligence Creation Environment
Intelligence Runtime
Closed Loop
Training Corpus
Explicit and Implicit User Feedback