Nathan Eagle Sandy Pentland Learning by Listening The OverHear project is one instance of attempting to move computers beyond simply speech recognition and towards speech understanding - specifically, it's geared toward teaching the computer an awareness of a conversation's context.   Now instead of using a wearable with all the sensors that Brian had, this project simply used a device that digitally recorded all of a user’s conversations. Nathan Eagle Sandy Pentland

Overview Omni-present “familiar” listens in a user’s conversations in a variety of contexts (working in office, chatting with girlfriend, at home, etc) The conversations are transcribed by a commercial voice recognition engine The agent learns the word patterns unique to a given context and creates a classification algorithm This algorithm can be leveraged to figure out a user’s current context given new conversation data   The goal of the project is to get a computer to distinguish whether a users is in a bar chatting with friends, in an argument with his girlfriend back at his apartment, or even giving a demo for sponsors here at the lab. By wearing something that always is listening in on everyday conversations, the computer can gradually start learning models of my behavior and ultimately, after a lot of training, it can give a good guess as to current location, people involved, and even the topic of the conversation.

Data Collection 2 months / 30 hours of labeled conversations Labels location ie: home, lab, bar persons ie: girlfriend, officemate, advisor type/topic ie: argument, meeting, chit-chat

Initial Results 90+% accurate using only location information Demonstrated some speaker independence (many people have common priors – esp at the media lab) Information from correlations among classes is necessary to establish the people in the conversation (typically my officemate is not in my apartment)

This is a short chip to get an idea of the training phase: Over 30 hours of daily conversations were recorded and input into the algorithm over the course of a month.