1. Boyu Wang and Minh Hoai Stony Brook University
Predicting Body Movement and Recognizing Actions: an Integrated Framework for Mutual Benefits
Good afternoon. My name is Minh Hoai, and I am from Stony Brook University.
This is the joint work with my student Boyu Wang, who cannot be here due to a summer internship.
Boyu Wang and Minh Hoai
Stony Brook University
FG2018 Poster Session #: I
Poster ID #: 12

2. What is she about to do? Shake my hand?
Predicting Body Movement and Recognizing Actions: an Integrated Framework for Mutual Benefits
What is she about to do? Shake my hand?
Consider building a robot to physically interact with humans.
Suppose the person initiates a handshake.
For a fluent interaction, the robot must be able to: 1) recognize the handshake action as soon as possible and 2) anticipate where the person’s hand will be so the robot can start extending its hand to the right spot. These two tasks are called early recognition and movement prediction.
If the robot cannot do either of these two tasks, then cannot react until the person has stopped moving their hand.
Thus the interaction would be slow and unnatural, just like how the current generation of robots interact with humans.
Where will her hand be? I need to extend my hand to meet hers.

3. Movement Prediction LSTM
Predicting Body Movement and Recognizing Actions: an Integrated Framework for Mutual Benefits
Movement Prediction LSTM
past
future
present
We propose a method for joint movement prediction and early recognition.
Our method combines two LSTM RNNs, one for movement prediction and one for early recognition.
The recognition LSTM uses the predicted sequence of body movement produced by the movement prediction LSTM.
While the LSTM for movement prediction is designed to accommodate the output of the recognition system
Experiments on two datasets demonstrate the benefits of jointly performing movement prediction and early recognition.
Please drop by our poster
Recognition LSTM
Results: we observe mutual benefits:
Can recognize the action  better movement prediction
Can predict movement  recognize the action sooner
Class probability vector