Toward Real-Time Extraction of Pedestrian Contexts with Stereo Camera Kei Suzuki, Kazunori Takashio, Hideyuki Tokuda, Masaki Wada, Yusuke Matsuki, Kazunori Umeda Graduate School of Media and Governance, Keio University Department of Precision Mechanics, Faculty of Science and Engineering, Chuo University Fifth International Conference on Networked Sensing Systems June , 2008 // Kanazawa, Japan

Introduction Vision-based monitoring systems are becoming important Many researches of pedestrian recognition from vision data have been conducted. –Pedestrian Detection, Tracking, Activities Security camera

Motivation Current works are difficult for extracting pedestrian’s high-level contexts. Pedestrian Detection, Tracking, Activity Pedestrian high-level Contexts How suspicious is this situation? Snatch Skirmish

Goals Extract the high-level contexts of pedestrians from vision data in real-time –We aim to extract suspicious individuals, groups and uneasy atmosphere b ased on the position, velocity, width, and height of those individuals. –We developed a prototype system that can find people tumbling, walking in a group. At a street cornersystem image

Our Approach Use of a stereo camera system that focuses on a moving region –We can make real-time 3D measurement of more than one pedestrian’s region center of gravity coordinates, height, width, label number –Easy to install in a new place Infer the pedestrian contexts by using Bayesian Networks –We model pedestrian movement as time-series data –We make Bayesian models and let them learn for each context

The target contexts table individual group Atmosphere of the place Prototype system Not yet walking in a group tumble Project target Walking Running Tumble Snatch Skirmish Crowded Quiet

System Overview Hardware architecture At a street corner Stereo camera Analyze vision data network Stereo camera system Pedestrian context Infer system Display result of contexts Infer the context

System Overview The flow of inferring contexts e.g. A: velocity vector similarity B: the average distance C: the average vector angle The thread of event detection The inference thread using the Bayesian Networks sliding window Input variables into Bayesian Model Stereo camera system data

Experiment ． Extract contexts with real-data Extract two pedestrian contexts –Tumble as a individual context –Walking with friends as a group context Result –Show the effectiveness of extracting two or more contexts in real-time. –The required time of extracting the contexts was 58 msec, but it worked in real-time due to the event trigger model.

Conclusion / Future Work We extracted the pedestrian contexts with stereo camera system We developed the prototype system, and confirmed by the experiments. Future work –Further evaluation of accuracy, and compare its performance with some other methods.

Thank you!

Experiment1 ． Stereo camera’s output test Walk 6.5m from the side of a camera, then turn right. Stereo camera’s frame: 14 frame/sec Camera Distance from camera (m) Time (sec) Turn right

Bayesian Network Model The sample of tumble model

Pedestrian Recognition at a street corner Many projects of pedestrian recognition from video data have been conducted. –Pedestrian Detection –Pedestrian Tracking –Pedestrian’s activities Security camera

Characteristics of Pedestrian The activities of Individuals –walking, running, tumbling The activities of Groups, mobs –Harmless group Companion ， with the same intention –Contingent group Moving to same direction, temporary crowded with people –Suspicious group Snatchers, fighting, entering no admittance area Atmospheres of a place –Crowded, quiet Pedestrians have 3 characteristics based on their movement.

Goals Extracting from suspicious individuals and uneasy atmosphere by using video cameras –We aim to extract high-level pedestrian contexts in real-time –The system infers contexts based on pedestrians’ moving region data

Stereo Camera System that focusing on moving regions Calculate Moving region feature –Center of gravity coordinate –Distances from camera –Height and width –Timestamp –Label number

Project abstract (1/2) Extracting from suspicious individuals and uneasy atmosphere by using stereo cameras –We aim to extract high-level pedestrian contexts in real-time –The system infers contexts based on pedestrians’ moving region data from stereo camera At a street corner

Project abstract (2/2) Recognizing group and mobs –harmless crowd Companion group –Accidental mobs spectators –Suspicious mobs ⇒ target snatchers, fighting mobs

Bayesian Network Extracting of high-level pedestrian contexts using Bayesian Network.

System Over View At the street Stereo camera Video data analyze network Stereo camera system Pedestrian context Infer system Display result of contexts Context infer

Hardware architecture Pedestrians at street corner Stereo camera Processing video data network Stereo camera system Inferring Pedestrian Contexts in real-time Contexts inferred result Infering contexts

Prototype System’s target Contexts target individuals group Atmosphere Of a place Prototype system’s target contexts Not yet Walking with friends tumble Project target contexts Normal walking Running Tumble Snatchers, Fighting Crowding Quiet

実験 2 ．コンテクスト推定部の負荷実験 コンテクスト推定の最大負荷時