Visual Multi – User Tracking in Instrumented Rooms Developing MaMUT Chair of Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster Advisor: Dipl.-Inform. Michael Schmitz - Bachelor’s Thesis- Henning Zimmer December 2006 MaMUT: Why the name … you will se later !!!

Problem Ubicomp environment (i.e., instrumented room) usually public  multiple users in environment interacting with intelligent objects (products, displays, …) At the moment: Single-user support E.g., Shopping Assistant at product shelf We want: Associate interactions with objects to distinct users E.g., for personalized services based on interaction-history At the moment only possible via auxiliary devices (PDA, …) Henning Zimmer

 Where persons are located, who interacts with which objects Our Approach One idea: Visual approach (LAN cameras) Advantage: No instrumentation of users, LAN-cameras available in SUPIE, surveillance cameras in supermarkets, … Advantage & Disadvantage : Quite complex sensors Computer Vision (CV) Tracking technique Allows to detect and track persons in camera images Extend it with interaction detection and positioning of persons Answers: Homography: Helps to estimate person positions … more: on slide 8 !!!  Where persons are located, who interacts with which objects Henning Zimmer

Related Work Different ideas for person tracking exist No one could solve our problem in total decided to develop own system “from scratch” For example: Siebel’s Reading People Tracker (RPT) + robust Person Tracking + more than 1 camera + Off-the-shelf PC hardware -- No skin detection -- No positioning CV: large research area, overview would exceed scope ! Author: N.T. Siebel [Sie03] RPT: PH.D. Thesis of Siebel, University of Reading = Person tracking module for ADVISOR-project integrated realtime surveillance system for underground stations”  Only detect number of persons, but very reliable (Active Shape Model) Henning Zimmer

Images  complex problem space  “Divide and Conquer” Our Approach in Detail Images  complex problem space  “Divide and Conquer” Macro-tracking (MaT) [1] Whole room, position of persons Micro-tracking (MiT) [2] Special areas of interest (e.g. product shelf), hands of persons Fusion of Macro- and Micro-tracking results [3] Associate detected hands (MiT) to tracked persons (MaT) dedicated camera(s) [1] [2] [3] Henning Zimmer

Macro – Tracking (1) Motion Detection Overview: Input: Output: Camera Image 1 Motion Image 2 Blobs 3 Regions 4 Person Regions 5 Person Positions (1) Motion Detection Via Background subtraction Input: Background model m(x,y) [1] Captured frame f(x,y) [2] Threshold T(x,y) Output: Binary motion image d(x,y) [3] needs filtering [4] [2] [1] [3] [4] T(x,y) just a scalar-valued-function R^2  R may be constant, i.e., T(x,y) = T forall (x,y) \in R^2 Henning Zimmer

Macro – Tracking (2) Camera Image 1 Motion Image 2 Blobs 3 Regions 4 Person Regions 5 Person Positions [2] Connected Component Labeling in Motion Image d(x,y) [3] One region (rectangular bounding box) per blob [4] Multiple regions per person  Region merging Motion Image Blobs [2] Blob Regions [3] Person Regions [4] Blobs = set of connected white pixels Henning Zimmer

Macro – Tracking (3) (5) Position estimation via Homography estimation Camera Image 1 … 4 Person Regions 5 Person Positions (5) Position estimation via Homography estimation Idea: Static cameras, persons stand on ground plane Person position = position of tracepoint Problem: Map coordinates of tracepoint from image-coordinates (x,y) to world-coordinates (x’,y’) Find matrix s.t. Uses n ≥ 4 point-to-point correspondences (xi ↔ xi’) for DLT-algorithm [HZ04] x_i = (x,y) ~ [x’,y’] (Img) ~ [World] Henning Zimmer

Micro – Tracking Skin Detection  Binary image S classifying skin pixels Model: Skin Images  HSV color space (adopted from [Bra98]) H-Histogram (H-channel carries color information, separated from brightness) Skin hue lies in the range from 0 to 10°  Skin probability image Sprob via Thresholding of Sprob yields S Image: B. Kapralos et al. Sprob S Henning Zimmer

Fusion Assign hands ( MiT) to persons ( MaT) Simple approach: Cluster skin regions  1 cluster per person at shelf Assign clusters to persons according to their position at the shelf Order skin clusters from “left” to “right” Order persons at shelf from “left” to “right” 1-to-1-mapping from clusters to persons Proof-of-concept: Works with > 1 Person, but problematic when crossing hands Left Right Order skin clusters from “left” to “right” (shelf coordinates) Order persons at shelf from “left” to “right” ( world coordinates) Henning Zimmer

Person Tracking Simple tracking approach (for persons) From time t to t+1: Match overlapping BB or color models Color Models: (Updated) Color Histograms of persons At match: update according person object Person BB @ t Detected region @ t+1 Henning Zimmer

Conclusions & Outlook Proof of concept of an extended person tracker Main problem: Realtime needs Several highly complex algorithms, lots of data (3 cameras at 10 FPS  ~ 2 500 000 pixels / second / algorithm) Robustness: Achilles heel of CV applications E.g., iIllumination  shadows, crowded rooms, … Future Improvements More sophisticated BG-subtraction methods with shadow removal Kalman filtering for tracking More clever capturing of images ( C++ class [Met]) Use PTZ cameras to cover whole room ( partial visible persons) Problem: Homography estimations Every field (Motion Detection, Skin Detection, Homography, …) poses its problems  solved all, but none totally satisfying ! Kalman filter = recursive filter, estimating state of a dynamic system Henning Zimmer

Possible Applications Multi-User support for Shopping Assistant [Sta05] Want to distinguish which user carries out interactions at shelf Virtual Room Inhabitant [KSS05] Adapt VRI position to position of tracked persons Integration into Indoor Positioning System [BS05] Position persons without any auxiliary devices Person Color-Profiles for UbisWorld [Hec] initially identification of persons needed  Basically useable for all interaction-based, personalized services in ubicomp environments Henning Zimmer

References [Sie03] : Siebel,N.T.: Design and Implementation of People Tracking Algorithms for Visual Surveilance Applications, Department of Computer Science, The University of Reading (Diss), UK, March 2003 [HZ04] : R.I. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, 2004 [Bra98] : G. Bradski, Computer Vision Face Tracking For Use in a Perceptual User Interface, Intel Technology Journal, 1998 [Met] : http://www.metz.supelec.fr/~ersidp/Software/AxisPTZ/Home.htm [KSS05] : M. Kruppa, M. Spassova, M. Schmitz, The Virtual Room Inhabitant - Intuitive Interaction With Intelligent Environments, In Proceedings of the 18th Australian Joint Conference on Artificial Intelligence (AI05), 2005 [Sta05] : C. Stahl, J. Baus, B. Brandherm, M. Schmitz, T. Schwartz, Navigational and Shopping Assistance on the Basis of User Interactions in Intelligent Environments, In Proceedings of the IEE International Workshop on IE, 2005 Henning Zimmer

 Thank you for your attention !!! References (2) [BS05] : C. Stahl, J. Baus, B. Brandherm, M. Schmitz, T. Schwartz, Navigational- and Shopping Assistance on the Basis of Interactions in Intelligent Environments, In Proceedings of the IEE International Workshop on IE, 2005 [Hec] : D. Heckmann, UbisWorld.org, http://www.ubisworld.org  Thank you for your attention !!! Questions ?!? (Offline) DEMO !!! Henning Zimmer