0. Visual Multi – User Tracking in Instrumented Rooms Developing MaMUT
Chair of Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
Advisor: Dipl.-Inform. Michael Schmitz
- Bachelor’s Thesis-
Henning Zimmer
December 2006
MaMUT: Why the name … you will se later !!!

1. Problem
Ubicomp environment (i.e., instrumented room) usually public
 multiple users in environment interacting with intelligent objects (products, displays, …)
At the moment: Single-user support
E.g., Shopping Assistant at product shelf
We want:
Associate interactions with objects to distinct users
E.g., for personalized services based on interaction-history
At the moment only possible via auxiliary devices (PDA, …)
Henning Zimmer

2.  Where persons are located, who interacts with which objects
Our Approach
One idea: Visual approach (LAN cameras)
Advantage: No instrumentation of users,
LAN-cameras available in SUPIE,
surveillance cameras in supermarkets, …
Advantage & Disadvantage : Quite complex sensors
Computer Vision (CV) Tracking technique
Allows to detect and track persons in camera images
Extend it with interaction detection and positioning of persons
Answers:
Homography: Helps to estimate person positions … more: on slide 8 !!!
 Where persons are located, who interacts with which objects
Henning Zimmer

3. Related Work Different ideas for person tracking exist
No one could solve our problem in total
decided to develop own system “from scratch”
For example: Siebel’s Reading People Tracker (RPT)
+ robust Person Tracking
+ more than 1 camera
+ Off-the-shelf PC hardware
-- No skin detection
-- No positioning
CV: large research area, overview would exceed scope !
Author: N.T. Siebel [Sie03]
RPT: PH.D. Thesis of Siebel, University of Reading
= Person tracking module for ADVISOR-project integrated realtime surveillance system for underground stations”
 Only detect number of persons, but very reliable (Active Shape Model)
Henning Zimmer

4. Images  complex problem space  “Divide and Conquer”
Our Approach in Detail
Images  complex problem space  “Divide and Conquer”
Macro-tracking (MaT) [1]
Whole room, position of persons
Micro-tracking (MiT) [2]
Special areas of interest (e.g. product shelf), hands of persons
Fusion of Macro- and Micro-tracking results [3]
Associate detected hands (MiT) to tracked persons (MaT)
dedicated camera(s)
[1]
[2]
[3]
Henning Zimmer

5. Macro – Tracking (1) Motion Detection Overview: Input: Output:
Camera Image 1 Motion Image 2 Blobs
3 Regions 4 Person Regions 5 Person Positions
(1) Motion Detection
Via Background subtraction
Input:
Background model m(x,y) [1]
Captured frame f(x,y) [2]
Threshold T(x,y)
Output:
Binary motion image d(x,y) [3]
needs filtering [4]
[2]
[1]
[3]
[4]
T(x,y) just a scalar-valued-function R^2  R
may be constant, i.e., T(x,y) = T forall (x,y) \in R^2
Henning Zimmer

6. Macro – Tracking (2) Camera Image 1 Motion Image 2 Blobs
3 Regions 4 Person Regions 5 Person Positions
[2] Connected Component Labeling in Motion Image d(x,y)
[3] One region (rectangular bounding box) per blob
[4] Multiple regions per person  Region merging
Motion Image
Blobs
[2]
Blob Regions
[3]
Person Regions
[4]
Blobs = set of connected white pixels
Henning Zimmer

7. Macro – Tracking (3) (5) Position estimation via Homography estimation
Camera Image 1 … 4 Person Regions 5 Person Positions
(5) Position estimation via Homography estimation
Idea: Static cameras, persons stand on ground plane
Person position = position of tracepoint
Problem: Map coordinates of tracepoint from image-coordinates (x,y) to world-coordinates (x’,y’)
Find matrix s.t.
Uses n ≥ 4 point-to-point
correspondences (xi ↔ xi’)
for DLT-algorithm [HZ04]
x_i = (x,y) ~ [x’,y’]
(Img) ~ [World]
Henning Zimmer

8. Micro – Tracking
Skin Detection  Binary image S classifying skin pixels
Model: Skin Images  HSV color space (adopted from [Bra98])
H-Histogram (H-channel carries color information, separated from brightness)
Skin hue lies in the range from 0 to 10°  Skin probability image Sprob via
Thresholding of Sprob yields S
Image: B. Kapralos et al.
Sprob S
Henning Zimmer

9. Fusion Assign hands ( MiT) to persons ( MaT) Simple approach:
Cluster skin regions  1 cluster per person at shelf
Assign clusters to persons according to their position at the shelf
Order skin clusters from “left” to “right”
Order persons at shelf from “left” to “right”
1-to-1-mapping from clusters to persons
Proof-of-concept: Works with > 1 Person, but problematic when crossing hands
Left
Right
Order skin clusters from “left” to “right” (shelf coordinates)
Order persons at shelf from “left” to “right” ( world coordinates)
Henning Zimmer

10. Person Tracking Simple tracking approach (for persons)
From time t to t+1: Match overlapping BB or color models
Color Models: (Updated) Color Histograms of persons
At match: update according person object
Person t
Detected t+1
Henning Zimmer

11. Conclusions & Outlook Proof of concept of an extended person tracker
Main problem: Realtime needs
Several highly complex algorithms, lots of data
(3 cameras at 10 FPS  ~ pixels / second / algorithm)
Robustness: Achilles heel of CV applications
E.g., iIllumination  shadows, crowded rooms, …
Future Improvements
More sophisticated BG-subtraction methods with shadow removal
Kalman filtering for tracking
More clever capturing of images ( C++ class [Met])
Use PTZ cameras to cover whole room ( partial visible persons)
Problem: Homography estimations
Every field (Motion Detection, Skin Detection, Homography, …) poses its problems  solved all, but none totally satisfying !
Kalman filter = recursive filter, estimating state of a dynamic system
Henning Zimmer

12. Possible Applications
Multi-User support for Shopping Assistant [Sta05]
Want to distinguish which user carries out interactions at shelf
Virtual Room Inhabitant [KSS05]
Adapt VRI position to position of tracked persons
Integration into Indoor Positioning System [BS05]
Position persons without any auxiliary devices
Person Color-Profiles for UbisWorld [Hec]
initially identification of persons needed
 Basically useable for all interaction-based, personalized services in ubicomp environments
Henning Zimmer

13. References
[Sie03] : Siebel,N.T.: Design and Implementation of People Tracking Algorithms for Visual Surveilance Applications, Department of Computer Science, The University of Reading (Diss), UK, March 2003
[HZ04] : R.I. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, 2004
[Bra98] : G. Bradski, Computer Vision Face Tracking For Use in a Perceptual User Interface, Intel Technology Journal, 1998
[Met] :
[KSS05] : M. Kruppa, M. Spassova, M. Schmitz, The Virtual Room Inhabitant - Intuitive Interaction With Intelligent Environments, In Proceedings of the 18th Australian Joint Conference on Artificial Intelligence (AI05), 2005
[Sta05] : C. Stahl, J. Baus, B. Brandherm, M. Schmitz, T. Schwartz, Navigational and Shopping Assistance on the Basis of User Interactions in Intelligent Environments, In Proceedings of the IEE International Workshop on IE, 2005
Henning Zimmer

14.  Thank you for your attention !!!
References (2)
[BS05] : C. Stahl, J. Baus, B. Brandherm, M. Schmitz, T. Schwartz, Navigational- and Shopping Assistance on the Basis of Interactions in Intelligent Environments, In Proceedings of the IEE International Workshop on IE, 2005
[Hec] : D. Heckmann, UbisWorld.org,
 Thank you for your attention !!!
Questions ?!?
(Offline) DEMO !!!
Henning Zimmer