Copyright © 2007 Indiana University Automated Customer Tracking and Behavior Recognition Raymond R. Burke and Alex Leykin Kelley School of Business Indiana University November 2, 2007 Copyright © 2007 Indiana University

What is Retail Shoppability? Definition: The ability of the retail environment to translate consumer demand into purchase

Copyright © 2007 Indiana University What is Retail Shoppability? Definition: The ability of the retail environment to translate consumer demand into purchase Components: Consumer Engagement: Making consumers’ needs salient in specific retail settings

Copyright © 2007 Indiana University What is Retail Shoppability? Definition: The ability of the retail environment to translate consumer demand into purchase Components: Consumer Engagement: Making consumers’ needs salient in specific retail settings Purchase conversion: Turning shoppers into buyers

Copyright © 2007 Indiana University What Determines Shoppability? Factors: Store, department, and category navigation Physical and visual clutter Product visibility and presentation Product organization Product information and value communication Presentation of new products Shopping convenience Shopping enjoyment

Copyright © 2007 Indiana University How Do We Measure and Manage Shoppability? Survey Research Measure consumer perceptions of the shopping experience and diagnose problems with store, department, and category shoppability

Copyright © 2007 Indiana University How Do We Measure and Manage Shoppability? Survey Research Measure consumer perceptions of the shopping experience and diagnose problems with store, department, and category shoppability Observational Research Track shopper behavior, identify points of engagement and purchase obstacles, and then manipulate and measure response

Copyright © 2007 Indiana University Observational Measures Engagement: –Examination of signs, displays, circulars –Category dwell time –Salesperson contact –Product/package/display interaction Conversion: –Aisle and category penetration –Purchase conversion rate –Product price/margin (absence of incentive) –Shopping basket size –Returns

Copyright © 2007 Indiana University Key Customer Touchpoints Store Entrance and Window Displays Lead Fixtures and Merchandising End-of-Aisle Displays High Volume / Margin Departments Customer Service Desk Checkout

Copyright © 2007 Indiana University Benefits of Computer Tracking Breadth of Coverage: –Census of customers/items (e.g., for security, inventory) –24/7 tracking (time of day/crowding analysis) –Potential to track entire store (path analysis) –Scalable to multiple stores (benchmarking, experiments) Speed: –Real time data (e.g., for staffing, replenishment) Data Integration: –Link path, penetration, conversion data to consumer demographics, shopping basket, purchase history

Copyright © 2007 Indiana University Computer Tracking Solutions: Tracking Carts with Infrared/RFID Sensors Limitations –Only applicable in retail stores using carts and/or baskets (e.g., grocery, mass retail) –Only tracks customers who choose to use carts/baskets, losing “fill-in” shoppers –Unable to track customers who leave carts. May overestimate perimeter traffic, dwell times –No measure of gaze direction or package interaction –No information on group size or behavior

Copyright © 2007 Indiana University Computer Tracking Solutions: Tracking Shoppers with Video Cameras Limitations –Cameras have a limited field of view and work best in smaller stores (e.g., specialty retail stores, drug stores, convenience stores, banks) –Tracking entire customer path requires multiple cameras with overlapping views –Occlusions (e.g., shelving, signage, other customers) and shadows can interfere with tracking –Difficult to distinguish between employees and customers

Copyright © 2007 Indiana University Tracking - System Overview Low-level Processing Camera Model Obstacle Model Foreground Segmentation Head Detection Tracking Jump-diffuse transitions Priors and Likelihoods Accept/Reject Candidate Event Detection Actor Distances Deterministic Agglomerative Clustering Validity Index Activity Detection Event Distances Fuzzy Agglomerative Clustering Adaptively Remove Weak Clusters

Copyright © 2007 Indiana University Tracking – Background Subtraction Color μ RGB I low I hi codeword codebook ……

Copyright © 2007 Indiana University Tracking – Background Subtraction The result of background subtraction is a binary bitmap Foreground regions corresponding to moving people are represented as blobs

Copyright © 2007 Indiana University Tracking – Detecting Heads The head is usually the least occluded part of the human body. Therefore, to reliably detect multiple people within one blob, we look at their head locations: 1.Estimate the height of each vertical line of the blob 2.Find a number of local maxima in the resulting histogram

Copyright © 2007 Indiana University Tracking – Detecting Heads (cont.)

Copyright © 2007 Indiana University Temporal Tracking Goal: find a correspondence between the bodies, already detected in the current frame with the bodies which appear in the next frame. Apply Markov Chain Monte Carlo (MCMC) to estimate the next state ? ? ? x t-1 xtxt ztzt ? Add body Delete body Recover deleted Change Size Move

Copyright © 2007 Indiana University

Swarming Shopper groups detected based on “swarming” idea in reverse –Swarming is used in graphics to generate flocking behaviour in animations. –Rules define flocking behaviour: Avoid collisions with the neighbors. Maintain fixed distance with neighbors Coordinate velocity vector with neighbors.

Copyright © 2007 Indiana University Tracking Customer Groups We treat customers as swarming agents, acting according to simple rules (e.g. stay together with swarm members) Customer groups

Copyright © 2007 Indiana University Defining Swarming Rules Two actors come sufficiently close according to some distance measure: –Relative position p i =(x i, y i ) of actor i on the floor –Body orientations α i –Dwelling state δ i ={T,F}. Distance between two agents is a linear combination of co-location, co-ordination and co-dwelling

Copyright © 2007 Indiana University Swarming The actors that best fit this model signal a Swarming Event Multiple swarming events are further clustered with fuzzy weights to find out shoppers in the same group over long periods

Copyright © 2007 Indiana University Activity Detection The shopper group detection is accomplished by clustering the short term events over long time periods. –The events could be separated in time, but they will be part of the same shopper group if the actors are the same (the first term).

Copyright © 2007 Indiana University Activity detection Higher level activities (shopper groups) detected using these events as building blocks over longer time periods Some definitions: –B ei ={b  e i } the set of all bodies taking part in an event e i. –τ ei and τ ej are the average times of events e i and e j happening.

Copyright © 2007 Indiana University Results: Swarming activities detected in space-time Dot location: average event location Dot size: validity Dots of same color: belong to same activity

Copyright © 2007 Indiana University Group Detection

Copyright © 2007 Indiana University Tracking Sequence number FramesPeople People missed False hits Identity switches %

Copyright © 2007 Indiana University Group Detection SequenceGroupsP+P−P−Partial Total Percent Ground truth (manually determined) false positives false negatives (groups missed) Partially identified groups (≥2 people in the group Correctly identified)

Copyright © 2007 Indiana University Qualitative Assesments Longer paths provide better group detection ( p val << 1 ) Two-people groups are easiest to detect Simple one-step clustering of trajectories is not sufficient for long-term group detection Employee tracks pose a significant problem and have to be excluded Several groups were missed by the operator in the initial ground truth –System caught groups missed by the human expert after inspection of results.

Copyright © 2007 Indiana University Tracking Example: Store View

Copyright © 2007 Indiana University Summary of Tracking Insights 1.Track customer path 2.Measure category penetration, dwell time, and conversion 3.Measure line queues and crowding 4.Cluster shoppers based on path similarity Evaluate store layout and product adjacencies Manage in-store communication, product assortment, and pricing Manage service levels, staffing Behavioral segmentation

Copyright © 2007 Indiana University Resources Questions? Indiana University’s Kelley School of Business