1. Behavior Analysis Midterm Report Lipov Irina Ravid Dan Kotek Tommer

2. Main Goal Analyzing the people’s behavior in an office environment using streams from 3 video cameras and objects identified by a tracker as input. Analyzing the people’s behavior in an office environment using streams from 3 video cameras and objects identified by a tracker as input.

3. Main Goal (cont.) Initialization: Mapping the background objects (such as a computer, a phone, etc.) Initialization: Mapping the background objects (such as a computer, a phone, etc.) For each object, decide whether it is a person For each object, decide whether it is a person For each person, decide whether they are using a background object For each person, decide whether they are using a background object

4. Current State Manual Mapping Manual Mapping  For one camera Skin Detection Skin Detection Face Detection Face Detection Naïve Behavior Analysis Naïve Behavior Analysis  For one frame

5. Manual Mapping The function objects performs the manual mapping. The function objects performs the manual mapping.  Displays the input background image.  For each object, the user selects a polygon using the mouse and names it.  The polygon represents the object and/or it’s relevant surroundings.  Outputs A list of masks, one for each object, a list of object names, and the number of objects

6. Manual Mapping (cont.) michael001.jpg Objects

7. Skin Detection Builds a histogram of the colors in a file containing skin samples. Builds a histogram of the colors in a file containing skin samples. Skin samples Skin color histogram

8. Skin Detection (cont.) Remove background by subtraction. Remove background by subtraction. RemoveBG

9. Skin Detection (cont.) For each pixel in the subtracted image, the probability of it being skin is computed according to the histogram and a grayscale probability image is generated. For each pixel in the subtracted image, the probability of it being skin is computed according to the histogram and a grayscale probability image is generated.

10. Skin Detection (cont.) The image is thresholded to create a skin mask. The image is thresholded to create a skin mask.

11. Face Detection Assumption: A connected component in the skin mask image representing a face must contain at least one hole (eyes, nostrils, eyebrows, mouth, etc. will not have skin color.) Assumption: A connected component in the skin mask image representing a face must contain at least one hole (eyes, nostrils, eyebrows, mouth, etc. will not have skin color.) Small holes are removed by blurring the image. Small holes are removed by blurring the image. All connected components containing a hole are stored in a list of masks. All connected components containing a hole are stored in a list of masks. Each such mask represents a single face. Each such mask represents a single face.

12. Face Detection (cont.)

13. Naïve Behavior Analysis For each person (from the person list) and each background object (from the object list), checks whether their mask intersect. If so, the person is using the object. For each person (from the person list) and each background object (from the object list), checks whether their mask intersect. If so, the person is using the object.

14. What’s next Improvements Improvements  Mapping – generalize to 3 cameras (generate 3 lists of corresponding masks)  Skin detection – improve run times  Face detection – Find a more accurate criteria What’s really next What’s really next  Decide whether a tracker object is a person based on the images from all cameras by relating each face to the relevant object  Analyze the person’s actions  Choose the analysis perspective, i.e. whether to run state machines describing each person’s behavior or describing each object’s use  Write and implement the state machines as chosen