SM1205 Interactivity Topic 09: Motion Tracking Part I Spring 2012SCM-CityU1

Demos First Snow sweeper by PlaydoCAM (link)link – Make sure webcam is ready Spring 2012SCM-CityU2

Demos First Eyekanoid by PlaydoCAM (link)link Spring 2012SCM-CityU3

Demos First PlaydoJam by PlaydoCAM (link)link Spring 2012SCM-CityU4

How Come? Reason: you are watched by computers via webcams Computers know your movement – Relying on technology, called motion tracking Spring 2012SCM-CityU5

Motion Tracking (wiki)wiki To locate a moving object or several ones – Output: the location of moving objects How can a program know if there are moving objects and where they are? Idea: use special image processing algorithms to get regions of moving objects – Many algorithms exist for this purpose Spring 2012SCM-CityU6

Solution I Find the difference between current frame and a given background – Disadvantage: background must be static & fixed Disallow camera movement Disallow changes of lighting condition Currnt frameStatic background Spring 2012 SCM-CityU7

Solution II Find difference between consecutive frames – E.g., difference between frames i and i+1 – A more flexible solution! Spring 2012SCM-CityU8 Current frame Previous frame Note: white colors correspond to boundary of moving objects

Motion Tracking Library In w:\SM1205\L2_hongbo\ASTrackerLib\ In cityu.scm package (developed by Hongbo Fu) – Webcam class An easy-to-use wrapper for webcam access – MotionTracker class Main class for motion tracking – MotionTrackerSettingUI class User interface for parameter settings – MotionTrackerButton class Use motion to trigger button clicking Spring 2012SCM-CityU9 Note : classes are just special data types, which usually have their own properties & methods/functions

Motion Tracking Library The other three folders (namely, com, jp, net) contain third-party libraries, which our motion tracking library relies on Want to have a local copy of the libraries? – Simply duplicate the folder ASTrackerLib (including ALL its four sub-folders) Spring 2012SCM-CityU10

Source Path Source path + package tells exact location of source code of individual classes in the library To specify source path, add a new path into “Source path” in Dialog “Advanced Actionscript 3.0 Settings” – This step is needed for every application using motion tracking library Spring 2012SCM-CityU11

Source Path If you have a local copy of the library, add the path pointing to this local copy – E.g., if ASTrackerLib is within D:\SM3603\, then you need to add D:\SM3603\ASTrackerLib\ to source path in Flash settings Instead of w:\SM1205\L2_hongbo\ASTrackerLib\ Spring 2011SCM-CityU12

SM1205 Interactivity Webcam Class Spring 2012SCM-CityU13

Webcam Class Allow you to access streaming video from webcam Original webcam source has size 320 x 240 But you can scale up to any size, e.g., 640 x 480 Spring 2012SCM-CityU14

Webcam Class To use Webcam class, you first need to tell Flash CS5 where the corresponding source code (Webcam.as) is – Under cityu/scm/ folder Beside having specified source path, we also need to specify package name + class name – This is done by import statement in AS Spring 2012SCM-CityU15 import cityu.scm.Webcam; // import Webcam class Package name Class name

Webcam Class Initializing webcam takes time Webcam class provides a special event type, called Webcam.READY – To make sure you’re always using an initialized webcam Only within event listener for this event, you can guarantee webcam is ready to use Spring 2012SCM-CityU16

Webcam Class Basic usage of Webcam class Spring 2012SCM-CityU17 import cityu.scm.Webcam; // import Webcam class // create a webcam with size 400 x 300 var webcam: Webcam = new Webcam (400, 300); // webcam will dispatch READY event when the webcam is ready to use webcam.addEventListener( Webcam.READY, onCameraReady); function onCameraReady(e:Event): void { addChild(webcam); } import cityu.scm.Webcam; // import Webcam class // create a webcam with size 400 x 300 var webcam: Webcam = new Webcam (400, 300); // webcam will dispatch READY event when the webcam is ready to use webcam.addEventListener( Webcam.READY, onCameraReady); function onCameraReady(e:Event): void { addChild(webcam); }

Important Issue Webcam can be used by only one application Before running your application, make sure the camera is not being occupied by other applications Spring 2012SCM-CityU18 This light is on if webcam is being used by any application

Default Values for Function Parameters You can define functions with default values for function parameters – Advantage: you don’t need to pass all values every time when using functions E.g., function definition of Webcam usage of Webcam Spring 2012SCM-CityU19 function Webcam(w:Number=640, h:Number=480) {... } function Webcam(w:Number=640, h:Number=480) {... } // create a webcam with size 640 x 480 (default size) var webcam:Webcam = new Webcam() ; // equivalent: Webcam(640, 480) // create a webcam with size 640 x 480 (default size) var webcam:Webcam = new Webcam() ; // equivalent: Webcam(640, 480)

Webcam Class Like other symbol instances, Webcam instances provide a lot of display properties – E.g., Spring 2012SCM-CityU20 webcam.alpha = 0.4; webcam.x = 200; webcam.y = 30; Webcam’s (x, y) is located at its top-left corner

Webcam Class Use Webcam’s sendToBack method to avoid occlusion of stage objects by webcam – webcam.sendToBack(); Spring 2012SCM-CityU21

SM1205 Interactivity MotionTracker Class Spring 2012SCM-CityU22

MotionTracker Class Default visualization for tracked regions Able to track multiple objects Each tracked object can have its own clipping region – Motion happened outside clipping region will be ignored Spring 2012SCM-CityU23 Clipping regions

Simple Usage of MotionTracker Step 1: import MotionTracker class – I.e., import cityu.scm.MotionTracker; Step 2: initialize a MotionTracker object – With initialized Webcam object as input Spring 2012SCM-CityU24 var tracker:MotionTracker=new MotionTracker(webcam);

Spring 2012SCM-CityU25

Center of Tracked Region Use MotionTracker’s properties – trackX: Number; trackY: Number Center of tracked region With respect to top-left corner of webcam (instead of stage) – Usage example tracker.trackX; Spring 2011 SCM-CityU26 (0, 0) (trackX, trackY) Spring 2012

SCM-CityU27 trackX = 10 trackY = 20

Class Exercise Create a symbol instance on stage Make this shape follow center of moving region – Update symbol’s x, y using trackX and trackY, respectively Need event listener for ENTER_FRAME for continuous updating Spring 2012SCM-CityU28

Coordinate System Transformation Use DisplayObject’s methods – localToGlobal, globalToLocal Stage to Webcam – var ptWebcam:Point = webcam.globalToLocal(ptStage); Webcam to Stage – var ptStage:Point = webcam.localToGlobal(ptWebcam); Spring 2012SCM-CityU29 var pt:Point = webcam.localToGlobal( new Point(tracker.trackX, tracker.trackY)); var pt:Point = webcam.localToGlobal( new Point(tracker.trackX, tracker.trackY));

Hide Default Visualization Use hideTrackedArea and hideTrackedCenter methods – E.g., Spring 2012SCM-CityU30 tracker.hideTrackedArea(); tracker.hideTrackedCenter();

Collision with Tracked Regions Use MotionTracker’s methods – hitTestObject E.g. tracker.hitTestObject(cursor); – hitTestPoint E.g., tracker.hitTestPoint(cursor.x, cursor.y, true); You can also use isTracked method to perform collision detection only when object is tracked – isTracked return true if object is being tracked; false, otherwise. Spring 2012SCM-CityU31

Collision with Tracked Regions Usage of MotionTracker’s hitTestObject is very similar to that of DisplayObject’s Only difference: order matters – Wrong: cursor.hitTestObject(tracker); – Correct: tracker.hitTestObject(cursor); Spring 2012SCM-CityU32

Collision with Tracked Regions Usage of MotionTracker’s hitTestPoint is exactly the same as that of DisplayObject’s – hitTestPoint checks if a given point is within regions of pixels with white color Spring 2012SCM-CityU33

Pausing, Resuming and Removing of Trackers You may use the following methods – pause(): for temporarily stopping tracking – resume(): for resuming paused tracker – dispose(): for permanently removing a tracker Typically useful when leaving a frame with active tracker Similarly, use Webcam’s dispose() method to destroy a webcam object See example: MotionTrackerDemo-MultiFrame.fla – In W drive Spring 2012SCM-CityU34

Spring 2012SCM-CityU35 btnResume.addEventListener(MouseEvent.CLICK, buttonClick); btnPause.addEventListener(MouseEvent.CLICK, buttonClick); btnNextFrame.addEventListener(MouseEvent.CLICK, buttonClick); function buttonClick(e:MouseEvent): void { if (e.target == btnResume) { tracker.resume(); } else if (e.target == btnPause) { tracker.pause(); } else if (e.target == btnNextFrame) { tracker.dispose(); removeChild(webcam); webcam.dispose(); gotoAndStop(2); } btnResume.addEventListener(MouseEvent.CLICK, buttonClick); btnPause.addEventListener(MouseEvent.CLICK, buttonClick); btnNextFrame.addEventListener(MouseEvent.CLICK, buttonClick); function buttonClick(e:MouseEvent): void { if (e.target == btnResume) { tracker.resume(); } else if (e.target == btnPause) { tracker.pause(); } else if (e.target == btnNextFrame) { tracker.dispose(); removeChild(webcam); webcam.dispose(); gotoAndStop(2); }

Example: Shape Sweeper Spring 2012SCM-CityU36

Class Exercise Create a large number of circles on the stage and store them into an array for future use – Random size – Random position – Random opacity Tips: – addChild – push Spring 2012SCM-CityU37

Webcam Interaction For each shape which falls into tracked region, we remove it from stage – Use removeChild method – Need for loop How about corresponding elements in the array? – How to remove them from the array? – Use Array.splice method Spring 2012SCM-CityU38

Removing Array Elements in for Loop Correct solution: bottom-up iteration Spring 2012SCM-CityU39 var a:Array = [3, 3, 5, 6, 7, 3, 3, 5, 7, 3]; for (var i:int = a.length -1; i >=0; i--) { if (a[i] == 3) { a.splice(i, 1); trace(a); } trace("final: ", a); var a:Array = [3, 3, 5, 6, 7, 3, 3, 5, 7, 3]; for (var i:int = a.length -1; i >=0; i--) { if (a[i] == 3) { a.splice(i, 1); trace(a); } trace("final: ", a); i=9 i=8 Iteration 0 Iteration