1. SM1205 Interactivity Topic 10: Motion Tracking Part II Spring 2011SCM-CityU1

2. Clipping Regions By default, motion tracking is done over the entire video domain – Check out MotionTracker-Skeleton.fla Spring 2011SCM-CityU2

3. SM1205 Interactivity Clipping Regions Spring 2011SCM-CityU3

4. Clipping Regions What if motion only within specific regions (called clipping regions) are needed? – Motion outside those clipping regions will be ignored Demo Spring 2011SCM-CityU4 clipping regions

5. Clipping Regions Specify a clipping region when creating a motion tracker – cl, ct, cr, cb are with respect to top-left corner of webcam – trackX, trackY are also with respect to top-left corner of webcam – You can have only ONE clipping region for one tracker Spring 2011SCM-CityU5 tracker = new MotionTracker(webcam, cl, ct, cr, cb ); tracker = new MotionTracker(webcam, cl, ct, cr, cb ); cl cr ct cb Clipping Region

6. Multiple Clipping Regions To have multiple clipping regions, you can create multiple trackers – All trackers share the same Webcam instance – Each tracker has its own associated clipping region Spring 2011SCM-CityU6 // initialize 1st motion tracker tracker = new MotionTracker(webcam, 0, 0, 200, 200); // initialize 2nd motion tracker tracker2 = new MotionTracker(webcam, 300, 300, 500, 500); // initialize 1st motion tracker tracker = new MotionTracker(webcam, 0, 0, 200, 200); // initialize 2nd motion tracker tracker2 = new MotionTracker(webcam, 300, 300, 500, 500);

7. Class Exercise Extend the previous example to have two clipping regions as follows – Hint: create two trackers with their own clipping regions – How trackers work at the overlapped areas of clipping regions? Spring 2011SCM-CityU7

8. Example: Ping-Pong Game MotionTracker-PingPong-Skeleton.fla in W drive Spring 2011SCM-CityU8 box barL ball barR box2

9. Class Exercise Use tracker and tracker2 to control barL and barR, respectively Spring 2011SCM-CityU9

10. Class Exercise Make ball bounce against stage borders – Please refer to Ping-Pong example in Topic 05 Spring 2011SCM-CityU10

11. Hit Test between Ball and Bars Let’s just use hitTestObject method Spring 2011SCM-CityU11 if (ball.hitTestObject(barR)) { trace("barR got hit"); } if (ball.hitTestObject(barL)) { trace("barL got hit"); } if (ball.hitTestObject(barR)) { trace("barR got hit"); } if (ball.hitTestObject(barL)) { trace("barL got hit"); }

12. Hit Test between Ball and Bars Collision response Spring 2011SCM-CityU12 Case 1: Ball is on left side of barR Case 2: Ball is on right side of barR

13. Hit Test between Ball and Bars But how to determine which case to handle? Solution: use signs of ball’s speedX – speedX > 0  Case 1 Response: put ball exactly at the left side of barR – speedX < 0  Case 2 Response: put ball exactly at the right side of barR Spring 2011SCM-CityU13 ball.x = barR.x - barR.width * 0.5 - r

14. Class Exercise Add sound effects when ball hits stage borders/bars – You can use Flash built-in sounds Hints – Export sounds for ActionScript Spring 2011SCM-CityU14 var ws:WoodSound = new WoodSound(); ws.play();

15. Moving Speed Use MotionTracker’s speedX and speedY to get the moving speed of tracked object speedX (in pixels) – Positive value if object moves along positive x-axis – Negative value if objects moves along negative x-axis speedY (in pixels) – Positive value if object moves along positive y-axis – Negative value if objects moves along negative y-axis Spring 2011SCM-CityU15 trace(tracker.speedX, tracker.speedY);

16. Example Transfer moving speed of tracked object to ball – E.g., hand Demo Spring 2011SCM-CityU16

17. Example Let’s modify the previous ping-pong example – Keep one tracker only, without any clipping region – No vertical bars – Add friction Spring 2011SCM-CityU17

18. Example Change ball’s speeds only if there is some intersection between ball and moving object Spring 2011SCM-CityU18 if (tracker.isTracked() && tracker.hitTestPoint(ball.x, ball.y, true)) { ball_speed_x += tracker.speedX; ball_speed_y += tracker.speedY; } if (tracker.isTracked() && tracker.hitTestPoint(ball.x, ball.y, true)) { ball_speed_x += tracker.speedX; ball_speed_y += tracker.speedY; }

19. Summary of MotionTracker Specify source path pointing to the library Spring 2011SCM-CityU19

20. Summary of MotionTracker Without clipping region With clipping region – cl, ct, cr, cb are with respect to top-left corner of webcam Spring 2011SCM-CityU20 tracker = new MotionTracker(webcam); tracker = new MotionTracker(webcam, cl, ct, cr, cb ); tracker = new MotionTracker(webcam, cl, ct, cr, cb );

21. Summary of MotionTracker Properties – trackX, trackY: center of tracked region With respect to top-left corner of webcam – speedX, speedY Moving speed of tracked object Spring 2011SCM-CityU 21 Spring 2010 21 (0, 0) (trackX, trackY)

22. Summary of MotionTracker Methods – hitTestObject To test if bounding boxes of display object and tracked region intersect with each other – hitTestPoint To test if a given point is within the tracked region – isTracked To check if any moving object is tracked – hideTrackedArea – hideTrackedCenter Spring 2011SCM-CityU22

23. SM1205 Interactivity Motion-Driven Buttons Spring 2011SCM-CityU23

24. Button Click Usually done via mouse click Spring 2011SCM-CityU24

25. Button Click Can we click digital buttons by our fingers directly? Answer: Yes! – Through motion – Of course, you can also rely on touch sensors Need extra hardware Spring 2011SCM-CityU25

26. Today’s Examples Motion-driven button interaction Spring 2011SCM-CityU26

27. Motion Tracking Library In cityu.scm package (developed by Hongbo Fu) – Webcam class An easy-to-use wrapper for webcam access – MotionTracker class Main class for motion tracking – MotionTrackerButton class Use motion to trigger button clicking Spring 2011SCM-CityU27

28. Key Idea Analyze motion over a button on the stage – If motion is detected over it, dispatch OVER event Cf. MouseEvent.MOUSE_OVER – If motion continuously exists for a while, dispatch CLICKED event Cf. MouseEvent.MOUSE_DOWN To avoid undesired clicks after dispatching CLICKED event, clicking is disabled for a short period (say 2 sec) – If clicking is re-enabled, dispatch CLICKABLE event Cf. MouseEvent.MOUSE_UP Spring 2011SCM-CityU28

29. To create a motion-driven button, you need – Step 1: create some display object (e.g., btn) on stage By either design tool or ActionScript programming – Step 2: create a MotionTrackerButton object and associate it with btn – Step 3: add event listeners for motionBtn MotionTrackerButton.OVER MotionTrackerButton.CLICKED MotionTrackerButton.CLICKABLE MotionTrackerButton Class Spring 2011SCM-CityU29 var motionBtn: MotionTrackerButton = new MotionTrackerButton (webcam, btn ); var motionBtn: MotionTrackerButton = new MotionTrackerButton (webcam, btn );

30. Step 1: Create Display Object MotionTracker-Skeleton.fla in W drive – Draw some shape on stage – Convert it to symbol – Assign an instance name (e.g., btn) Spring 2011SCM-CityU30

31. Step 2: Create MotionTrackerButton Object Before adding a MotionTrackerButton object, we need to initialize a Webcam object – We don’t need to explicitly create a MotionTracker object, since it is automatically generated for the associated button Spring 2011SCM-CityU31 import cityu.scm.Webcam; // import Webcam class var webcam:Webcam = new Webcam(640, 480); webcam.addEventListener(Webcam.READY, onCameraReady); function onCameraReady(e:Event): void { addChild(webcam); // to avoid occlusion of stage objects by webcam webcam.sendToBack(); webcam.alpha = 0.4; } import cityu.scm.Webcam; // import Webcam class var webcam:Webcam = new Webcam(640, 480); webcam.addEventListener(Webcam.READY, onCameraReady); function onCameraReady(e:Event): void { addChild(webcam); // to avoid occlusion of stage objects by webcam webcam.sendToBack(); webcam.alpha = 0.4; }

32. Step 2: Create MotionTrackerButton Object First import MotionTrackerButton class Then create a button with motion control – 3rd parameter: amount of motion needed to activate clicking How easy a button can be clicked – 4th parameter: time (in milliseconds) needed to re-enable clicing after dispatching CLICKED event How often a button can be clicked Spring 2011SCM-CityU32 import cityu.scm.MotionTrackerButton; var motionBtn:MotionTrackerButton = new MotionTrackerButton( webcam, btn, 10, 500 ); var motionBtn:MotionTrackerButton = new MotionTrackerButton( webcam, btn, 10, 500 );

33. Step 2: Create MotionTrackerButton Object Motion detection is clipped to the button area only – To see tracking visualization, make the button a bit transparent Spring 2011SCM-CityU33

34. Step 3: Add Event Listeners Event types – MotionTrackerButton.OVER – MotionTrackerButton.CLICKED – MotionTrackerButton.CLICKABLE Spring 2011SCM-CityU34 motionBtn.addEventListener( MotionTrackerButton.OVER, onButtonOver); motionBtn.addEventListener( MotionTrackerButton.CLICKED, onButtonClick); motionBtn.addEventListener( MotionTrackerButton.CLICKABLE, onButtonClickableAgain); motionBtn.addEventListener( MotionTrackerButton.OVER, onButtonOver); motionBtn.addEventListener( MotionTrackerButton.CLICKED, onButtonClick); motionBtn.addEventListener( MotionTrackerButton.CLICKABLE, onButtonClickableAgain); function onButtonClick(e:Event): void { trace("button clicked"); } function onButtonClick(e:Event): void { trace("button clicked"); }

35. Class Exercise Task 1: when button is clicked by motion – If btn’s current frame label is “up”, go to frame “down” – If btn’s current frame label is “down”, go to frame “up” – Tips Add code in onButtonClick Use currentLabel, gotoAndStop Spring 2011SCM-CityU35

36. Class Exercise Task 2: play with different values for 3rd and 4th parameters of MotionTrackerButton. – 3rd parameter: how easy a button can be clicked – 4th parameter: how often a button can be clicked Compare – MotionTrackerButton(webcam, btn, 1, 500); – MotionTrackerButton(webcam, btn, 20, 500); Compare – MotionTrackerButton(webcam, btn, 5, 50 ); – MotionTrackerButton(webcam, btn, 5, 2000 ); Spring 2011SCM-CityU36

37. Pausing, Resuming and Removing of MotionTrackerButtons Like MotionTracker’s pause, resume and dispose methods, MotionTrackerButton class has same set of methods for similar functionalities Spring 2011SCM-CityU37

38. Pausing, Resuming and Removing of MotionTrackerButtons Spring 2011SCM-CityU38 btnResume.addEventListener(MouseEvent.CLICK, buttonClick); btnPause.addEventListener(MouseEvent.CLICK, buttonClick); btnNextFrame.addEventListener(MouseEvent.CLICK, buttonClick); function buttonClick(e:MouseEvent): void { if (e.target == btnResume) { motionBtn.resume(); } else if (e.target == btnPause) { motionBtn.pause(); } else if (e.target == btnNextFrame) { motionBtn.dispose(); removeChild(webcam); webcam.dispose(); gotoAndStop(2); } btnResume.addEventListener(MouseEvent.CLICK, buttonClick); btnPause.addEventListener(MouseEvent.CLICK, buttonClick); btnNextFrame.addEventListener(MouseEvent.CLICK, buttonClick); function buttonClick(e:MouseEvent): void { if (e.target == btnResume) { motionBtn.resume(); } else if (e.target == btnPause) { motionBtn.pause(); } else if (e.target == btnNextFrame) { motionBtn.dispose(); removeChild(webcam); webcam.dispose(); gotoAndStop(2); }

39. Example: Piano with Motion Control How to interact multiple MotionTrackerButton objects? Spring 2011SCM-CityU39

40. Music Notes Like previous digital piano example, we use an array to store music notes Spring 2011SCM-CityU40 var notes:Array = [new S1(), new S2(), new S3(), new S4(), new S5(), new S6(), new S7()]; var notes:Array = [new S1(), new S2(), new S3(), new S4(), new S5(), new S6(), new S7()];

41. Create Motion Buttons We use arrays to store both piano keys and their corresponding motion buttons – Corresponding key and motion button have same index Spring 2011SCM-CityU41 // a list of motion buttons. e.g., keysMB[0] is for keys[0] var keysMB:Array = new Array(); var keys:Array = [key1, key2, key3, key4, key5, key6, key7]; // a list of motion buttons. e.g., keysMB[0] is for keys[0] var keysMB:Array = new Array(); var keys:Array = [key1, key2, key3, key4, key5, key6, key7]; for (var i:uint = 0; i < keys.length; i++) { keysMB.push(new MotionTrackerButton(webcam, keys[i], 5, 500)); } for (var i:uint = 0; i < keys.length; i++) { keysMB.push(new MotionTrackerButton(webcam, keys[i], 5, 500)); }

42. Event Listeners Spring 2011SCM-CityU42 for (var i:uint = 0; i < keys.length; i++) { // motion-driven keysMB[i].addEventListener(MotionTrackerButton.CLICKED, onKeyPressed); keysMB[i].addEventListener(MotionTrackerButton.CLICKABLE, onKeyReleased); } for (var i:uint = 0; i < keys.length; i++) { // motion-driven keysMB[i].addEventListener(MotionTrackerButton.CLICKED, onKeyPressed); keysMB[i].addEventListener(MotionTrackerButton.CLICKABLE, onKeyReleased); } function onKeyPressed(e:Event): void { for (var i:uint = 0; i < keys.length; i++) { if (keysMB[i] == e.target) { trace("index of pressed key", i); } function onKeyPressed(e:Event): void { for (var i:uint = 0; i < keys.length; i++) { if (keysMB[i] == e.target) { trace("index of pressed key", i); }

43. Extension to Other Instruments Spring 2011SCM-CityU43