1. SM1205 Interactivity Topic 11: Motion Tracking Part III Spring 2010SCM-CityU1

2. Button Click Usually done via mouse click Spring 2010SCM-CityU2

3. Button Click Can we click digital buttons by our fingers directly? Answer: Yes! – Through motion – Of course, you can also rely on touch sensors Need extra hardware Spring 2010SCM-CityU3

4. Today’s Examples Motion-driven button interaction Spring 2010SCM-CityU4

5. Today’s Examples Motion-driven button interaction Spring 2010SCM-CityU5

6. Motion Tracking Library In cityu.scm package (developed by Hongbo Fu) – Webcam class An easy-to-use wrapper for webcam access – MotionTracker class Main class for motion tracking – MotionTrackerButton class Use motion to trigger button clicking Spring 2010SCM-CityU6

7. Key Idea Analyze motion over a button on the stage – If motion is detected over it, dispatch OVER event Cf. MouseEvent.MOUSE_OVER – If motion continuously exists for a while, dispatch CLICKED event Cf. MouseEvent.MOUSE_DOWN To avoid undesired clicks after dispatching CLICKED event, clicking is disabled for a short period (say 2 sec) – If clicking is re-enabled, dispatch CLICKABLE event Cf. MouseEvent.MOUSE_UP Spring 2010SCM-CityU7

8. To create a motion-driven button, you need – Step 1: create some display object (e.g., btn) on stage By either design tool or ActionScript programming – Step 2: create a MotionTrackerButton object and associate it with btn – Step 3: add event listeners for motionBtn MotionTrackerButton.OVER MotionTrackerButton.CLICKED MotionTrackerButton.CLICKABLE MotionTrackerButton Class Spring 2010SCM-CityU8 var motionBtn: MotionTrackerButton = new MotionTrackerButton (webcam, btn ); var motionBtn: MotionTrackerButton = new MotionTrackerButton (webcam, btn );

9. Step 1: Create Display Object Draw some shape on stage Convert it to symbol Assign an instance name (e.g., btn) Spring 2010SCM-CityU9

10. Review: Instance Name vs Class Name Symbol instance name (cf. variable in AS) – Use this when you want to directly access this symbol instance in AS (e.g., btn.x = 20; ) Symbol class name (cf. complex data type in AS) – Use this when you want to dynamically create (possibly many) symbol instances in AS E.g., var btn2: MyButton = new MyButton(); Spring 2010SCM-CityU10

11. Step 2: Create MotionTrackerButton Object Before adding a MotionTrackerButton object, we need to initialize a Webcam object – We don’t need to explicitly create a MotionTracker object, since it is automatically generated for the associated button Spring 2010SCM-CityU11 import cityu.scm.Webcam; // import Webcam class var webcam:Webcam = new Webcam(640, 480); webcam.addEventListener(Webcam.READY, onCameraReady); function onCameraReady(e:Event): void { addChild(webcam); // to avoid occlusion of stage objects by webcam webcam.sendToBack(); webcam.alpha = 0.4; } import cityu.scm.Webcam; // import Webcam class var webcam:Webcam = new Webcam(640, 480); webcam.addEventListener(Webcam.READY, onCameraReady); function onCameraReady(e:Event): void { addChild(webcam); // to avoid occlusion of stage objects by webcam webcam.sendToBack(); webcam.alpha = 0.4; }

12. Step 2: Create MotionTrackerButton Object First import MotionTrackerButton class Then create a button with motion control – 3rd parameter: amount of motion needed to activate clicking How easy a button can be clicked – 4th parameter: time (in milliseconds) needed to re-enable clicing after dispatching CLICKED event How often a button can be clicked Spring 2010SCM-CityU12 import cityu.scm.MotionTrackerButton; var motionBtn:MotionTrackerButton = new MotionTrackerButton( webcam, btn, 10, 500 ); var motionBtn:MotionTrackerButton = new MotionTrackerButton( webcam, btn, 10, 500 );

13. Step 2: Create MotionTrackerButton Object Motion detection is clipped to the button area only – To see tracking visualization, make the button a bit transparent Spring 2010SCM-CityU13

14. Step 3: Add Event Listeners Event types – MotionTrackerButton.OVER – MotionTrackerButton.CLICKED – MotionTrackerButton.CLICKABLE Spring 2010SCM-CityU14 motionBtn.addEventListener( MotionTrackerButton.OVER, onButtonOver); motionBtn.addEventListener( MotionTrackerButton.CLICKED, onButtonClick); motionBtn.addEventListener( MotionTrackerButton.CLICKABLE, onButtonClickableAgain); motionBtn.addEventListener( MotionTrackerButton.OVER, onButtonOver); motionBtn.addEventListener( MotionTrackerButton.CLICKED, onButtonClick); motionBtn.addEventListener( MotionTrackerButton.CLICKABLE, onButtonClickableAgain); function onButtonClick(e:Event): void { trace("button clicked"); } function onButtonClick(e:Event): void { trace("button clicked"); }

15. Add More Visual Effects Idea: create multiple frames for the button and go to certain frame when a certain condition is satisfied – The button in our example is just a movie clip Review: MovieClip’s properties and methods (ref)ref – Note: we can access a frame by either frame number or label Spring 2010SCM-CityU15 Properties currentFrame: int currentLabel: String totalFrames: int Properties currentFrame: int currentLabel: String totalFrames: int Methods stop(), play() nextFrame(), prevFrame() gotoAndStop(frame) frame is a number or string gotoAndPlay(frame) Methods stop(), play() nextFrame(), prevFrame() gotoAndStop(frame) frame is a number or string gotoAndPlay(frame)

16. Add More Visual Effects Modify button symbol to have two key frames Frame 1 – Use the original shape – Frame label: up Frame 2 – Add some shadow filter – Frame label: down Spring 2010SCM-CityU16

17. Class Exercise Task 1: when button is clicked by motion – If btn’s current frame label is “up”, go to frame “down” – If btn’s current frame label is “down”, go to frame “up” – Tips Add code in onButtonClick Use currentLabel, gotoAndStop Spring 2010SCM-CityU17

18. Class Exercise Task 2: play with different values for 3rd and 4th parameters of MotionTrackerButton. – 3rd parameter: how easy a button can be clicked – 4th parameter: how often a button can be clicked Compare – MotionTrackerButton(webcam, btn, 1, 500); – MotionTrackerButton(webcam, btn, 20, 500); Compare – MotionTrackerButton(webcam, btn, 5, 50 ); – MotionTrackerButton(webcam, btn, 5, 2000 ); Spring 2010SCM-CityU18

19. Example: Piano with Motion Control How to interact multiple MotionTrackerButton objects? Spring 2010SCM-CityU19

20. Class Exercise Open MotionTracker-Piano-Skeleton.fla and insert a new symbol Frame 1 of this new symbol Frame 2 of this new symbol – No AS code needed for this frame Spring 2010SCM-CityU20

21. Class Exercise Create 7 instances of the new symbol on stage – Instance names from left to right: key1, key2, key3, key4, key5, key6, key7 Alternatively, you can create those instances in AS using for loop Spring 2010SCM-CityU21

22. Music Notes We use an array to store music notes Spring 2010SCM-CityU22 var notes:Array = [new S1(), new S2(), new S3(), new S4(), new S5(), new S6(), new S7()]; var notes:Array = [new S1(), new S2(), new S3(), new S4(), new S5(), new S6(), new S7()];

23. Create Motion Buttons We use arrays to store both piano keys and their corresponding motion buttons – Corresponding key and motion button have same index Spring 2010SCM-CityU23 // a list of motion buttons. e.g., keysMB[0] is for keys[0] var keysMB:Array = new Array(); var keys:Array = [key1, key2, key3, key4, key5, key6, key7]; // a list of motion buttons. e.g., keysMB[0] is for keys[0] var keysMB:Array = new Array(); var keys:Array = [key1, key2, key3, key4, key5, key6, key7]; for (var i:uint = 0; i < keys.length; i++) { keysMB.push(new MotionTrackerButton(webcam, keys[i], 5, 500)); } for (var i:uint = 0; i < keys.length; i++) { keysMB.push(new MotionTrackerButton(webcam, keys[i], 5, 500)); }

24. Event Listeners Spring 2010SCM-CityU24 for (var i:uint = 0; i < keys.length; i++) { // motion-driven keysMB[i].addEventListener(MotionTrackerButton.CLICKED, onKeyPressed); keysMB[i].addEventListener(MotionTrackerButton.CLICKABLE, onKeyReleased); } for (var i:uint = 0; i < keys.length; i++) { // motion-driven keysMB[i].addEventListener(MotionTrackerButton.CLICKED, onKeyPressed); keysMB[i].addEventListener(MotionTrackerButton.CLICKABLE, onKeyReleased); } function onKeyPressed(e:Event): void { for (var i:uint = 0; i < keys.length; i++) { if (keysMB[i] == e.target) { trace("index of pressed key", i); } function onKeyPressed(e:Event): void { for (var i:uint = 0; i < keys.length; i++) { if (keysMB[i] == e.target) { trace("index of pressed key", i); }

25. Extension to Other Instruments Spring 2010SCM-CityU25