1. Interactive Sand Art Draw Using RGB-D Sensor Presenter : Senhua Chang

2. Main Purpose Present an interactive sand art drawing system which supports the common sand drawing functions, such as sand erosion, sand spilling, and sand leaking by using one RGB-D sensor, such as Kinect. Firstly, system adopts a vision based bare hand detection method which computes the hand position and recognizes the hand gestures. Secondly, the drawing position and the drawing action are sent to the sand drawing system. Finally, the sand drawing system performs the corresponding sand manipulation action.

3. Introduction What is Sand Art ? Sand drawing is a kind of drawing technique that people manipulate the sand on beach or sandy ground to create beautiful pictures. Sand art is a style of live performance art that introduces an indoor and dynamic way of sand painting. Sand art Performance is improvisational and continuous. How about see a good video to get experience what is Sand Art ?

4. Introduction Critical issues that need to tackle First, we need to use the gestures of one hand to control the drawing actions. Second, once the drawing actions are detected, we need to analyze the motion and gesture of another hand to draw virtual sand.

5. Introduction Critical issues that need to tackle A vision-based hand tracking system using one RGB-D sensor is used which is integrated with the sand drawing system. The hand tracking system computes the positions of the hands of a user and also recognizes the hand gestures. Then, the hand tracking system drives the sand drawing system to draw sand.

6. Introduction Why Kinect ? Kinect is a low-cost RGB-D sensor. What is RGB-D sensor? RGB plus Depth ( represent as a gray pic) Each pix represent the depth from sensor to objective.

7. Introduction

8. Related Work HCI based on hand interaction There are fruitful techniques that have been developed in the field of HCI, such as hand detection and gesture recognition techniques. The resulting information can be applied to interact with virtual objects. As a hand has high degrees of freedom, it is critical to compute hand features that are needed in performing HCI. The hand detection and gesture recognition are two major tasks that should be performed properly in the applications of using hand interaction.

9. Related Work HCI for artistic authoring The HCI itself is an important element in designing artistic authoring system. In painting, the writing pad is an intuitive tool that people can use the stylus pen to draw directly on the drawing surface. Haptic-based interface Gesture- based interface

10. Related Work Particle System and Sand Simulation Sand has liquid-like fluidity. But its shape is solid when it is static. Also, the sand drifts in the air like gas. Since sand has so interesting motion behaviors, the methods for sand simulation are studied extensively in graphics. The methods can be categorized into particle based methods and height map based methods. Also, there are hybrid methods that combine both a particle system and a height map to simulate sand.

11. Related Work Particle System and sand simulation There are techniques which are devoted to sand art called SandCanvas which is a multi- touch media for sand art drawing. The system support a rich set of gestures to manipulate sand. The user study indicates that the system is useful in creating interesting sand art pictures. The sand drawing system combined a particle system and a height map. For the sand in the air, particles are used. If the sand piles up on the drawing plane, the sand is converted into a height value. Thus, the computation of the approach is efficient. Lets see a video about SandCanvas.

12. System Overview System allows A user can use gestures of one hand to select the action and uses the gestures of another hand to manipulate sand in the virtual environment. system consists of three phases: image preprocessing, hand feature extraction, and motion/gesture management. The purpose of the image pre-processing phase is to eliminate the noise in the captured data and the regions that do not belong to a hand, such as human face. After the image is filtered and the portion for a hand is kept, we can compute the features of the hand, such as hand position, and then determine the hand gesture. The motion/gesture management is the bridge between the user and the sand art drawing system. The extracted hand data, such as the position of the hand and the hand gesture, are passed to the sand art drawing system. Then, the virtual sand is drawn in the virtual environment.

13. System Flow Chart and Interface

14. System Flow Chart and Interface

15. System Image Pre-processing In the image pre-processing phase, our goal is to remove the redundant portions that do not belong to hands in the captured color and depth images. In this way, the search space for the hands can be reduced. To extract the portion of the hand in an image, we perform background subtraction, depth segmentation, and skin color detection.

16. System Image Pre-processing Depth segmentation

17. System Hand feature extraction In the hand feature extraction phase, the goal is to analyze the image and extract the information that is useful for the application of sand art drawing.

18. System Hand feature extraction Step1: distance transform Step2: finger extraction based on the time series curve. Step3: Computation of the finger position

19. System Motion/gesture management After studying user's habits in sand art drawing so that we can properly design the key gestures for the sand art drawing system. We also note the following facts about sand drawing actions: 1. When users use their hands to erode the sand, they tend to use only one finger. The index finger is used the most. 2. In sand spilling and leaking, the users grab the sand and release a portion of the sand from their hands. 3. In area erosion, users may open their palm, and use their palm to feel about the sand and erode the sand within the area.

20. System Motion/gesture management

21. System Motion/gesture management

22. Experimental Result Environment

23. Experimental Result Environment

24. Experimental Result Environment

25. Experimental Result Result of gesture recognition

26. Experimental Result Result of sand art drawing

27. Experimental Result User testing study

28. Critique thinking This paper elaborate what is this sand art system and how it implemented. Do a lot of experiment, and the result is convincing. To much details can seem as an advantage and a shortcoming. ( A lot of paragraphs appear repeatedly)

29. MotionDraw: A Tool for Enhancing Art and Performance Using Kinect Presenter : Senhua Chang

30. Introduction Contemporary staged performances frequently utilize advanced lighting and projection techniques. The design and creation of these stage effects are rarely accessible to actual performers and must be designed by professional lighting designers or highly-paid programmers. With MotionDraw we want to create an affordable system that is easily controlled and manipulated by performers. With intuitive gestures, non-specialized users can control the MotionDraw visual library and interact with the captured visual record of their own movements.

31. Introduction Computer graphics, projection techniques, animations, and professional lighting enable effective stage effects in large-scale events that support the combination of live performance, such as dance, and digital arts. However, creating those exciting environments comes with the high cost of specialized hardware and often with dedicated software developed by professional programmers. MotionDraw, a Kinect-based tool that augments performance with projected visualizations created from tracked live movements of users, enables artists, performers, dancers and even the audience to easily design, create and control hybrid digital performances.

32. Related work Problem (1) the choreographer can be in close conversation with the computer operators and graphic artists, dancers and choreographers can not easily make changes to the graphic display on their own. (2) Projects like Sandde allow for three-dimensional digital drawing using gesture, but rely on hardware (a magnetic pen) to track movement.

33. The MotionDraw System MotionDraw consists of a set-up with one or two Kinects, that record the position of joints of a person's body over time and draws the resulting trail on a 3D canvas. One Kinect points at the dancers and tracks their movements. A GUI-based control panel, enables a separate user to change the parameters of the projected drawing in real-time, such as (a) change the color and width of the trails, (b) move the position of the camera-view, and (c) enable or disable the visualization of specific joints tracked. The program also saves the Kinect stream to do for further post-production. With a two-Kinect set-up the second Kinect captures a "conductor" who can interact directly with the MotionDraw system and change the visualization parameters through gestures. This integration is covered in more details below

34. Future Work

35. All is over Thank you very much