Authoring Directed Gaze for Full-Body Motion Capture Roald Melssen Matej Milinkovic Tomislav Pejsa Daniel Rakita Bilge Mutlu Michael Gleicher Matej

Table of Contents Introduction Related work Implementation Evaluation Limitations & Future Work Discussion Matej

Introduction Directed Gaze - Movement of line of sight Shows focus, personality and intent Involves movement of the eyes, head and torso Usually done by hand The Paper’s Goal - automatically adding editable directed gaze to a captured motion. Matej

Related Work Gaze Synthesis: - focus on Rapid eyeball movements (Deng et al. [2005];Lee at al.[2002]), or synthesize coordinated movements of the eyes, head and body toward targets (Lence et al. [2010];Heck [2007]) Peters et al. [2010] and Pejsa et al. [2015] introduce procedural models inspired by neurophysiological observations These methods are extended Designed to provide biologically plausible movements across the parametric range Matej

Related Work Gaze Inference: - Describes methods that analytically determine when and where the character should look Research on gaze control by Henderson [2003], has shown that people’s gaze is influenced by two mechanics: The spontaneous bottom-up attention and the deliberate top-down attention The paper focuses on bottom-up attention which determines the gaze using a contrast, orientation and motion[Peters and O’Sullivan 2003; Peters and Itti 2008] In other work gaze targets are determined from spatial and kinematic properties of scene entities, such as proximity, velocity and orientation [Cafaro et al. 2009; Grillon and Thalmann 2009; Kokkinara et al. 2011] Matej

Implementation Gaze Inference Gaze Synthesis Gaze Editing Determining gaze instances from mocap data, which consist of a duration and look-at target Gaze Synthesis Adjusting motion to gaze at target Based on inverse kinematics Gaze Editing Allow user control over generated gaze instances Feeds back into step 2 after edits are done Roald

Implementation: Gaze Inference A gaze instance is a tuple: is the gaze shift start frame is the fixation start frame is the fixation end frame is the gaze target (3D vector) is the head alignment parameter is the torso alignment parameter where... Roald

Implementation: Gaze Inference Input: motion capture data Output: a list of gaze instances That means, construct a list of tuples of Algorithm: Gaze Instance Inference Determine timing: Gaze Target Inference Determine target: Computing Alignment Parameters Determine head and torso alignment: G1 G2 G3 G4 Roald

Implementation: Gaze Instance Inference Need to calculate timing parameters: Idea: analyze joint angular velocities When someone gazes at a point, the body accelerates and decelerates The maximum acceleration is attained at the midpoint of the gaze event Angular acceleration of joint j at frame f is Normalized to 0...1 range: Probability of a significant gaze event occuring at frame f is J is the set of all joints is a weight for joint j which determines how important it is Higher weight for joints closer to the eye Roald

Implementation: Gaze Instance Inference Find frame that maximizes This frame likely contains a significant gaze event (a gaze start or gaze end) Thus, the interval between two significant gaze events is classified as a gaze instance G1 G2 G3 G4 Roald significant gaze events

Implementation: Gaze Target Inference We now know the timing of the gaze instance, defined by Now we want to find the look-at target Three heuristics: Character likely looks at point along movement direction of the head Character likely looks at important objects Character likely looks at objects just before picking them up or touching them Idea: build a 2D probability distribution of the eye’s view and pick as the target with the highest probability = directional term, = importance term, = hand-contact term Roald

Implementation: Gaze Target Inference

Implementation: Computing Alignment Parameters Only parameters left to compute: Idea: Project end rotation on the arc from the start rotation to the rotation that would fully align the head to the look-at target t and then determine arc ratio is the torso rotation at the gaze start is the torso rotation at the gaze end is the rotation that would fully align the torso with the target is the rotation that corresponds to no alignment at all Projecting on the arc between and gives us Finally, we calculate

Evaluation Authoring Effort is compared by measuring two metrics: Time taken and Number of keys set The average was 25 minutes and 86 keys for each minute of animation The computation time per minute of animation was 1.5 minutes Second comparison: effort required to edit gaze animation in a scene that already contained eye movements The experienced animator used MotionBuilder and a novice animator used their tool, the novice animator took ⅓ of the time and number of operations compared to the experienced animator Matej

Evaluation Animation Quality: ( 1 ) No gaze, ( 2 ) Recorded gaze (using eye tracker), ( 3 ) Hand-authored gaze and ( 4 ) synthesized gaze (Approach in the paper) Hypothesis: ( 1 ) synthesized gaze would be preferred over no gaze; ( 2 ) synthesized gaze would be preferred over recorded gaze; and ( 3 ) synthesized gaze would be seen as non-inferior to hand-authored gaze. Design: three separate studies, each consisted of five task trials and followed a within-participants design where participants were asked to choose between videos with each gaze, presented in a randomized order Matej

Evaluation Stimuli: included 5 x 4 video clips (five scenes with each condition), 9 to 14 seconds in length Scenes included ChatWithFriends, MakeSandwich, StackBoxes, StealGem and WalkCones Measures: ( 1 ) animator competence ( 2 )realism ( 3 ) communicative clarity Matej

Evaluation Results: The data from first study support hypothesis 1 The data from second study showed that synthesized gaze was not significantly preferred over recorded gaze The third study showed that synthesized gaze was not seen as non-inferior to hand-authored gaze The scene type had a significant effect on participants choice of recorded gaze Matej

Evaluation Results: The study shows that adding eye animation using the approach from the paper leads to improvement in the perceived quality of the animation over having no eye animation at all There is a small loss in quality compared to expert-crafted animations https://vimeo.com/191892437 (3:00) Matej

Limitations & Future Work Focus on modeling only directed gaze Improve on the gaze’s accuracy and sensitivity The current implementation of the system is in Unity Integration with commonly used 3D-animation software such as MotionBuilder or Maya Matej

Questions?

Discussion Is this technique actually useful for animators, or is it one of those techniques where researchers think it is useful, but in practice it’s not? We think it’s mostly useful for novice animators or short movies (kids shows) Roald