Accelerometer-based User Interfaces for the Control of a Physically Simulated Character Takaaki Shiratori Jessica K. Hodgins Carnegie Mellon University

Physical Simulation in Games Controllable interface for physically simulated character Jurassic Park: Trespasser, Simulation for everything. - Very limited. Little Big Planet, Simulation ONLY for passive objects. - Effective!

Half speed Physically Simulated Character Natural-looking motion. Natural responses to environment/disturbance.

Natural-looking motion. Natural responses to environment/disturbance. Hard to control due to delayed response (anticipation). Physically Simulated Character Walk  Jump – Our Hypothesis – Performing similar actions might make the delay seem intuitive Speed [m/s] Time [sec] Walk Jump Anticipation

Our Approach Performance interface: User Imitates character’s motion with Wiimotes. – Wrist interface – Arm interface – Leg interface Test our hypothesis about delay with user study. Leg interface

Accelerometer IR sensor $40 / unit 30 million copies Wiimote TM

Related Work Interface for controlling a character – Physically simulated character in 2D. Virtual navigation – Control “view point”. Wii games – Either dynamic or static measurement. [Johnson et al. 1999] [van de Panne and Lee 2003][Slyper and Hodgins 2008] [Chai and Hodgins 2005] [Templeman et al. 1999] [Razzaque et al. 2002] [Nintendo 2006][Nintendo 2007][Ubisoft 2007]

Overview 2-3 Wiimotes Physically simulated motion Mapping Controller selection Parameter change Acceleration Analysis Amp., Mean, Inclination Moving or not, Freq., Phase Motion controller (walk, run, jump, step) Physical Simulation

Overview 2-3 Wiimotes Physically simulated motion Mapping Controller selection Parameter change Acceleration Analysis Amp., Mean, Inclination Moving or not, Freq., Phase Motion controller (walk, run, jump, step) Physical Simulation

Focus on periodicity of character’s motion. User Input Walking RunningJumping Basic command: swing Wiimotes In-phase Out-of-phase

Acceleration Analysis Features Yes No Raw acceleration Frequency Amplitude Inclination Phase diff. L R Mean Kalman filter Moving? Variance

Frequency Auto-correlation function a : acceleration data T : current time t p : window size

Acceleration Analysis Features Yes No Raw acceleration Frequency Amplitude Inclination Phase diff. L R Mean Kalman filter Moving?

Phase Difference Cross-correlation function a : acceleration data a : mean acceleration T : current time t p : window size

Acceleration Analysis Features Yes No Raw acceleration Frequency Amplitude Inclination Phase diff. L R Mean Kalman filter Moving?

Inclination Estimation If Wiimote is not moving, If Wiimote is moving,  Wiimote’s local coordinate. x y z g t acc. x t mean

Overview 2-3 Wiimotes Physically simulated motion Mapping Controller selection Parameter change Acceleration Analysis Amp., Mean, Inclination Moving or not, Freq., Phase Motion controller (walk, run, jump, step) Physical Simulation

Hopper Model Hip: Ball joint (3 DoFs) Knee: Slider joint (1 DoF) Simulation Rendering

Basic Motions Stepping in place (Stopping) RunningWalkingJumping

Consists of 3 contact states with Proportional- Derivative (PD) controller. Support foot passes under hip. Rear leg leaves ground. Swing leg contacts ground. FallRise Double Support Walking Controller [Raibert and Hodgins, 1991] Stepping-in-place controller: target velocity = 0

Consists of 4 contact states with PD controller. Running Controller [Raibert and Hodgins, 1991] Jumping controller: both legs in phase.

Based on the robustness of motion. Gait Transition Jumping Stepping in place WalkingRunning WalkingRunning Slow Fast

Overview 2-3 Wiimotes Physically simulated motion Mapping Controller selection Parameter change Acceleration Analysis Amp., Mean, Inclination Moving or not, Freq., Phase Motion controller (walk, run, jump, step) Physical Simulation

Mapping Wiimotes to Physical Simulation Wiimotes not moving. Wiimotes out of phase. Wiimotes in phase. Height: Wiimote amplitude Jumping Stepping in place WalkingRunning WalkingRunning Slow Fast Wii frequency

Wrist Interface Imitate character’s leg motions with user’s wrists.

Wrist Interface Imitate character’s leg motions with user’s wrists. Walking (Slow swing) Running (Fast swing) JumpingTurning

Arm Interface Imitate arm motion of human’s biped motion. (though the character doesn’t have upper body)

Arm Interface Imitate arm motion of human’s biped motion. (though the character doesn’t have upper body) WalkingRunningJumpingTurning

Leg Interface Imitate character’s leg motion with user’s legs.

Leg Interface Imitate character’s leg motion with user’s legs. Walking (Slow step) Running (Fast step) JumpingTurning

Typical usage. Joystick Interface RightLeft Forward Run Walk Turn in place Step in place LocomotionJumping

User Study 15 subjects. Tasks – Straight track completion. – Test track completion. Questionnaire – Fun, ease of use, stress, familiarity, immersion, how much they liked it? – Free-form questions. Courses:

Task 1.Motion transition at line. 2.Keep straight walking/running until line. Straight Track Completion Average failure count Straight walking: All of our interfaces < Joystick

Failure for Straight Walking Approximate motions RightLeft Forward Run Walk Turn in place Step in place Precise manipulation Wrist ArmLeg Joystick Straight walk

Test Track Completion Simulation failureJump failureCurve failure + Time to completion Time to failure

Result of Test Track Completion Average countTime [sec] Curve failure: Leg interface < joystick Our interfaces are easier to control than joystick.

Result of Questionnaire “Fun”“Ease of use”“Stress” “Familiarity”“Immersion” “Like” Score

Free-form questions: Most subjects did not complain about the delay. A few subjects complained about the delay of all interfaces (including joystick). Questionnaire Rating score: “Immersion”: Wrist, Leg > Joystick “Like”: Wrist, Leg > Joystick

Insights from User Study Delay factors: – Acceleration analysis: 100 – 500 ms (Not included in joystick interface) – Physical simulation (anticipation): 200 – 500 ms Task completion: Easy to control: our interfaces > joystick. Questionnaire: “Immersive” and “Like”: our interfaces > joystick. Immersive performance interface might help with the delay issue.

Competitive Game

Conclusion Summary Performance interfaces for controlling a physically simulated character. User study mainly focusing on delay issues. – Performance interface might be able to help with delay issue. Future Work Reduce delay. – Improve acceleration analysis. Further user study. – Delay issue: Compare with data-driven control. – Other scenarios (e.g. fighting game, FPS).