Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research

Motivation u There are 3 elements in computer graphics –lights, scene objects, and –the camera (focus of this talk) u Camera control is hard (7 DOFs) –position (3), direction (3), field of view (1) u We may learn from cinematography

Roles in cinematography u Low-level responsibilities –Cameraman Position and move the cameraPosition and move the camera –Film editor Keep the film to proper lengthKeep the film to proper length Decide shot transitionsDecide shot transitions u High-level responsibilities –Script writer and director Story-tellingStory-telling

Camera control level 1 u Input: camera position and direction u Output: camera transformation u Application: low-level graphics library routines (Direct3D)

Camera control level 2 u Input: show both A and B, follow C, etc. u Output: camera position and direction u Application: 3D game, computer animation package

Camera control level 3 u Input: high-level user directions –show a conversation –show a car chase scene u Output: a sequence of level 2 camera spec u Applications: graphical chat, tele- conferencing, Virtual Reality games

Camera control level 3 (cont.) u Encodes cinematographic expertise –an interesting problem in itself u Provides an interface that is –real-time –visually entertaining –more informative

Eyepeep BARG07011 Feldegast Microsoft V-Chat Lunar Island Eyepeep says "really" Feldegast says "yup" BARG07011 has joined the conversation Eyepeep says "im still older" Feldegast says "lol" DGeste309 does something silly. BARG07011 DGerste309 Eyepeep Feldegast GrenDayGod

Principles of Cinematography

Film structure Film Scenes Shots

Camera distance Close up Close shot Medium shot Full shot Long shot

The line of interest AB

External camera AB

AB

Internal camera AB

Apex camera AB

Moving cameras AB pan(A) track(B)

Some rules in film editing u Don't cross the line of interest u u Avoid jump cuts u u Let the actor lead u u Break movement

Don’t cross the line of interest 1 2

Some rules in film editing u u Don't cross the line of interest u Avoid jump cuts u u Let the actor lead u u Break movement

Avoid jump cut

Some rules in film editing u u Don't cross the line of interest u Avoid jump cuts u u Let the actor lead u u Break movement

Two-person conversation AB

2 3 1 ABAB A 4 B

Three-person conversation B A C

4 3 2 AB C AB 1 AB C

Bad

Good

Three spaces The universe space

Three spaces The eye space (z-axis is the look at direction)

Three spaces The screen space (screen is ctan(fov/2) away from eye point)

Standard look-at transformation Given eye position P from, a look at direction Want a rotation matrix R and a translation to transform a vector from universe space to eye space

Standard look-at transformation But this is not unique

Standard look-at transformation But this is not unique

Standard look-at transformation But this is not unique

Standard look-at transformation We will need to specify an up vector, usually [0,1,0]

Standard look-at transformation In eye space Z-axis is X-axis is Y-axis is Rotation matrix Translation is -P from

Internal close-up of actor A

Look at [x at,y at ] transformation Given eye position P from, look at point P at, and P at in screen space [x at, y at ] Want In universe space

Look at [x at,y at ] transformation In screen space: H = [x at, y at, ctan(fov/2)] Solve from the following equations:

External of actor A and actor B

Look at [x A,y A ] and [x B,y B ] Given P A,P B in universe space and eye space, and distance of eye position P from and P a. Want P from and

Use numerical method: 1.Set P from to 0 2.Solve R using the Look at [x at,y at ] method 3.Transform E by inverse of R, getting a new P from approximation 4.Goto 2 Look at [x A,y A ] and [x B,y B ]

Medium panning shot of actor A

References u Jim Blinn’s CG&A ’88 article –Where am I? What am I looking at? u Steve Drucker’s Interactive 3D ’92, ’94, ‘95 papers –Cast camera positioning as an optimization problem

System Design

Overall system diagram Real-time Application Virtual Cinematographer Renderer

At each time step 1. Application generates events to VC –( subject, verb, object ) 2. VC determines – –camera specifications – –acting hints 3. Renderer outputs the image Real-time Application Virtual Cinematographer Renderer

At each time step 1. Application generates events to VC – –( subject, verb, object ) 2. VC determines –camera specifications –acting hints 3. Renderer outputs the image Real-time Application Virtual Cinematographer Renderer

At each time step 1. Application generates events to VC – –( subject, verb, object ) 2. VC determines – –camera specifications – –acting hints 3. Renderer outputs the image Real-time Application Virtual Cinematographer Renderer

A networked virtual party game

Actors are simulated Actor’s mind Loneliness Thirst Boredom u Walk, converse, look around, drink, etc. u Can be controlled by the users

Networked virtual party game Server Client User actions Events

Inside the Virtual Cinematographer

VC architecture Idioms Camera modules

VC architecture Idioms Camera modules

u Geometric placement of specific cameras for each shot u u Choose the side of the line of interest u u Influence acting

Camera module: ext(B,A) AB

Static camera modules AB ext(B,A)ext(A,B) apex(A,B) int(B)

Moving camera modules AB pan(A) track(B)

Camera modules u u Geometric placement of specific cameras for each shot u Choose the side of the line of interest u u Influence acting

Don’t cross the line of interest 1 2

Camera modules u u Geometric placement of specific cameras for each shot u u Choose the side of the line of interest u Influence acting

Acting hints ABA’B’

VC architecture Idioms Camera modules

Previous works u Dave Christianson’s AAAI ’96 paper –Declarative camera control for automatic cinematography –An off-line algorithm –Need to know all events ahead –Use plan algorithms in AI

Film idioms u Capture a particular type of scene u Register relevant events u Select shot types u Determine the transitions between shots

Hierarchical idiom structure Lookaround Master Converse Drinking Moving 3Talk 2Talk

Hierarchical idiom structure u Avoids exponential growth of states u Allows expertise to be reused u Provides robustness

Moving idiom track(C)pan(A) ext(A,B) ext(B,A)apex(A,B)

Moving idiom track(C)pan(A) ext(A,B)ext(B,A) apex(A,B) T > 8

Moving idiom track(C) pan(A) ext(A,B)ext(B,A)apex(A,B) T > 8

Moving idiom track(C)pan(A)ext(A,B)ext(B,A) apex(A,B) D < 10

Moving idiom track(C)pan(A) ext(A,B) ext(B,A)apex(A,B) D < 5

Moving idiom apex(A,B) D < 5 apex(A,B) pan(A) ext(B,A) T > 8 ext(A,B) D < 10 T > 8

3Talk idiom int(C) ext1to2 2Talk(A,B)

Invoking a sub-idiom int(C) ext1to2 2Talk(A,B) ext(B,A) ext(A,B) 2Talk

Contributions u Real-time camera control –lightweight –automatic camera placement –automatic shot transitions u Do a reasonable job –not to replace human in computer animation

Work in Progress

Camera control for chat

Camera control for live meeting

Image after warping

Camera control for classroom u Multiple cameras per classroom –Lecturer cameras –Audience camera u Use audio and vision techniques to –Track the lecturer –Activate the audience camera

Dramatic lighting

Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research