Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science University College London

Overview Problem Statement & Requirements Data Representations (Contents) Execution Models (Dynamics) Latency

Reminder - VE is an Immersive, Mediated Communication Medium User Interface Devices Environment User Synthetic Environment Real Environment Mediated Medium

Ellis’s Content, Geometry, Dynamics Model Contents is data Dynamics is code or rules to change content Today we will look at common data and code representations

Sources and Sinks of Data Input “Data” –Data from trackers (e.g. 3D position plus 3D rotation for head and hand = 12D data) –Data from input devices (e.g. 2D joystick, buttons) –Possibly audio, haptic, physiological input Output “Data” –Displays systems, audio, video

Displays Have Different Requirements Video (N copies – for stereo and multiple screens) –Maintain copy of visual state –Render as fast as possible (~60Hz) –Synchronise with other renders Audio –Maintain copy of audio state –Render without glitches (requires fast interrupt) Haptics –Maintain copy of haptic data Render as fast as possible (~1000Hz)

“Under the Hood” of the Code We will find code modules for many different tasks: –Managing data and assets –Reading devices –Audio rendering –Video rendering –User input –Networking –Complex calculations such as physics (see seminar tomorrow!)

Overview Problem Statement & Requirements Data Representations (Contents) Execution Models (Dynamics) Latency

Graphs A graph consists of vertices and edges Vertices define the “state” information Edges define “relationships” Scene-graphs are directed and acyclic Arbitrary graphDirected graphDirected acyclic graph

Scene-graphs In a scene-graph, vertices are often called nodes –Store state information –Can include arbitrary property information All graphs have a root node which defines the base of the tree All other nodes divided into two types: –Group nodes –Leaf Nodes Root node Group nodes Leaf nodes

Group Nodes Group nodes have multiple nodes as children –Child nodes can be other group nodes or leaf nodes Applies common state information to multiple objects –State information propagates down the graph Examples include: –Transformations –Switch nodes –Effects Bump mapping, scribing, specular highlights –(In recent times) Shader programs

Examples (OpenSceneGraph) Anisotropic Lighting Scribing Cartoon Bumpmapping

Leaf Nodes Leaf nodes cannot have children State information relates to the appearance of specific objects Examples include: –Geometry –Image based rendering Billboards Impostors

Examples (OpenSceneGraph) ImpostorsBillboards

Overview Problem Statement & Requirements Data Representations (Contents) Execution Models (Dynamics) Latency

Dynamics These are the rules of interaction between the contents These can be: –Differential equations of Newtonian dynamics to describe kinematic and dynamic relationships –Grammatical rules for pattern-matched triggered actions Many different ways of doing this from imposing numerical approximations to Newtonian physics, through to plain old C++ / Java / XVR coding

Dynamics Representation Model Separate from the contents, how do we want to represent our dynamics? Its leads to quite critical computer science questions about the separation between code and data From a scene-authors point of view two basic models –Standalone process model –Within-scene-graph model

Implementing Dynamics as Standalone Processes Dynamics implemented as separate processes / threads Can change state of the graph in arbitrary ways –Change values of nodes –Add / remove nodes Dynamics

Problems of Standalone Treats scene-graph as a black-box, makes initial scene-creation very easy (load nodes in to scene- graph, changes a few fields on nodes) Composing scenes together, or instantiating existing objects is very hard End up with quite complicated code The code probably has variables in local scope which “should” be in the scene-graph Couples together rendering and animation to the same frame rate

Implementing Dynamics Within the Scene-Graph Fairly “autonomous” dynamics can be achieved by embedding dynamics within the scenegraph Animations are group nodes which apply state changes to their children Examples include: –Animation paths –Particle systems Animation Node Animated nodes

Example Animation and Particle System

Problems with Within-Scene-Graph Model Difficult to get all the code in there –e.g Difficult to write code that interfaces to other applications –Some valiant attempts such as VRML97/X3D Does provide for easy composition of scenes Different parts of the scene do need an explicit way of talking to each other –In VRML/X3D there are routes –In many other systems there is an event system Many systems provide a scripting language support –Python and Lua are very common

Execution Model Ties Everything Together So far we we’ve talked about a disparate set of systems: –A master environment –Separate representations for different output modes –User interfaces for controlling the environment The execution model “glues” all these parts together –Closely related to distributed systems as well

Execution Models Tying Things Together Example: –The position on an object is changed –The update needs to be reflected in: The master database The different scenegraphs Over the network (if connected) –How can all of this be coordinated? Two main models: –Kernel model –Actor model

Simple Kernel Model Treats a VR application like a traditional graphical application: while(true) { read_trackers(); set_body_position();//Changes scene graph do_animation(); //Changes scene graph render_left_eye(); render_right_eye(); render_sound(); poll_trackers(); }

do_animation After the user’s position is set by the read_trackers and set_body_position (see later in course) … … do_animation manipulates the scene As we have noted, is responsible for: –Animations, physics, interaction This is where the standalone or all the within-scene- graph dynamics are updated

Problems with Simple Kernel Model Everything happens in serial order Rendering only happens at a fixed rate, so if part of the animation slows down, the rendering slows down (very noticeable in many video games!) What if there are different output requirements such as haptics (1000Hz)? What if the input rates are much higher (e.g. 200Hz)

Modified Kernel Model (1) Has a fast loop which calls different functions at different frame rates: while(true) { fast_function(); if (elapsed>30ms) slow_function(); } fast_function() {read_trackers(), do_haptics();} slow_function() {render_left_eye();render_right_eye(); render_sound(); }

Modified Kernel Model (2) Uses a simple main loop, for the video rendering, but “background” threads for reading devices Main thread is very much like the Simple Kernel Model, but the function read_trackers and poll_trackers are now in a separate thread, and the main thread just copies information This is a good match to how operating systems actually schedule non-blocking input and output

Pros / Cons of the Kernel Model Advantages: –Simple to understand –Application programmer keeps their own data structures within the loop Disadvantages: –Implementation needs care because of different update rates –Usually requires some awareness of parallel programming issues –Lots of complexity ends up in the do_animation() method

Actor Model Virtual environment is realised by a set of collaborating asynchronous processes (actors) Actors send messages to one another Processes share a common database Database is a central repository of the scene graph Database Audio Video1 Video2 Tracking Speech Collision Application

Setting Object State Using in the Actor Model Setting the object state is often achieved using the subject-observer design pattern The object in the database is the subject Different renderers / networking systems are the observers When the subject’s state is updated, the observers are automatically notified

Pros / Cons of the Actor Model Advantages: –Application program does not care about distribution / what rendering systems used –Update rates are locally very good –Scales well to multiple cores Disadvantages: –Difficult to code efficiently –Difficult to query between code modules in different actors, needs very clear separation of responsibilities –End to end latency difficult to control (see next section)

Overview Problem Statement & Requirements Data Representations Execution Models Latency

End-to-End Latency Total time from head movement to scene movement Mine, Mark (1993). Characterization of End-to-End Delays in Head-Mounted Display Systems, UNC Chapel Hill Computer Science Technical Report TR

End-to-end latency

Problems Single process For the main CPU, tracker processing is actually mostly just “waiting”, could schedule something else IF and only IF (IFF) we know how long it will take –Previously we did poll and read separately If there are multiple video streams, we could do them in parallel IFF rendering changes no state in the scene graph (depending on your scene graph this isn’t a good bet!)

Example Timing Problem AnimateRenderAnimateRender T 0 = tracker poll T 1 = video output AnimateRenderAnimateRender T 0 = tracker poll T 1 = video output Ideal case Worst case

Summary Representing the environment is difficult The representation has to be rich enough to capture the contents, geometry and dynamics Each display mode requires its own form of the environment to optimise the display Want to make content as rich as possible to support dynamic models Otherwise behaviour is expressed only in code. At run-time there are logically concurrent processes (rendering, collision, audio etc…) Execution models need to reflect this concurrency