Introduction to Virtual Reality Mark Green School of Creative Media

Introduction Explore virtual reality: creative side technical side interaction between technology and creativity in a new medium concentrate on the practical side, with some theoretical themes as well

Virtual Reality I use the term Virtual Reality to encompass several related technologies: traditional VR, immersing the user in a computer generated 3D space augmented reality, combining real and virtual spaces computer / video games location based entertainment

Pushing the Envelop Virtual reality pushes the envelop in several ways: viewer participation technology interaction

Pushing the Envelop: Participation Most art / media separates the creator / performer from the viewer no interaction between the two modern media, such as film, video and sound recording have enforced this separation, both in place and time the viewer has no way of influencing the art work

Pushing the Envelop: Participation This separation of performer and viewer has been accelerated by modern technology some non-computer media support participation: jazz, improv theatre, some installation art, street performers but, in general artists don’t consider the viewer as an active participant in the creation of the art work

Pushing the Envelop: Participation For interactive art the viewer must be considered as partner in the creative process one of the main reasons for failure in interactive pieces must be more than trivial interaction, must pull the viewer into the piece provide the framework for the experience, not dictate the experience

Pushing the Envelop: Technology This is the bleeding edge working with technology that doesn’t always work early technology isn’t reliable early users must be prepared for failure, in return they have more freedom mature technologies are frozen, little opportunity for exploration and variations

Pushing the Envelop: Technology What are the implications? Must plan ahead, can’t do everything the night before must be flexible, it may not work the way you think it should share the facilities, expensive equipment is typically one of a kind

Pushing the Envelop: Interaction Most interaction is poor: keyboard and mouse “designing computer systems for one eyed dogs” use very few of our senses use very little of the I/O bandwidth of modern computers

Pushing the Envelop: Interaction How do we interact in 3D? Would like to have large display surfaces and a richer interaction vocabulary why build 3D objects with a mouse? Sculptors don’t do that, they work in a 3D space with 3D tools how can we do this with computers?

Art and Technology Haven’t always been separate, relatively recent mainly due to post-WWII compartmentalization of knowledge and push for specialization would not see this separation in a university 100 years ago

Art and Technology Artists only have access to a portion of what technology can do, sometimes very small software packages bias the media and the creative process without knowledge of the technology can only do what the geeks let you do restricts creative freedom

Art and Technology Example: Alias and SoftImage define what computer animation is they set a particular style which you must work within other types of computer animation are possible, but they don’t occur because the tools don’t support them

Art and Technology Beware of the bias that comes from software packages these tools could impose extreme restrictions on your creativity understand the technology and its possibilities, then push for the tools that allow freedom of expression, or do it yourself

What is VR About? Lets skip the technology and think about the medium thesis: “VR: an experience produced by computer technology” the key word is experience, convince users they are somewhere else quite often the term immersion is used, similar idea

Immersion or Experience User is unaware of the computer, other real surroundings “suspension of disbelief”, know its not real, but act like it is: dodge a ball try to lean on virtual furniture emotions, fear, phobias, etc

Experience How do we produce this experience? Consistency: all senses match, do what we expect cartoon visuals require cartoon motion and behavior photo-realistic visuals require accurate real world motion and behaviors follows the rules of the game

Experience Responsive: the user can influence the environment more than a ghost moving through an unreal space objects must respond to the user: can’t move through walls kick an object and it moves

Experience Complete: users don’t fall off the edge of the virtual world users can’t move into blank spaces environment should meet their expectations: sound when an object is hit experience may be limited, but it folds back on itself, doesn’t end unexpectedly

Design Process How do we design a virtual environment? No established design methodology, but we can start thinking about the process start with the theme, what the environment is about think about the ideal experience, what we could do without technical limitations

Design Process Determine the main features of the experience, what is important, what defines the experience now think about the technology how can we translate the ideal experience into something that is technologically feasible? Preserve the most important aspects of the experience

Example: MTR Consider an everyday experience, riding the MTR what are the important aspects of this experience? What happens on a typical MTR trip? Start with a high level sequence of events, the main events in the trip, key events

Example: MTR Key events: Start by waiting on the platform for the train try to get on the train train stops at each station, people get on and off optionally transfer between trains get off the train this gives us the basic script for the environment

Example: MTR What have we ignored at this level? Going to the MTR station leaving the MTR station the boundary of our environment is marked by the stations next look at more of the details of the experience, what happens on a finer time scale

Example: MTR Most of the action occurs in the train car what are the main aspects of this experience? People, density of people motion of the train sound of the train sound of the people in the train

Example: MTR Density of people varies with time and location within the system has major impact on the experience: what you can see, range of sight what the individual people look like how we can move through the car noise level

Example: MTR Motion of the train is an important part of the experience acceleration and deceleration in and out of stations, somewhat jerky motion train motion as it follows the tracks noise produced by train, due to tracks and motion of the train

Example: MTR Noise produced by people on the train loud conversations mobile phones ringing moving about train, getting on and off the train at stations

Example: MTR How do we translate the experience into VR? Take into account the limitations of the technology there are three main things to consider: geometry - what it looks like behavior - how it behaves sound- what we hear

Example: MTR How do we produce the visual appearance? Building detailed geometrical models is time consuming, want to avoid it as much as possible what are the main objects? stations train car people

Example: MTR Station: don’t need detailed model, only spend a small amount of time in them do a rough 3D model and then use texture mapping to fill in the details Train car: need more detail for this, we move through the car and possibly interact with its features some components move: doors

Example: MTR Need to model the main features of the car: doors seats railings and handles windows area between cars many of the visual details can be handled by texture mapping

Example: MTR People: doing good people is really hard, so we would like to avoid it individual people aren’t a major part of the MTR experience, rarely randomly meet people we know we need generic people, and we may need a lot of them this is our main modeling problem

Example: MTR Could do a full 3D model, but each model takes time and we need a lot, maybe 50 or 100 could do a few generic 3D models and dress them individually, could reduce the effort considerably and maintain quality we may need to use 3D models for the people close to us, can see depth

Example: MTR Most of the people aren’t close, and they are at least partially hidden by other people do we need a full 3D model for them, maybe something cheaper would be better use a flat model, ignore the 3D structure, use photos and texture maps to provide the details

Example: MTR What moves in this environment? Train car People The train car makes a number of motions The simplest is the doors that open and close, a pre-programmed animation can be used for this

Example: MTR A more complex example is the train’s motion along the tracks The car moves as it follows the tracks, but the user is sitting or standing in the car, their motion will follow the motion of the car If the following was perfect there would be no visible motion of the car

Example: MTR But this isn’t what we experience, we feel the car moving underneath us How do we simulate this? We could use a motion platform to physically move the user, but this is expensive Need to simulate the motion of the user relative to the car

Example: MTR If motion is smooth, no relative motion Jerkiness of train motion produces relative motion Time delay between motion of train and motion of user Delay has more impact when motion is jerky Need to build a simple physical model

Example: MTR Human motion is more complex How do we deal with the large number of people? Can’t animate each one individually: Too much work Many won’t be completely visible Won’t add much to the experience

Example: MTR Behavior is at two levels: High level, deciding what the character will do, where its going Low level, walking cycles, reaching, basic motions Each character will have a repertoire of low level motions, set of primitive motions that let it move around

Example: MTR flat models can have quite simple low level motions, a few pre-computed images that cycle as the person moves For 3D models need better motion could possibly develop a set of generic animations that could be used with a range of models, save some work

Example: MTR Higher level behavior is a bit more complicated what are the basic actions? Get on train get off train move about car the first is relatively easy, when the train arrives move to closest door

Example: MTR To get off the train, each character will make a decision when it arrives at station this could be a random decision, possibly based on how long the character has been on the train character could move from regions of high density to regions of low density, could also have random motions

Example: MTR Most of the sound in this example will be background noise easiest way to do this is capture sound on a real MTR train have a set of samples that are cycled through noise level could vary based on people density in cars

Example: MTR Other sounds are generated by people in the car: mobile phone rings conversations coughing, etc again they can be pre-recorded, but they should be spatial sounds, so they sound like they are coming from the right direction

Example: MTR The last thing to consider is how the user interacts with the environment there is not too much to interact with in this environment, if you ignore the possibility of interacting with other passengers the main thing the user does is move, this may require some response from the characters to get out of the user’s way

Example: MTR Some type of controller could be used for navigation could be based on pressing buttons or pointing could also use body motions to guess where the user wants to go, use some type of video technique

Observations Everyday experiences are infinitely detailed, need to determine the important details insects: part of many experiences, but rarely important there could be many insects, large effort to model, require lots of resources in the environment

Observations If insects aren’t important, we are wasting time and resources this problem isn’t restricted to “real world” environments scientific computations produce large amounts of data, sometimes many tera bytes how do we understand the results of these computations?

Observations Can’t look at each of the numbers, too many of them use visualization! in a still image each pixel could represent millions of numbers, animation can help even visually can’t represent each number in any detail

Observations Need to know the important features, ignore the details a more abstract domain, but the issues are the same how do we select the important details and present them in the best way highlight the important features of the data

Where do we go from here? Investigate technology: hardware software Look at some tools: modeling environment design

Where do we go from here? Look at some techniques environment design interaction collaboration Explore some possibilities where can we take the technology? what types of tools do we need? Possible applications