UCSD Center for Hybrid Multicore Productivity Research

crca.ucsd.edu/sheldon crca.ucsd.edu/sheldon

Next Generation Culture will develop out of the capabilities that Multicore Computing enables in relationships that we can create between these areas: Assets, Dynamics and Behavior Computation for Virtual worlds and games Multi-user, Extensible Virtual Worlds Future Cinema – 3D, Ultra-high Resolution Production Next Generation Culture will develop out of the capabilities that Multicore Computing enables in relationships that we can create between these areas: Assets, Dynamics and Behavior Computation for Virtual worlds and games Multi-user, Extensible Virtual Worlds Future Cinema – 3D, Ultra-high Resolution Production Initial proposed industry partners: IBM, Intel, Sony, Emergent Game Technologies, Zaxel Inc.

Future Cinema – Ultra-high Resolution 3D Networked Production Delivery Prototyped and authored in virtual worlds Real (vs. Reel) time cinema Multi-point Telecommunications -with all of the above requirements. Future Cinema – Ultra-high Resolution 3D Networked Production Delivery Prototyped and authored in virtual worlds Real (vs. Reel) time cinema Multi-point Telecommunications -with all of the above requirements.

Ultra high resolution projection technologies 4K – 3840 x 2160 pixel displays today 8K and beyond tomorrow Provide qualitative advantages over film projection, driving adoption of digital cinema in coming years. Digital Cinema Auditorium ultra-high resolution cinema, 3-D sound; tele-presence conferencing; stereo imaging, Cine-Grid optical ultra-high bandwidth High Resolution gives 3D the needed fidelity. Drives the development of production techniques that require greater computing capacities at all levels.

Co-hosting virtual world at UCSD and Peking University, distributing application development and delivery between US and China – using high speed global optical network. Z/Series Mainframe Computers – providing large centralized processing for multi-user virtual worlds, large memory footprint for digital cinema development, extensive storage and data management techniques for distributed development Cell Processor Compute Servers provide accelerated parallel, vector computation IBM Mainframe and Bladeservers at at both UCSD and PKU. San Diego Supercomputer Center, Calit2 and CRCA collaborate on resource management. Compute/Power Advantages. Co-hosting virtual world at UCSD and Peking University, distributing application development and delivery between US and China – using high speed global optical network. Z/Series Mainframe Computers – providing large centralized processing for multi-user virtual worlds, large memory footprint for digital cinema development, extensive storage and data management techniques for distributed development Cell Processor Compute Servers provide accelerated parallel, vector computation IBM Mainframe and Bladeservers at at both UCSD and PKU. San Diego Supercomputer Center, Calit2 and CRCA collaborate on resource management. Compute/Power Advantages. Mutable Media Environments Relationships between Virtual Worlds and Digital Cinema

Global lambda grid is the “new Hollywood” as cinema becomes an end to end digital medium. This is the platform to deploy multicore resources.

WAAG, Amsterdam Calit2, UCSD Calit2 and NSF Optiputer Project Allow for High Speed Networks and Ultra-High Fidelity environments.

First ever real-time IP transmission of 4K resolution digital motion picture and 24 channel digital audio synchronized and mixed from multiple remote locations Special event was part of CineGrid demonstrations of networked digital cinema Demo took place during the Audio Engineering Society (AES) convention at the Letterman Digital Arts Center, Premier Theater in the Presidio of San Francisco, October 5-8 th, 2006 Audio Engineering Society Demo

Streaming 4K video and 24 channel audio from multiple locations Compensate for fixed latency with Max/MSP patch to maintain synchronization Demonstrate remote collaboration and tele-matic production

LucasFilm –San Francisco UCSD –San Diego USC –Los Angeles Keio University –Tokyo

AES Network Usage Vast majority of bandwidth is required for video 24 channels of uncompressed 24bit/96KHz is still relatively small SMPTE control data is also very minor

Stereoscopic 4K for D-ILA CineGrid 2008 Demo By Todd Margolis, Andrew Prudhomme, Jurgen Schulze, Robert Twomey, Rod Sterling, Stuart Levy, Robert Patterson, Greg Dawe, Sheldon Brown

Current Modes of Stereo 4K Pre-recorded Animations Rendered on parallel, multi-core systems. Live Interactive Computer Graphics GPGPU computation on NVIDIA With Multi-threaded x86

Hardware Diagram

(left eye) Zaxel Servers (right eye) (left eye) Zaxel Servers (right eye) (left eye) JVC Projectors (right eye) (left eye) JVC Projectors (right eye) (left eye) Dell Workstations (right eye) (left eye) Dell Workstations (right eye)

Producing CG animations 29 Mental Ray licenses running on 15 dual Opterons Scalable City (clip length 4:04) 244 seconds = 5856 frames = ~500 hours render time

Prepare Source Files Transfer data to encoding systems –12,067 frame transfer from NCSA to Cinethumper at UCSD Using RBUDP over optiputer, the speed varied greatly by hour of day Mb/s = minutes per/animation –Copy files from Cinethumper to 4 zaxels = Mb/s (SFTP) Transfer video to display systems –Copy AVI movies from external drive = 4 x 10 minutes

Encode TIFF files into AVI movies 1 minute of movie takes ~20 minutes to encode Progressive or Interlace(psf)? 25 – 60 minutes for all 4 videos to encode

Zaxel AVI Encoding (lossless)

FILE New Media Art Festival Sao Paulo Brazil, Yokohama Japan, San Diego US 1 st ever 4K feature film to premiere streamed to 3 continents at once. Program had HD teleconference between 3 sites projected in 4K quadrants.

4K 70 minute film with 5.1 audio streamed along with HD teleconference to: UC San Diego ---- FILE Sao Paulo --- Keio University - Tokyo 4K 70 minute film with 5.1 audio streamed along with HD teleconference to: UC San Diego ---- FILE Sao Paulo --- Keio University - Tokyo

Jim Chen’s VLAN proposal using Multicast in Chicago Ampath GOLE USP T-LEX GOLE U Mackenzie vlan 2712 vlan 2711 C-wave vlans 2711/2

Final Configuration: unicast VLAN assignments Ampath GOLE USP T-LEX GOLE U Mackenzie vlans 2712/3 vlan 2713 vlan 2712 vlan 2711 C-wave vlans 2711/2

4K Production Times 4K frame renders took over 24 hours Encoding to AVI movies took another 12 hours File transfers onto & off external drives took another hour Entire film transfer with ftp would take about 30 days ~5Mbps Using RBUDP entire film at 800 Mbps took about 40 minutes

HD transmission Sao Paulo – SD – Keio 3 plans Special Thanks to Hector Bracho for rendering these drawings

In: HD-SDI Sony SXRD 4K PLAN A 1. Sao Paulo - Keio = Ultragrid bi-directional with AJA cards a) send - ultragrid b) receive - ultragrid Sao Paulo – Keio Ultragrid Send to Keio Ultragrid Send to Keio In: Hd-SDI HDV cam Out: component Ultragrid Receive from Keio Ultragrid Receive from Keio Out: HD-SDI Converte r In: component Out: HD-SDI Audio AD In: analog Out: AES/EBU In: AES NETWORK O1V mixer O1V mixer Out: AES/EBU

iHDTV Send to SD iHDTV Send to SD In: component HDV cam Out: component NETWORK iHDTV Receive from SD iHDTV Receive from SD Out: component Converte r In: component Out: DVI Sony SXRD 4K PLAN A 2. Sao Paulo - UCSD = iHDTVw/ Blackmagic a) send - iHDTV b) receive - iHDTV Sao Paulo – San Diego In: DVI In: analog Out: analog O1V mixer O1V mixer

VLC Send to SD VLC Send to SD In: component HDV cam Out: component NETWORK VLC Receive from SD VLC Receive from SD Out: component Converte r In: component Out: DVI Sony SXRD 4K In: DVI In: analog Out: analog O1V mixer O1V mixer PLAN B 1.Sao Paulo - Keio a) send - VLC or iHDTV b) receive - VLC or iHDTV Sao Paulo – Keio

VLC Send to SD VLC Send to SD In: component HDV cam Out: component NETWORK Qvidium Receive from SD Qvidium Receive from SD Out: component Converte r In: component Out: DVI Sony SXRD 4K In: DVI In: analog Out: analog O1V mixer O1V mixer PLAN B 2. Sao Paulo - UCSD a) send - VLC b) receive - Qvidium Sao Paulo – San Diego

HD VTC H.323 HD VTC H.323 HDV cam NETWORK Converte r In: component Out: DVI Sony SXRD 4K In: DVI Out: analog O1V mixer O1V mixer PLAN C 1.Sao Paulo – SD - Keio a) send – H.323 b) receive – H.323 Sao Paulo – SD – Keio Out: component

2 cameras, 2 HD systems & 2 4K Playback systems

Install Equipment Zaxel 4K Servers 10Gb Switch UltraGrid HD 4K Streamers

2 cities with very different configurations

Configuring networks and equipment for this level of global service is currently non-trivial Dozens of people reconfiguring switches on many private networks Many dedicated technical staff at each site to configure content, computers, AV infrastructure, and networking Some of these people don’t get to sleep for a few weeks.

High Resolution Cinema development with same asset pipeline as game environment. Mining virtual world for behavior data and use as cinema prototyping platform for rapid development of cinematics.

Virtual World Generated Movie Trailer

Future Cinema – Milestones Year 1 and 2 Utilize Virtual World as Cinematic Production Environment Create “Virtual World Director” application Explore hybrid schemes of network delivery methods utilizing global Lambda Grid. Test implementations of “Machinima” computed and delivered across global Lambda Grid. Future Cinema – Milestones Year 1 and 2 Utilize Virtual World as Cinematic Production Environment Create “Virtual World Director” application Explore hybrid schemes of network delivery methods utilizing global Lambda Grid. Test implementations of “Machinima” computed and delivered across global Lambda Grid.

Future Cinema – Deliverables Year 1 and 2 Create real time 3D 4K projection environment Create 3D 4K movie production pipeline utilizing virtual world as production environment Devise methodologies for rendering algorithms to automatically generate 3D 4K frames Create database schemes that track changes in assets from virtual worlds to cinematic uses of same assets. Future Cinema – Deliverables Year 1 and 2 Create real time 3D 4K projection environment Create 3D 4K movie production pipeline utilizing virtual world as production environment Devise methodologies for rendering algorithms to automatically generate 3D 4K frames Create database schemes that track changes in assets from virtual worlds to cinematic uses of same assets.

crca.ucsd.edu/sheldon crca.ucsd.edu/sheldon