High Performance VR Jérémie Allard, ID/IMAG Jeremie.Allard@imag.fr Commodity Clusters for Immersive Projection Environments High Performance VR Jérémie Allard, ID/IMAG Jeremie.Allard@imag.fr

Goal From a bunch of PC toward a high performance plateform for intensive VR applications

VR Cluster How-To (1) Our new cluster First step: install the OS

OS Requirements on a Cluster Distributed environment Fast installation/update on several nodes Shared directory (NFS) User accounts (NIS) Remote shell service (SSH) Network access (DHCP, DNS)

Linux Cluster Distribution http://clic.mandrakesoft.com Based on MandrakeLinux Preconfigured solution to easily deploy and manage a cluster

Installation Simple and Fast Installation Install one node as a server Install one slave Automatic cloning to install all other slaves (200+ PCs in 20 min.) Cluster ready!

Tools Lots of integrated tools: Communication: MPICH, LAM-MPI Computing: ATLAS, LAPACK, SCALAPACK, PETSC Job Managment: OpenPBS Monitoring: Ganglia Administration: Ka-tools, pconsole, urpmi –parallel VR/Graphics: Povray, Net Juggler

VR Cluster How-To (2) Cluster installed Next step: Connect the projectors and stereo system

SoftGenLock: Multi-Display Active Stereo Alternate projection of the right and left eye images synchronized with shutter glasses Multiple projectors: glasses shutters must be flipped when all the displays are in the vertical blanking period genlock Vertical blanking Projector 1 Left Right Left Projector 2 Left Right Left Left eye Right eye Time

SoftGenLock: Software Implementation http://softgenlock.sf.net/ Software approach to enable active stereo and genlock on Linux clusters Synchronize the video signals Sequentially display left and right images Send signal to the glasses (LCD shutters) Works with potentially any graphics card New pixel-clock based algorithm (SoftGenLock 2.0) Master/Slave synchronization using the parallel port

SoftGenLock: Master/Slave Algorithm For each frame: On the master send a flip signal to the glasses and the slaves flip the displayed image On the slaves read the master signal accelerate or slowdown the video signal

SoftGenLock: Custom Parallel Port Network Slaves Glasses + ground connections Master Can be a simple tweaked cable for a few nodes:

SoftGenLock: Requirements Environments supported Linux with a standard 2.4 kernel Useful for testing but genlock may be unstable (jittering) Real-time kernel: RT-Linux, RTAI (preferred) Best performance (no jitter) Require to patch the kernel Future 2.5/2.6 kernels expected to have less jittering and much better scheduling precision

VR Cluster How-To (3) Displays ready Next step: VR Toolkit

VR Toolkits: VR Juggler http://www.vrjuggler.org Open Source Platform for VR Abstraction of the I/O devices Direct access to graphics API: OpenGL, Performer, Open Inventor, OpenSG Take advantage of the latest shader technologies Linux, Windows, IRIX, Solaris, HP Unix VR Juggler 1.0 released in april 2001, 2.0 currently in active development (first alpha in march 2003)

On-line Reconfiguration VR Juggler Simulator Mode Performance Monitoring PC + mouse + keyboard On-line Reconfiguration Configuration Files VJ Application Tracker + Displays + … vjUser, vjButton,…

VR Juggler Configuration system Configuration data are organized in ConfigChunks JugglerUser Name "User1" headPos { "VJHead" } interocular_distance { "0.229" } end Configuration editor GUI:

VR Cluster How-To (4) VR toolkit installed Next step: cluster support

Clustering Solutions for VR Juggler Data Lock at Input Event Level: Cluster Juggler Net Juggler Scene Graph Level: OpenSG Graphics Primitive Level: WireGL

Net Juggler http://netjuggler.sf.net Extends VR Juggler 1.0 for clusters Abstract the cluster into one system Only few modifications to VR Juggler applications source code One configuration for the whole cluster One operation to launch the application (using command line or GUI)

On-line Reconfiguration Net Juggler Simulator Mode Performance Monitoring Net Juggler On-line Reconfiguration Configuration Files (add each PC role) VJ Application PC1 -> Tracker + left display PC2 -> Front display PC3 -> Wand + Right display

Net Juggler communication layer (data lock) Net Juggler Implementation Configuration events Tracker event Wand events Net Juggler communication layer (data lock) VJ Application VJ Application VJ Application Net Juggler swaplock

Low bandwidth required Fast Ethernet Network sufficient Net Juggler Communications Data in transit (device + config events): Small amount Independent of the scene’s graphics complexity Communication Layer implementation: Data aggregation Optimized collective communications Scalable Low bandwidth required Fast Ethernet Network sufficient

Net Juggler application launching GUI: NjRun Set/Load/Save a configuration Actual nodes of the cluster Template configuration files : hosts are defined as @pc1@ Application executable Generate the configuration files and the mpi script Launch the application Kill the application

Cluster Juggler / VR Juggler 2 http://www.vrjuggler.org Currently (july 2003) in development Data Lock: Input event distribution scheme Directly integrated in VR Juggler 2.0 Based on TCP Heterogeneous cluster support

Net Juggler vs Cluster Juggler Both use similar distribution schemes Net Juggler is based on VR Juggler 1.0 and MPI Collective communication for high scalability Applications can use MPI based distributed computations Cluster Juggler is based on VR Juggler 2.0 and TCP Heterogeneous clusters ApplicationData: API for broadcasting application-specific chunk of data

VR Cluster How-To (5) VR toolkit with cluster support Next step: performance intensive applications

adhoc parallelization Developing applications for Net Juggler/Cluster Juggler Performance issues Replication on all nodes: Low communication overhead data and computations duplicated fps on one PC = fps on a X nodes PC cluster Performance improvements: distribute data and computations Application dependant bottleneck adhoc parallelization Tradeoff between distribution and extra communication costs

Distributed Interactive Fluid Simulation Fluids simulation: Stam’s Navier-Stokes equation solver 2D grid distributed on the nodes + parallel Navier-Stokes equation solver Implementation using the PETSc math library (based on MPI) Node 1 Node 2 Node 3 Node 4

NjFluid Results: Dual PIII-800 MHz GeForce 2 GTS 64 MB 100 Mbits/s Fast Ethernet Network 128x128 grid 20 fps 21 fps Solver distributed on 4 nodes 22 fps 24 fps Solver distributed on 6 nodes 8fps 8 fps Solver duplicated 4 displays 1 display

Everything is Open-Source! Summary We combined several tools Cluster OS: CLIC http://clic.mandrakesoft.com Stereo system: SoftGenLock http://softgenlock.sf.net VR Toolkit: VR Juggler http://www.vrjuggler.org Cluster support: Net Juggler http://netjuggler.sf.net Distributed computations: PETSC http://www.mcs.anl.gov/petsc Everything is Open-Source!

VR Cluster How-To (6) High performance VR plateform Next step: invent!

Advanced Applications Real-time video processing and scene reconstruction Simulation-based design Rich virtual environments Complex virtual environments MOVI, INRIA Jin Tech, VRAC

Toward Rich Virtual Environments Integrating multiple simulations in a graphical intensive world Fluid, Grass/Wind, Fur Better immersion Interactive simulations for industrial design Fast workflow Require advanced tools on going work at ID/IMAG