Networking and Management Frameworks for Cluster-Based Graphics Benjamin Schaeffer Integrated Systems Lab University of Illinois

Syzygy Snapshots

Mathematics: Hyperbolic Geometry

Syzygy Library for Virtual Reality on PC clusters Supports multiple programming styles –Distributed Scene Graph –Synchronized application instances Portable: Linux, Win32, Irix High performance and high quality

The Problem PC graphics and networking performance is becoming comparable to the graphics supercomputers popular in the 1990’s How can we exploit this? Unfortunately, PC’s can only host 1 high performance graphics card (AGP bus) PC cluster is the answer

The Problem PC cluster has much lower communications bandwidth between components (shared memory vs. network) This bottleneck means that PC cluster VR software must be highly optimized to get high performance Different applications will require different architectures. Difficult for the library writer!

The Problem Cluster applications are more difficult to manage than applications running on a single computer. Starting the application? Stopping the application? Configuring the application? What if the cluster is heterogeneous?

Related Work CAVElib: Cluster support in the latest version. Limited automatic data sharing. Sockets-based network facility for other data sharing. NetJuggler and Cluster Juggler: Two different extensions to VR Juggler with cluster support. Multiple copies of the application share an input event stream.

Related Work WireGL: Replacement for OpenGL shared library that broadcasts OpenGL primitives over the network to be displayed by cluster nodes. Avango: Shared scene graph for shared virtual environments. Can be adapted for cluster-based graphics.

Application Architecture Two basic application architectures (there can be more) for cluster-based graphics Client/server Master/Slave

Client/Server The application runs on a central node and sends rendering information to the render nodes. Uses a fixed protocol. Generic rendering clients run on the render nodes. These are the same for all applications. Examples: WireGL, Avango (?), Syzygy distributed database

Master/Slave Multiple copies of the application run, one on each render node. A master instance of the application controls what the slave instances do, via broadcasting state information each frame. No fixed protocol. Unique for each application. Examples: CAVElib, Cluster Juggler, Syzygy Master/Slave application framework

Pros and Cons Client/Server can require high bandwidth (WireGL). Client/Server can require software rewriting (scene graphs). These higher-level protocols can be network efficient. Master/slave in many cases requires low bandwidth. Master/slave might not be suitable for complex applications.

Syzygy Provides a common networking and management framework for building cluster VR applications. Why? Many different ways to do cluster VR, each appropriate for different situations. As practical examples, build a distributed scene graph and a master/slave application framework.

Communications Layer PhleetI/O FrameworkMedia Protocols Media ObjectsI/O Drivers Application Frameworks

Messaging Syzygy communication is message-based. Programmer can construct message types and languages from those types. Binary data format with built-in translation between different machine architectures (big-endian to little-endian). Built-in message serialization and de- serialization

Communication Objects Syzygy includes server objects that can manage multiple full-duplex network connections. Communication occurs via Syzygy messages. Also client objects that communicate with the servers. Fault tolerance is built in. Objects can activate in any order, stop, restart, reconnect, disconnect, etc.

Synchronization BarrierServer object controls synchronization across the cluster. BarrierClient objects connect to it. Built on the standard Syzygy client/server objects. Synchronization groups are dynamic. Clients can enter and leave the group at any time. Fast. Buffer swap latency is comparable to the ping time on the cluster network for ethernet. Important for cluster graphics!

Synchronized Buffer Transfers For efficiency in graphics, information should be transferred once per frame. Small messages need to be buffered. Syzygy includes support for objects that automatically buffer messages in the background, transferring the buffers on the graphics buffer swap. Double-buffering of messages is used to get the most out of slow networks. While it is buffering the new messages, Syzygy is transferring the last buffer.

server client Send frame (n) Fill frame (n+1) Draw frame (n-1) Receive frame (n) Send Connection Request Send State Recv State Draw frame (n) IdleRecv frame (n+1) Idle Send frame (n+1) Fill frame (n+2) Send frame (n+2) Fill frame (n+3) Draw frame (n+1) Recv frame (n+2) Draw frame (n+1) Recv frame (n+2)

Phleet Distributed operating system (minimal) szgserver controls. szgd provides remote execution services. szgserver is merely a connection broker. Most communication occurs directly between programs.

Phleet Parameter database stored in szgserver. Meta- config file that can be managed from the command line of any computer. Global locks Simple message API routes through szgserver to all managed programs (like Unix signals). Can tell a program to reload its parameters or exit.

Distributed Scene Graph Client/server. One node serves geometry to the render nodes. Bandwidth is automatically conserved. Only scene changes are sent from one frame to the next. Semantic nature of scene graph helps ease programming. Robust. Render clients can disconnect and reconnect while the application is running. Dynamically change your display. Similarly, stop your application and start a new one without bringing down the render clients.

Distributed Scene Graph Built on a general hierarchical distributed database. Nodes in the database are created and later altered via sending Syzygy messages to the database. Incremental changes can be made to a node (for instance, altering just some points in a point set). Saves bandwidth. An API is provided to aid the programmer in generating these messages.

Distributed Scene Graph Node types include: –Transform –Points –Lines –Triangles –Texture coordinates –Textures –Visibility –Text Billboard –Etc.

Syzygy Server Barrier Server Graphics Server Input Client szgd Input Server Barrier Client Render

Master/Slave Framework Master/slave = same application runs synchronized on render nodes. One node is the master and distributes I/O or other control info. Framework makes it easy to write a synchronized application, handle input devices, and manage the whole thing. A good way to write a custom distributed graphics protocol!

Master/Slave Framework Programmer registers memory chunks. Event loop: –Pre-exchange callback (computations), input device polling on master. –Data exchange (handled by library) –Post-exchange callback. Input device state is synchronized on all application instances. As is registered memory. –Draw callback. –Synchronization.

MasterSlave Barrier Server Barrier Client Input Client Input Server Master/Slave Application Detail

Practical Experiences Volume visualization using master/slave framework. Visualization of MPI-based codes using distributed scene graph. Quake 3 level viewer. Master/Slave. Mathematical visualizations: Sphere eversion and hyperbolic geometry. Master/slave. Fully articulated avatars animated by motion capture data. Distributed scene graph.

CAVE Quake 3 //

CARMEN: shared virtual world //

Mathematics: Hyperbolic Geometry.edu/.edu/

Supercomputer Performance Analysis

Getting the Software Syzygy is licensed under the GNU LGPL It comes with everything you need to start showing demos right away! Download the latest release at (follow the software link) Or ask for a free Syzygy-0.4 CD after this talk!