Visualization systems for driving immersive 3D (i3D) displays can be characterized as a combination of three hardware components. The first is computational power, which includes the CPU and main memory, and for simplicity here, storage systems. The second is graphics power, which at the core is the GPU (Graphics Processing Unit). The GPU is often referred to as a 'graphic pipe' or simply 'pipe', and can output to 1 or more 'video channels' or simply 'channels'. The third component is communications, which is the glue that holds the first two components together. It includes things such as networking, busses, etc.
Currently, many enterprise systems are capable of having multiple CPU's and GPU's as well as an internal architecture that interconnects them with high-speed buses. These systems easily support multi-screened systems and are very commonly used as the image generator for i3D displays. But, there is a growing trend to use commodity off-the-shelf hardware and PC workstations to drive multi-wall visualization display systems. The difficulty with using workstations is that they generally are single GPU systems. In order to display to a multi-wall system a cluster of these PC's need to be interconnected and have software that allows multiple synchronized images to occur. In response to this need the CAVELib has been enhanced to have the capabilities to support PC cluster setups.
For a cluster setup, the CAVELib has the responsibility for handling all internal data and frame synchronizations between the multiple PCs, and does so without the developer having to do anything to take advantage of it. In a cluster setup, the CAVELib works by having each PC running an exact copy of the application. Because of this, the CAVELib only needs to pass minimal state information each frame as well as have a barrier/acknowledgement mechanism for synchronization of each PC. Because only minimal data needs to be passed each frame, the CAVELib in a cluster setup does not have any unique networking requirements.
The one hardware requirement that any active stereo cluster visualization application will require is video genlocking. Genlocking is the mechanism of having independent graphics cards all raster, or display a particular frame buffer at the same time. The video signal that comes out of each graphics card is controlled by the card and not the CAVELib software. Since many interactive 3D displays use active stereo, the sequentially rendering of a left eye frame followed by a right eye frame, each of the graphics cards need to be displaying its left eye image at the same time as the other cards. The same is true for the right eye image. If the cards were not in sync, then when the shutter glasses had their left eye open the user may not see the left eye image on all of the screens, the images would be out of sync. Genlocking is a mechanism for making sure that the same image buffer is displayed from each card at the same time. This mechanism is usually solved by a hardware mechanism on the graphics cards themselves, as such, not all graphics cards have this capability.
To use a CAVELib-based application in a cluster setup, there is typically one PC per display screen. One of these PCs should be considered the master. Since the PC's are all running the same application, they should also have the same system capabilities. The system will only run as fast as the slowest PC in the cluster. Sometimes the master node will be more powerful than the slave nodes because it does extra work that is shared to the other PC's. If the setup is a four wall CAVE, for example, then there should be four duplicate PCs with the CAVELib application running on each, with a CAVELib configuration file that describes which wall that PC is responsible for. Occasionally, an extra PC is used for attaching input devices to, such as trackers and controllers. The data would then be shipped from that extra PC using the trackdserver to the master PC in the CAVELib cluster. The master PC in the cluster must also be the one that is running trackd if input devices are being used.
To run the CAVELib in a cluster setup, there needs to be one PC per display screen. One of these PCs should be called the master, and it does not matter which one is configured as the master. The CAVELib does not require an extra non-graphical PC to be the master node. If the setup is a four wall CAVE, then there should be four PCs with the CAVELib application running and each with a CAVELib configuration file that describes which wall that PC is responsible for. Occasionally, an extra PC is used for attaching input devices to, such as trackers and controllers. The data would then be shipped from that extra PC using the trackdserver to the master PC in the CAVELib cluster. The master PC in the cluster must also be the one that is running trackd if input devices are being used.
To run a cluster CAVELib application, the configuration file must be changed to denote that applications should attempt to run in cluster mode. The main CAVELib option is Distribution and it must be set to TCP to denote that this is a PC running in cluster mode. If the developer uses any of the CAVELib cluster API calls for sharing application data then another option, AppDistribution, must also be set to TCP in the configuration file. The total number of PCs in the cluster that are displaying to a wall or a screen must be specified with the DistribNodes option. The hostname or IP address of the PC that is considered to be the master node needs to be specified with the option DistribTCPMaster. Also, the IP port the CAVELib should use to transfer data between the master and the slaves must be set with DistribTCPPort option. The machine that is the master node is specified as the master by having its DistribID set to 0 in its configuration file; the other machines (slaves) must each have a unique value for the DistribID from 1 to N-1, where N is the number of PCs in the cluster.
The master node must also be the system that is running trackd if tracker or input devices are being used. When distribution is enabled, the tracker data, controller data (including keyboard and mouse), navigation matrix, and CAVELib's frame time are shared between the distributed machines. The master node sends the latest values to the slave nodes at the beginning of each frame, and receives an acknowledgement that each slave PC received the data before any PC is allowed to render the next frame. All of this happens without any effort by the developer. In fact, many CAVELib applications that run multi-wall on enterprise systems may run without modification on a PC cluster simply by recompiling for that platform, and running with a proper CAVELib configuration file.
If an application is doing large computations or reading data from a remote resource (disk, computational cluster, grid application, etc.) then the master node only should be responsible for this data as well as sharing it to the slave nodes. A developer would program the application so that only the master node is responsible for calculating or retrieving this new data and then have it share the data to each of the slave nodes. This could be done using standard networking libraries (RPC, PVM, etc.) or the CAVELib API for sharing data in a cluster may be used. To use the CAVELib functions for distributing application data, the AppDistribution configuration option must be set to TCP.
If an application is doing large computations or reading data from a remote resource (disk, computational cluster, grid application, etc.) then it's unlikely that a developer will want each node in the PC cluster to be doing this activity. That would be redundant. Instead, what a developer should do is program the application so that only the master node is responsible for obtaining or retrieving this new data and then have it share the data to each of the slave nodes. This could be done using standard networking libraries (RPC, PVM, etc.) or the API for data sharing that's part of the CAVELib may be used. To use the CAVELib functions for distributing application data, the AppDistribution configuration option must be set to TCP.
The CAVELib provides two mechanisms for sharing data, one is synchronized and one is not. The synchronized (aka syncdata) method forces each call to block until all of the data is received by itself and all of the other nodes. With larger amounts of data it is recommended that the syncdata API be used in the application's main process, as opposed to the display thread, because it is decoupled from the display thread, i.e. it runs independent of the frame rate. By passing the data into the main process it is the users responsibility to determine how and when the data should then be used by each of the display threads.
The other mechanism for passing data within a cluster of PCs uses a double buffering technique and does not require any of the nodes to block, this is called the display data (aka dispdata) method. In this mode, each node has two buffers, the write buffer that data is transferred in to by a separate thread, and the read buffer that the application gets the data from. When data is done being transferred from the master node into a slave's write buffer, the data is then swapped to the read buffer, and each call to the read function will access this same data until new data is swapped into it. There is no guarantee that all of the nodes will have the same values for their read buffer each frame, or that each node will perform the buffer swap at the same time. Unlike the syncdata mechanism dispdata does not use blocking to guarantee each node will have the exact same data each frame. Because of this, there is a possibility that different nodes will have slightly different data each frame. For data that changes by small amounts each frame, this lack of synchronization may be acceptable. The user needs to decide if overall application frame rate or data synchronization is the most important and then use the appropriate functions in the best way for their application.
The functions CAVEDistribOpenConnection(), CAVEDistribCloseConnection(), CAVEDistribWrite() and CAVEDistribRead() are the functions that pass data by the syncdata method. The functions CAVEAllocDisplayData(), CAVEPassDisplayData(), and CAVEGetDisplayData() are the functions pass data by the dispdata method.
Additionally, CAVEDistribMaster() returns true if its node is the master node, it can be used to have only the master node perform certain computations. CAVEDistribBarrier() can be used to synchronize processes on the multiple nodes. If Distribution isn't enabled, hence the application is not running in a cluster, these functions will all return immediately without doing anything and CAVEDistribMaster() will return true no matter which node it is executed on. Assuming AppDistribution is set in the configuration file CAVEDistribOpenConnection() must be called before CAVEDistribWrite(), CAVEDistribRead(), or CAVEDistribBarrier() are called; all of these functions should only be called from same thread/process that called CAVEDistribOpenConnection() for the connection channel they use (an application can open multiple data passing channels, in different threads/processes). CAVEDistribCloseConnection() should be called when the application exits, to close the communications channel and guarantee no system resources are left around.