Visualization of Large-Scale Distributed Data

INTRODUCTION
The primary goal of visualization is insight. An effective visualization is best achieved
through the creation of a proper representation of data and the interactive manipulation and
querying of the visualization. Large-scale data visualization is particularly challenging because
the size of the data is several orders of magnitude larger than what can be managed on an average
desktop computer. Data sizes range from terabytes to petabytes (and soon exabytes) rather than a
few megabytes to gigabytes. Large-scale data can also be of much greater dimensionality, and
there is often a need to correlate it with other types of similarly large and complex data.
Furthermore the need to query data at the level of individual data samples is superseded by the
need to search for larger trends in the data. Lastly, while interactive manipulation of a derived
visualization is important, it is much more difficult to achieve because each new visualization
requires either re-traversing the entire dataset, or compromising by only viewing a small subset of
the whole. Large-scale data visualization therefore requires the use of distributed computing.
The individual components of a data visualization pipeline can be abstracted as:
The degree to which these individual components are distributed or collocated has historically
been driven by the cost to deploy and maintain infrastructure and services. Early in the history of
scientific computing, networking bandwidth was expensive and therefore scarce. Consequently
early visualization pipelines tended to minimize the movement of data over networks in favor of
collocating data storage with data processing. However, as the amount and variety of data
continued to grow at an exponential pace, it became too costly to maintain full replicas of the data
for each individual that needed to use it. Instead, by leveraging the widespread expansion of the
Internet and other national and international high-speed network infrastructure such as the
National LambdaRail 1 , Internet-2 2 , and the Global Lambda Integrated Facility 3 , data and service
providers began to migrate toward a model of widespread distribution of resources.
In this chapter we will first introduce the various instantiations of the visualization pipeline
and the historic motivation for their creation. We will then examine individual components of the
pipeline in detail to understand the technical challenges that must be solved in order to ensure
continued scalability. We will discuss distributed data management issues that are specifically relevant to large-scale visualization. We will also introduce key data rendering techniques and
explain through case studies approaches for scaling them by leveraging distributed computing.
Lastly we will describe advanced display technologies that are now considered the “lenses” for
examining large-scale data.
Collocated Data, Filtering, Rendering and Display Resources
Most visualization software packages have a pipeline architecture where raw data comes in at
one end of the pipeline from disk or the network, moves through a sequence of filters that process
the data on the CPU and generate computer graphics primitives (e.g. lines, triangles, splats,
pixels) which are rendered on the GPU, and displayed on a monitor at the other end of the
pipeline. Some filters deal with accessing data or generating data. Other filters convert data from
one form to another. Finally there are filters that deal with the creation of computer graphics.
Each filter has an explicit input and output format allowing compatible filters to be linked
together to move data through the pipeline, and new filters can be added to the library of filters to
support new data formats, graphics formats or rendering techniques.
For example, in 2008 the NASA funded ENDURANCE project sent an autonomous
underwater vehicle (AUV) into ice covered West Lake Bonney in Antarctica. The AUV took
measurements of temperature, conductivity, pH, and five other values at over 18,000 points
within the lake. If a scientist wants to see overall temperature trends in the lake then the pipeline
could read in the data, and plot each temperature value as a box at the appropriate X, Y, Z
location where each box is colored based on a transfer function that maps a given temperature
value to a particular color. To look inside the lake, the scientist may want to see the temperature
values only on a plane slicing through the lake. Here the pipeline could add a filter to interpolate
raw temperature values across the surface of the plane. The scientist might also want to compare
multiple values (e.g. temperature, conductivity, and pH) by plotting all three simultaneously.
The advantages of this kind of pipeline/layered approach were described in the 1987 report
from the first workshop on Visualization in Scientific Computing (McCormick, 1988) that in
many ways marked the official beginning of the field of Scientific Visualization. These
advantages include:
•
Easy adaptation to a variety of data sources and display devices;
•
Software that is tailorable to the needs of diverse applications;
•
The ability to add more techniques and algorithms;
•
Layers that can be distributed to different computers to better leverage their capabilities
and minimize cost;
•
Affording economically viable development of a shared and interoperable set of tools;
•
Software that can accommodate different budgets and equipment preferences of a broad
community of users;
•
Being able to address the demands of a broad scientific community in coherent unison
rather than on a case-by-case basis.
Rendering can produce geometric primitives such as points, lines, or surfaces made of
triangles that can be cached on the graphics card. When the user changes their viewpoint on the
visualization, these primitives can be quickly re-rendered without re-running the data access, data
conversion, and rendering filters in the pipeline. While adequate for point, vector, or surface data,
these polygonal representations are less appropriate for volumetric data. An alternative is direct
rendering where there is no geometric representation and individual data points are drawn as
‘splats’, or volumes are rendered through raytracing each pixel on the screen. Direct rendering
can give much better views of volumetric data, however as there are no graphics primitives to be
cached, the entire scene must be re-rendered when the viewpoint changes.
In addition to rendering the dataset itself, supplementary graphics are often needed to put the
data into context (e.g. in our example of the lake above, it might be helpful to see an outline of
the lake, or a North pointing arrow, or latitude and longitude lines) and to supply information
about the visualization (depth scale, legend, color bars, etc.) Filters in the pipeline generate these
supplemental graphics or read them from disk. These also must be re-drawn when the
visualization is re-rendered but supplementary graphics are usually very lightweight compared to
the data being visualized.
Rendering is usually done locally on the display machine. These visualizations can be viewed
in 2D on a typical monitor screen or projector, or in 3D using a stereoscopic monitor, CAVE©
(Figure 2) (Cruz-Neira, 1992), or other stereo projection setup. Rendering in 3D may make it
easier to disambiguate complex multi-dimensional datasets but the creation of the 3D version
requires rendering the dataset from two viewpoints – one for each eye. For users lacking a 3D
display, the ability to quickly rotate a visualization, and the ability to view the dataset from
multiple angles simultaneously can help disambiguate complex scenes. While this text focuses on
rendering data visually there is also a body of work focusing on ‘visualizing’ data using
sonification or haptics to augment visualization through the use of other senses. In these cases
most of the pipeline remains the same except that there is additional rendering and display needed
for these other senses.
The user interacts with the visualization by directly or indirectly modifying the filters in the
pipeline – either by adding or removing filters, or changing the parameters of the existing filters.
One of the major advantages of a pipeline architecture is that the entire pipeline does not need to
be re-run when a change is requested. If intermediate results are stored at each of the filters then
only the filters downstream from the affected filter need to be recomputed. This saves time at the
cost of needing more memory to hold both the original dataset and its intermediate forms. Once
the pipeline is set up correctly, similar datasets (e.g. multiple timesteps) can be fed through the
pipeline without changing the filter settings.The user can interact with the visualization in several ways. In the simplest case the user can
change their point of view, or rotate and scale the visualization to see the visualization as an
object from the outside looking in, or from within the visualization looking around. The user will
also typically want to turn individual visualizations or supplementary graphics on and off, change
the time-step in an animated sequence, change the color transfer function, or change the
representation of the visualization (points, surface, glyphs.) More sophisticated users will want to
dynamically change the pipeline itself. Interaction with the pipeline is typically through a
graphical user interface that is linked into the pipeline, allowing the user to interact with familiar
interface elements - buttons, sliders, check boxes that affect the filter parameters behind the
scenes. On desktop systems interaction is typically performed with a keyboard and mouse.
Immersive environments with head-tracking allow a user to more naturally change their point of
view by simply moving their head, but these environments typically use a hand-held pointer
instead of a mouse and keyboard making it harder to accurately change parameters quickly.
Given the large amounts of data in multi-dimensional datasets, the amount of data must be
reduced at several points prior to display; there will often be more raw data points than there are
pixels on the screen to display them, or space in main memory to hold them. For interactive
visualization, datasets often need to be reduced so they will fit on disk at the local site, further
reduced to fit in main memory at the local site, and further reduced to fit into memory on the local
graphics card(s). Visualization systems often store data at multiple resolutions to allow the user to
have fast interactive access to a lower resolution version of the visualization that can remain on
the graphics card. This allows a user to quickly manipulate the low-resolution version, for
example rotating a wireframe box showing the space the data takes up, or a visualization of only
every 100 th data item, while still seeing enough landmarks to do the manipulation accurately.
Once the user stops manipulating the low-resolution version of the visualization, the pipeline can
take the time to generate the full-resolution version, replacing the low-resolution version.
For real-time interaction with this kind of visualization there are limited filtering
opportunities. Interaction is very fast if the data being visualized fits in the graphics card, for
example changing the viewpoint on a polygonal surface representation or on volume data that fits
in the card’s 3D texture memory. If the user changes the visualization parameters on an existing
visualization (e.g. generating a new polygonal iso-surface or changing the viewable subset of
volume data) then various filters are going to need to be re-run. The CPU will need to use data
from main memory to compute the new visualization and send it to the graphics card. If the
necessary data are not currently stored in main memory then the CPU needs to move it in from
disk or from the network, and the user must wait for that data to be fetched before the pipeline
can execute and display the results. Prefetching data to local disk or main memory can be used to
reduce this wait time, but there is only so much storage space on the graphics card, in main
memory, and on local disk, so that space usage must be optimized for a given visualization. This
often means that an initial preprocessing step is needed to turn large datafiles into a hierarchy of
fixed sized blocks for faster transfer between the various caches.
With CPUs moving towards an increasing number of cores rather than each core becoming
increasingly faster, being able to parallelize the work of the visualization pipeline on a single
machine, as well as using clusters to parallelize the work across multiple machines is becoming
more important in decreasing computation time for very large datasets. Shading languages such
as GLSL and CG are being used to speed up the rendering on the GPUs. Languages like CUDA
are being used on GPUs to speed up computation by moving some of the computation off of the
CPU and onto the GPU. As with multiple CPUs, in these cases bus bandwidth can become
another limiting factor- if one cannot move the data fast enough into the CPU then one cannot
fully take advantage of the CPU’s processing power.
Non-Interactive, Offline Rendered Visualization
When computational simulations produce extremely large datasets, and/or when raw data
collected by scientific instruments, such as remote sensors, fill up storage systems, interactive
exploration of the results becomes extremely difficult. A common and effective approach has
been to produce animations and video files from the data that the scientist can explore at his or
her home institution and can share with colleagues.
Long running simulations such as weather or earthquake simulations, protein folding, high-
energy physics or astronomical computations fall in this category. They require large amounts of
data as initial conditions (the starting point of the simulation) and produce greater amounts of
data, often through a time series of results (a complete solution for each time step during the
length of the simulation).
When faced with such a scenario, scientists are often in a situation where the data are too large
to be gathered and shipped back to his or her office computer for analysis and visualization. The
data has to remain within the supercomputing facility hosting the computational simulation or the
data storage in the first place.
One solution then is to schedule batch (non-interactive) jobs that produce images based on the
simulation results. An image can be generated for each time step of the simulation, or an
animated camera can follow a predetermined path for a given dataset. More complex
visualizations can combine animations of numerous parameters (camera path, time series, change
in representation, selection of variables to be visualized, etc.). While the time to render one frame
can be rather large, the common approach is to apply an image-based parallelization where each
processor renders a distinct complete frame. This is relatively simple to implement and fits well
in the supercomputing infrastructure: a number of processor nodes can access a large commonly
shared file system hosting the complete simulation results. If rendering jobs access different time
step results, each processor will access unique files and maximizes I/O throughput. If rendering
jobs access the same data, the read-only nature of the process also maximizes file system usage.
The end result of the rendering job is a series of frames, usually kept in a non-destructive file
format, such as TIFF, to maintain data integrity. Since it’s a non-interactive process, high-
resolution, including stereoscopic 3D, as well as high quality images can be produced using
photo-realistic rendering techniques such as raytracing. Frames can be combined in various
movie formats of different resolution and quality for multiple purposes (result dissemination,
education and outreach, movie production, etc.).
The software used for such approaches varies greatly from very specific home-brewed
visualization software for a very narrow purpose (such as seismic wave propagation for
earthquake visualization), to scientific rendering toolkits (such as the ‘Visualization Toolkit’,
vtk.org, for medical or engineering applications), finally up to general-purpose rendering
packages (such as Autodesk Maya or Pixar’s Renderman). If available, graphics hardware can be
used to speed up such intensive rendering jobs.
Given the non-interactive characteristics of this process, the problem resides in specifying the
visualization and rendering parameters (variable selection, camera path, shading, lighting, etc.).
Scientists have devised different approaches: for a well-defined visualization technique such as
volume rendering, the main parameter to be selected is the transfer function that defines the
mapping from data values into color and transparency. This function can be selected off-line, on a
small dataset and then applied during full-scale rendering. A desktop user-interface or a web
portal can be used to define rendering job characteristics (data input, image size, quality, etc.) and
visualization parameters (viewpoint, transfer functions, etc.). More advanced systems let the user
preview the visualization in low resolution and quality but give full access to all the parameters of
the visualization (for instance Autodesk Maya). Even more advanced systems allow the user to be
immersed in the visualization by using a VR device such as the CAVE©, where the scientists can
interact with a proxy (reduced dataset) of the visualization and ‘choreograph’ the rendering to be
done. Donna Cox, leading the Advanced Visualization Laboratory at the National Center for
Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign, has
pioneered such an approach with the ‘Virtual Director’ system (Thiebaux, 2000).
With the advent of petascale computing, where the data produced is so large that it overflows
even the largest storage infrastructure, the approach of generating movies and animations directly
inside the supercomputing facility is gaining renewed interest taking the model in different
directions:
•
Compared to petascale sized datasets, even the highest resolution uncompressed
rendering is becoming attractive for its predictable and manageable size. A format
gaining traction is the so-called 4K format (4096x2160 pixel resolution) adopted by the
digital cinema initiative as the next generation movie distribution format. At four times
the resolution of high-definition television, it becomes possible to portray a simulation
dataset with exquisite detail.
•
While simulations can run for an extremely long time, it is often beneficial to couple the
simulation process with the visualization process, and generate early rendering results as
soon as possible while the simulation is still running. This technique, referred to as
computational steering if the user can interactively control simulation parameters,
enables early detections of problems such as programming bugs or incorrect initial
conditions in the simulation. The simulation-visualization coupling can be done through a
data checkpoint on storage, or through network communication between a simulation
cluster and a visualization rendering resource.
•
If a tight coupling between simulation and visualization is achieved by running both
programs on the same computational resource, in situ visualization can be performed,
removing the need for any storage or data transmission, hence speeding up the generation
of visual results.
•
Using high-speed networking, the rendering need not to be stored for later examination,
but can be streamed in full quality to the scientist at his or her desk.
The benefits of running the complete visualization pipeline all at once and generating
animations or video streams are:
•
Asynchronous work (“submit and forget”) for the scientists;
•
Non-interactive jobs where known batch optimizations can be applied, and predictable
usage of resources make the visualization process efficient and acceptable by the scientist
and the supercomputing community;
•
Early rendering as the simulation is running can be extremely beneficial;
•
The end product is much smaller than the simulation data (even for high-resolution
uncompressed rendering), and can still be cut and edited into a movie production pipeline
for education, dissemination and outreach.
The drawbacks of the approach are:
•
Non-interactive visualization- so the scientist needs to know what to look for, hence
limiting the visual exploration of simulation results.
•
May require work on proxy data beforehand to determine the visualization and rendering
parameters
•
Data has to be reloaded and re-rendered if modifications are needed.
Examples of Offline Rendered Visualizations
•
The San Diego Supercomputing Center and the Southern California Earthquake Center’s
TeraShake is the largest and most detailed simulation of what happens during a major
earthquake on the southern San Andreas fault, computed at 3000x1500x400 resolution.
Various volume renderings are used to produce movies. The parameters of visualization
are submitted by scientist through a web portal, which in turn schedules and launches
numerous jobs on large compute clusters such as the TeraGrid. The end results are time-
series animations for various earthquake scenarios.
•
The National Center for Supercomputing Applications’ classic tornado simulations
calculated the birth of a tornado, starting with data from weather conditions that produced
a powerful tornado in South Dakota. The results are choreographed and cinematic
renderings of this complex simulation data combining artistic and scientific expertise
translated the data into a dynamic, high-definition animated visualization. Data
representations (proxies) are loaded into software such as Maya, visualization parameters
are selected interactively in pre-visualization, and then jobs are launched for rendering on
a powerful cluster computer.
•
Accurately modeling the dynamics and structure of the Earth’s interior requires massively
distributed computational techniques - the use of high-performance PC clusters
generating terabytes of data which are then visualized through distributed volumetric
rendering methods to produce high-quality monoscopic and stereoscopic visualizations.
Global seismic wave propagation data are generated with software from the Theoretical
& Computational Seismology group at the California Institute of Technology. This
software produces highly accurate simulations of wave propagation. Global models for
each time-step are combined to create a global model of seismic wave propagation over
time. A one-hour seismic wave field divergence simulation of the 1994 earthquake in
Bolivia highlighting positive amplitude compression (P) waves at progressive stages
during propagation takes 21 hours to compute. The simulation data are redistributed
across the cluster according to the requirements of the specific visualization algorithm.
The 169 time steps of the simulation are sampled uniformly throughout model time,
making the total time for gathering and interpolating the data 33.8 hours, producing a 512
x 512 x 512 regular grid at 16-bit resolution (40GB-worth of data). An interactive tool is
used for staging and rendering monoscopic and stereoscopic 3D volumetric animations.
The approach allows the user to interactively select datasets, setup camera positions, pick
color, opacity and gradient transfer functions, and specify movie properties. The
rendering was performed on a TeraGrid cluster at the San Diego Supercomputing Center
on up to 64 nodes. The table below shows rendering times and speedups for a low-
resolution stereoscopic rendering and high-resolution monoscopic rendering of the 169
time steps of the simulation: it shows significant acceleration, where the stereoscopic
animation takes only 2 minutes and 30 seconds to generate on 64 processors, and the
high-resolution animation showing all the simulation details took only 28 minutes. The
end products, as shown in the following pictures, are time-series animations showing
three different views of the wave propagation through the earth.Fully Distributed Visualization
As the amount and variety of available data on the Internet grows, so will the means by which
the data can be filtered or mined, visualized and displayed. There is no single data mining or
visualization technique that will satisfy the needs of every application; the choice of technique is
highly dependent on the fundamental question being asked about the data. The power of the
Internet is maximally leveraged when storage, filtering / mining, visualization and display
capabilities are presented as independent services that can be combined in a myriad different
ways, distributing the visualization pipeline around the planet. Google maps is an example of how
the coupling of a map service with a data service can provide wholly new forms of information
and insight.
A key technological enabler for a distributed service-oriented paradigm (sometimes referred to
as Grid Computing or Cloud Computing) is an abundance of high-speed networking.
Traditionally, networking was considered expensive and it was therefore impractical to move
large quantities of data over them. As a result, data filtering and mining algorithms were
collocated with the source of the data. This made it difficult for new filtering and mining
algorithms to emerge as it was not possible for the original data providers to anticipate all the
different ways in which their users might want to use the data. However with the continual
decrease in networking costs, a fully distributed model is quickly becoming practical.
In 2002, the National Science Foundation’s OptIPuter project sought to envision how
unlimited networking bandwidth could change fundamental notions of computing and the way
scientists conduct basic research (Smarr, 2003). In the OptIPuter model, all computing resources
were fully distributed, and the networks that tied the resources together were thought of as wide-
area system busses rather than networks. Therefore a cluster of computers with massive amounts
of storage was collectively considered a single virtual disk drive, and a cluster of computers with
high performance graphics cards was thought of as a single graphics card. The project resulted in
the creation of numerous innovations which include: ultra-high-speed network transport protocols
that could routinely move data over 1-10 Gigabit/s networks between individual compute nodes
at close to the full line rate of the network (Vishwanath, 2007; He, 2002; Xiong, 2005); ultra-
high-speed national and international networks that could be provisioned directly by the
applications through the allocation of dynamic light paths (DeFanti, 2003); wide-area RAMDisk
caches that could improve I/O-bound scientific applications by five-fold (Vishwanath, 2008a);
and new middleware and hardware approaches to drive network-enabled displays with hundreds
of megapixels of resolution (Jeong, 2006). The OptIPuter project members were able to
successfully demonstrate that these individual systems could be daisy-chained to produce a
planetary-scale distributed computer (Zhang, 2003).
DATA MANAGEMENT FOR SUPPORTING DISTRIBUTED VISUALIZATION
Visual exploration of multi-terabyte and petabyte-sized datasets is critical to gain insight in
scientific domains, such as astrophysics, earth sciences and biosciences. When a hurricane is
about to make landfall, scientists involved in disaster management need to visualize, in real-time,
climate models running on geographically distributed clusters, overlay these models with high-
resolution map data, sensor data and traffic information, to make informed decisions regarding
the evacuation of people. The requirements of these visualizations include access to multiple 2D,
3D, 4D and higher dimensional datasets that are geographically distributed. However, the critical
challenges in achieving high performance are the access latencies of storage systems and wide-
area networking, rapid access to large multi-dimensional datasets, and transparent access to both
local and remote data repositories.
Scientists have traditionally copied remote datasets to local repositories in order to visualize
them. However, data replication is no longer a viable solution due to the enormous data-set sizes,
cost of the additional local storage, data consistency issues and real-time requirements. Prior
work enabling visualization for distributed data were limited to accessing a single multi-
dimensional dataset (Gao, 2005 and Ding, 2003), data present in a specific format (Prohaska,
2004) or data present on a specific storage system (Bethel, 2000 and Ding 2003). Additionally,
these solutions were demonstrated on 1Gbps wide-area networks using TCP as the transport
protocol to transfer data (Bethel, 2000, Ding, 2003 and Benyon, 2002). However, TCP fails to
scale to the 10x Gbps wide-area networks, and novel transport protocols are needed to fully
exploit the large available bandwidth. As visualization applications require access to data in
multiple scientific formats present in various filesystems on distributed data repositories, a
cohesive data management scheme that provides rapid and transparent access to datasets
independent of their formats, storage systems, and at the same time fully exploiting the available
bandwidth to provide low latency access is of paramount importance.
Case Study : LambdaRAM, a Distributed Data Cache
LambdaRAM (Vishwanath, 2009) is an approach to address the data management needs of
large distributed data visualization. LambdaRAM is a high-performance, multi-dimensional,
distributed cache that harnesses the memory of cluster nodes in one or more clusters that are
interconnected by ultra-high-speed networks, providing data-intensive scientific visualization
applications with rapid and transparent access to both local and remote data. LambdaRAM
employs novel latency mitigation heuristics based on the access patterns of an application,
including presending, a push-based mechanism, prefetching, a pull-based mechanism, and hybrid
combinations of them, proactively fetching data before an application needs it. It uses high-
performance data transfer protocols, including Celeritas (Vishwanath, 2008b), designed to fully
exploit the large available bandwidth of wide-area optical networks. LambdaRAM employs
multi-dimensional distributed data structures, including multi-grids and octrees, to manage multi-
dimensional datasets. This enables support for 2-D, volume, time-varying volume, and parallel
coordinate based visualization. LambdaRAM efficiently distributes and manages the data across
the nodes of the clusters. The data distribution in LambdaRAM is configurable and can easily be
tuned to the needs of an application. Scientific visualization requires access to datasets present in
formats including TIFF, HDF4, NetCDF, raw and binary data. Additionally, datasets could be
located on local storage, parallel filesystems or distributed filesystems. LambdaRAM has an
extensible design that aids in the design of plug-ins to interface visualization applications with
multiple scientific data formats and filesystems. Additionally, as a scientific dataset is typically
composed of multiple files, LambdaRAM presents an intuitive API to applications to access and
manipulate this dataset as without having to worry about data management. The API allows
visualization applications to easily subsample, subset and interact with the data. LambdaRAM
can manage multiple-datasets simultaneously. This is critical for visualization applications using
multi-resolution techniques wherein each resolution is managed as a dataset in LambdaRAM.
LambdaRAM provides memory quality of service guarantees necessary in interactive data
visualization to ensure that the low-resolution data are always cached in memory. Additionally,
As we scale towards petascale and future exascale systems, software testing is no longer
sufficient to guarantee reliable performance in mission critical environments. Formal verification
techniques have been applied to prove the safety and progress properties of LambdaRAM
(Vishwanath, 2008a). This helped identify a deadlock condition in the memory management of
LambdaRAM.
Figure 4 depicts a parallel visualization application using LambdaRAM to access the 600TB
multi-dimensional national aeronautics and space administration’s (NASA) modern era
retrospective analysis for research and application (MERRA) data distributed on a local parallel
filesystem. In this case, LambdaRAM harnesses the memory of the nodes of the cluster.
LambdaRAM can harness the memory of the nodes of multiple clusters interconnected by ultra-
high-speed networks to provide low-latency access to remote data. Figure 5 depicts a parallel data
visualization application running at NASA Ames using LambdaRAM to rapidly and transparently
access the MERRA data repository at NASA Goddard over 10 Gbps high-speed networks.
LambdaRAM helps data visualization scientists spend their efforts on the visualization algorithms
and facilitate scientific discovery instead of spending their efforts on data management issues
including subsampling and subsetting the dataset, memory management and copying data
between sites. LambdaRAM enables time-critical, high-performance data collaboration over both
the local and wide-area for data-intensive applications. Its extensible design enables high-
performance highly productive scientific visualization of large distributed data.
Distributed Data Access and Visualization of NASA’S MERRA Data for
Climate Analysis using LambdaRAM over Wide-Area Networks
The MERRA time period covers the modern era of remotely sensed data for the entire earth,
from 1979 through the present and is key in understanding climate variability The data are
published in the HDF4 format and consist of multiple 2D, 3D and 4D variables. The dataset size
is approximately 600TB and is stored at NASA Goddard. Given the size of the dataset,
replicating it at other sites incurs a heavy cost. Additionally, given the researchers’ need to
modify the dataset, replication could lead to data consistency issues. NASA would like to enable
their researchers and collaborators at various sites to seamlessly access the data located at
Goddard and use it in their weather analysis and visualizations. The ability to rapidly stride over
the multidimensional MERRA data using LambdaRAM would enable earth scientists to make
more informed weather predictions.
The performance of a climate analysis and visualization application accessing NASA’s
MERRA data using LambdaRAM over wide-are networks was evaluated. For this experiment,
MERRA data for 1979 was stored on an ultra-fast storage system at Chicago. The dataset is
approximately 1 Terabyte (TB) in size, and consists of 1440 data files in HDF4 format. The
storage system consists of a dual-processor dual-core AMD Opteron system with 4GB of RAM,
2TB of storage and a 10GE PCI-e based network interface card (NIC). The 2TB storage was
configured using eight 300 GB SATA – II disks on a PCI-e based 8-port 3ware RAID controller
using RAID 0. The data analysis and visualization was performed on a node at NASA Goddard.
This node consists of a dual-core dual-processor AMD Opteron with 4GB RAM and a 10GE PCI-
X based Intel NIC. A dedicated 10 Gbps network was provisioned for the experiment to
interconnect the two nodes. The network testbed is depicted in Figure 6 and consisted of the
CAVEWave network between Chicago and McLean, Virginia, and the DRAGON network
between McLean, VA and NASA Goddard. The total round trip latency for this network was
around 18ms.
Figure 7 depicts the times to access, compute and visualize the average ozone thickness of
MERRA data for 1979. From the figure, one can see as much as a 50% improvement in
performance with a single LambdaRAM server and a 100% improvement using two
LambdaRAM servers as compared to a typical ultra-fast storage system. On local storage,
accessing multidimensional data involves accessing multiple noncontiguous regions on disk
which incurs significant access overhead and leads to performance degradation. Improved
performance was observed using presending over prefetching as it does not incur the one-way
request latency in prefetching. Presending yielded 8.3% performance improvement over
prefetching using a single LambdaRAM server and 21.6% using two LambdaRAM servers. Also
observed is a 27% performance improvement by using Celeritas, an application-layer UDP-based
reliable transport protocol, in comparison to TCP over the wide-area network. Celeritas sustained
an average throughput of 3.5Gbps in the case of two LambdaRAM servers and 2.1Gbps using a
single LambdaRAM server and played a key role in achieving high-performance over the Wide-
area networks.
Particularly noteworthy was that it was faster to stride, compute and visualize the average
ozone thickness of the remote MERRA data using LambdaRAM than using a local ultra-fast
storage node. Additionally, parallel data analysis applications have demonstrated up-to a five-fold
performance improvement by using LambdaRAM to access multidimensional data on a PVFS2
parallel filesystem in comparison to using MPI-IO to directly access the data. This is due to the
multidimensional data management and latency mitigation heuristics in LambdaRAM. At
Supercomputing 2004 and 2005, an interactive visualization of large 2-D imagery using
LambdaRAM to access data over transatlantic networks, was demonstrated.
LambdaRAM’s model of aggressively using network bandwidth to overcome latency will be
crucial to ensuring the scalability of distributed computing and visualization applications whose
greatest bottleneck will be in getting the data to the processing units.
DATA RENDERING FOR SUPPORTING DISTRIBUTED VISUALIZATION
So far, we have seen a number of complete examples demonstrating different instantiations of
the visualization pipeline, both collocated and distributed, interactive and batch. The prior section
emphasized the early stages of the pipeline, managing and staging data. We now move further
down the visualization pipeline to the rendering step, again with a variety of architectures and
with an emphasis on large, distributed, multivariate data. Before presenting more examples,
though, we begin with a brief summary of basic methods and algorithms performed serially on a
single machine. Next, we examine how these methods must be modified when datasets grow in
size and complexity and architectures grow in parallelism and heterogeneity. We conclude with
two case studies in parallel volume rendering – one in hardware on a rendering cluster another in
software on a parallel supercomputer.
Rendering Techniques
Rendering methods for scientific visualization are organized according to the type of data
values and grid containing those values. Data values constitute a field over the grid, F(G), where
G ⊂ ℜ n . Most often, G is two-dimensional (n = 2) and F = F(x, y), or three-dimensional (n = 3)
and F = F(x, y, z). In the case of time-varying data, time can be treated as a fourth dimension,
where F = F(x, y, z, t). Alternately, a time varying dataset can be represented as a sequence of
steady state 3D datasets such that F = { F 0 (x, y, z), F 1 (x, y, z) … F t (x, y, z) }, where time is in
discrete steps from 0 to t. Each F t is a separate time step that may be stored in a separate file on
disk. G can be structured or unstructured, and F can consist of multiple scalar, vector, or tensor
values at each grid point. Different rendering techniques are used depending on various
combinations of F and G.
Field Types
Scalar Fields: A scalar field contains single-valued data at each grid cell, for example,
temperature. Multiple scalar values can exist at each location, for example, temperature, pressure,
and density. Scalar fields can be rendered by selecting a particular value (isovalue) and displaying
all of the locations where the field has that isovalue. The union of these locations forms a surface
(isosurface), and surface rendering techniques such as triangle meshes are used to render
isosurfaces (Lorensen, 1987; Gao, 2001). See Figure 8.
Rather than extracting an isosurface, the entire volume (assuming n = 3 or n = 4) can be
visualized directly using volume rendering techniques (Levoy, 1998; Ma, 2000). Volume
rendering creates an image of the entire 3D space using varying color and transparency. This
allows partial or complete visibility of the inside regions of the volume. Colors and opacities are
mapped the data values via a user-defined transfer function. See Figure 8.
Vector Fields: A vector field contains vector-valued data at each grid cell, for example,
velocity (v x , v y , v z ). The collection of velocity vectors represent the flow of particles in space and
in time. When the flow is steady-state, a set of seed points can be numerically integrated over the
velocity field to construct a set of streamlines. When the flow is time-varying, we call the
resulting locus pathlines. An alternative construction for time-varying flows is streaklines,
constructed by regenerating a constant set of seed points over time.
For rendering, each point in a streamline, pathline, or streakline can be connected to its
neighbor with a line or a higher order curve. To reduce aliasing and increase visibility, these lines
can be rendered as tubes or cylinders, or even semitransparent surfaces. To further improve
clarity, lighting and texturing can be applied (Johnson, 2004; vonFunck, 2008). See Figure 8.
There are direct methods of rendering a vector field without tracing particles through it,
including line integral convolution (LIC), glyphs, and topological methods. LIC renders the result
of applying an anisotropic convolution filter directly over the field (Cabral, 1993). Glyphs are
representations such as arrows, cones, etc., that are placed within the field and oriented tangent to
it (de Leeuw, 1993). Topological methods do not visualize the field itself, but rather the topology
of the flow by evaluating ridges, saddles, and critical points in the field (Gyulassy, 2008).
Tensor Fields: Tensors are matrices; for example 3x3 tensors are common in materials science
and medical imaging. Tensors that arise in science can be characterized by their eigenvalues and
eigenvectors. To render such a tensor field, a set of ellipsoidal glyphs is commonly used, where
the principle directions of the ellipsoid correspond to the eigenvectors, and the major and minor
ellipsoid radii reflect the maximum and minimum eigenvalues. Topological methods can also be
applied to tensor fields (Tricoche, 2008).
Grid Types
Structured Grids: A structured grid follows a regular pattern such that the coordinates of
individual grid cells are not stored; positional and topological information is derived from a few
parameters such as minimum, maximum, and spacing. Perhaps the most familiar example is a
Cartesian grid, but others such as polar, cylindrical, spherical are common. Rectilinear and
curvilinear grids are generalizations of these, where spacing varies across the extent of the grid
(Schroeder, 2006).
A special case of variable-spacing grid is the adaptive mesh refinement (AMR) grid (Ma,
1999; Weber, 2001a; Weber, 2001b). In an AMR grid, block shaped regions corresponding to
various data frequencies have different spatial resolution. These resolutions differ by a constant
factor, for example, half-spacing in each dimension. Many combinations and varieties of
structured grids exist, depending on the data requirements of the underlying simulation.
Structured grids are attractive because the regularity of grid locations saves storage space and
speeds access to data values. Figure 9 shows examples of structured grids.
Unstructured Grids: Some computational problems exist over a data domain that cannot be
effectively modeled by a structured grid without wasting a great deal of resources. Consider
simulating the airflow over an airplane. Most of the grid cells in a structured grid enveloping such
an irregular shape would represent air. This is appropriate if the objective is to understand the
airflow patterns surrounding the airplane. However, if the objective is to model the stress on the
fuselage, then most of the grid cells would be wasted.
In such cases, each grid cell of interest is listed in an indexed structure. The grid coordinates
of each point must be stored because they cannot be determined algorithmically. Also,
topological information must be stored. In particular, links from each cell to the neighboring cells
are necessary in order to traverse from one cell to an adjacent one. In 2D unstructured grids, cells
are usually triangular or rectangular in shape. The 3D analogy to these cell shapes are tetrahedral
and hexahedral grid cells (Nielson, 1997). See Figure 9.
Data Intensive Rendering
In computational science, simulations run for hours, days, weeks, and sometimes months at a
time across thousands or tens of thousands of compute nodes. These runs produce hundreds of
terabytes, even petabytes of results. Often these datasets are time varying and multivariate. The
scope of modern datasets poses new challenges for rendering; we will use the term data intensive
rendering when discussing approaches to meet these challenges, but we emphasize that this is not
a special case anymore. Eventually most if not all rendering of scientific computations will be
data intensive (Mount, 2004; Ross, 2008).
Data intensive rendering can affect the visualization pipeline; in particular, overall
performance depends on the successful use of parallelism and heterogeneity in the visualization
workflow. The computational performance of individual CPUs and GPUs is limited by electrical
consumption and heat dissipation, so parallelism is one alternative in order to continue to improve
performance. This is true regardless whether parallelism is inter-core with shared memory or
inter-node with distributed memory. Heterogeneity occurs when different architectures must
cooperate in parallel. Combining today’s multicore chips in parallel induces different types of
parallelism in the same rendering algorithm. The presence of multiple GPUs within the same
system is another example. Ultimately, we must use all of the architectural components in
parallel, not just the homogeneous ones, in order to raise the performance level of data intensive
rendering.
Most strategies used to parallelize the rendering process follow a similar form: data are first
decomposed and distributed to processing units for rendering after which the output of each task
is combined to form the final output image. Parallelization strategies are commonly categorized
based on how data are decomposed across processing units. Parallel direct volume rendering
techniques are usually categorized as either object-order or image-order techniques.
Parallel object-order techniques assign a processing unit to a section of data regardless of
where the data appears in the final output image. After each section of data are rendered, a
compositing step based on the theory described by Porter and Duff (Porter, 1984) constructs the
final image. Much effort has been devoted to developing efficient compositing methods including
Hsu’s direct send method (Hsu, 1993), Camahort and Chakravarty’s projection method
(Camahort, 1993), the binary-swap method (Ma, 1994) and the divided-screenwise hierarchical
compositing method (Ino, 2003). Object-order implementations have been developed for
distributed-memory clusters (Elvins, 1992; Gribble, 2004; Müller, 2006), and specialized
hardware has been developed that implements the binary-swap compositing method (Lombeyda,
2001, Muraki, 2003, Frank, 2005).
Parallel image-order rendering techniques take advantage of the fact that a contribution to a
pixel in the output image is independent of other pixels in the image. These techniques break the
output image into disjoint regions and assign a processing unit to render everything in that region.
Image-order implementations have been developed for shared-memory systems (Palmer, 1998) as
well as distributed-memory systems (Bajaj, 2000).
Object-order methods distribute input data across processing units before rendering begins.
Data remains stationary even when the view transformation changes. However, object-order
methods require an additional compositing phase to assemble the final output image. The cost of
this phase grows with the number of processing units. Image-order methods avoid the additional
compositing phase but require redistribution of some portion of the input data when the viewpoint
changes. The optimal choice of parallelization strategy is heavily dependent on the
implementation architecture. We will see examples of both strategies later in the section.
A common theme in data intensive rendering is heterogeneity. Heterogeneity may exist within
a compute node, as in the Roadrunner architecture (Barker, 2008), or it can take the form of a
rendering hierarchy, similar to a memory or storage hierarchy. We can consider three levels in
this hierarchy: supercomputer, graphics cluster, and workstation. Because computing power
decreases and interactivity increases as we move to different levels in the hierarchy, one may
expose different system characteristics at each level by mapping different visualization
algorithms and stages of the algorithm to each machine architecture and thereby optimize the end-
to-end performance.
For example, in a vector flow field visualization, the supercomputer can compute particle
traces and transfer finished particles to a visualization cluster for rendering. There, graphical
effects such as semitransparency and lighting can be added more effectively at the GPU hardware
level than in software at the supercomputer. A subset of the rendered surfaces can be transferred
to the scientist’s workstation for local interaction. This type of hierarchy has yet to be realized at
large scales in practice, but heterogeneity of this type will likely play a role in future data
intensive rendering.
Some rough guidelines for mapping algorithms to architectures are:
• Supercomputer: Algorithms such as ray casting that either have a software equivalent or
were originated in software; large scale data-bound problems where the cost of moving
data is prohibitive; high quality or large image size rendering where rendering time is
long; complex mesh problems where raw rendering time is not the bottleneck;
preprocessing, filtering, transforming, or numerical analysis operations performed prior
to rendering
• Graphics cluster: Algorithms such as marching cubes that originated as triangle
rendering methods; algorithms that rely heavily on triangles or textures for which the
GPU rendering pipeline is optimized; algorithms that benefit from multithread
parallelism that GPUs can offer, data domains that fit in GPU memory, and interactive
applications.
• Graphics workstation: Versions of graphics cluster algorithms operating on smaller data
subdomains; applications with even higher levels of interactivity; and algorithms that
make local approximations to the higher levels of the hierarchy, for example, image-
based rendering (Chen, 1995).
Finally, a common theme in data intensive rendering is that many algorithm design decisions
are governed by the high cost of reading, writing, and transporting data. Overall performance is
usually bound by the cost of data movement, rather than computation. This data movement can
take the form of transporting files across a grid or the internet, reading and writing to and from
storage systems, and communicating between the nodes of a cluster or supercomputer. We will
see the cost of data movement reappear several times in the following examples.
Case study: Parallel Volume Rendering on a Graphics Cluster
As the spatial size of volume data increases due to advances in scientific instrumentation, so
too does the need to visualize such large data. High-resolution displays allow scientists to see
their spatially large data at or nearer its native resolution. Most often, scalable high-resolution
displays are composed of an array of liquid-crystal displays (LCDs) or projectors and a
commodity distributed-memory cluster of computers with accelerated graphics hardware.
Schwarz and others have shown a distributed image-order volume rendering approach for
high-resolution displays that scales as the output resolution of the display and the number of
cluster nodes increases. This approach preprocesses data into a hierarchical structure which is
distributed across the local storage of a distributed-memory cluster. The cluster is equipped with
graphics cards capable of hardware accelerated 3D texture-mapping. Rendering is aborted and
restarted at the lowest level in the hierarchical data structure if user interaction occurs before the
highest-resolution level is completed.
The successful image-order parallelization of the rendering task relies on a data management
scheme that uses a multi-level cache and distributed shared-memory system. The primary responsibility of the multi-level cache is to keep the most recently used data bricks as close to the
graphics hardware as possible. The distributed shared-memory system keeps track of all data
bricks loaded in memory across cluster nodes. Data bricks are transferred between cluster nodes
when required via the cluster’s high-speed backplane. If a data brick is not in memory on one of
the nodes it is loaded from disk. Performance results show that the system scales as output
resolution and cluster size increases.
The Purkinje dataset in Figure 10 is shown on a six panel, twenty-four megapixel tabletop
display. The display is run by a three node cluster where each node is attached to two 2,560 x
1,600 LCDs. Each cluster node has one AMD Athlon 64 FX-60 Dual Core processor, 2 GB of
RAM, and a PCI-E nVidia GeForce 7900 GT graphics card with 256 MB of texture-memory. The
cluster is connected via a 10 Gbps Ethernet backplane with the MTU size set to 9,000 bytes.
The original raw data creates a 2,048 x 4,096 x 128 volume of 16-bit voxels. The real spatial
extent of the data is about 80 µm x 80 µm x 15 µm. Purkinje neurons exist in the cerebellar cortex
and are responsible for fine motor control. The study of Purkinje neurons may lead to treatments
for genetic mutations, such as autism, and neurodegenerative diseases, such as sporadic ataxias.
The rat kidney dataset in Figure 11 is shown on a fifty-five panel, 100 megapixel display. The
display is run by a twenty-eight node cluster where all but one node is attached to two 1,600 x
1,200 LCDs. Each cluster node has two AMD Opteron 246 processors, 4 GB of RAM, 500 GB of
local storage space, and an 8x AGP nVidia Quadro FX 3000 graphics card with 256 MB of
texture-memory. The cluster is connected via a 1 Gbps Ethernet backplane with the MTU size set
to 9,000 bytes.
The original raw data comprises a 32,768 x 32,768 x 128 volume of 24-bit samples. The real
spatial extent of the data is about 8mm x 5mm x 1.5mm. Scientists are particularly interested in
cell neucli which are between 5 µm and 10 µm in length.
Figure 12 shows the average time to render a single frame of the Purkinje neuron dataset.
Average rendering results for the rat kidney dataset are given in Figure 13. These results reflect
the average time taken to render and display all data in all levels of the tree for a given view. The
results show that as the output resolution increases along with the corresponding number of
rendering nodes, the time taken to render each dataset decreases.
Case Study: Parallel Volume Rendering on the IBM Blue Gene
In contrast to the previous case study, Peterka et al. (Peterka, 2008) wondered whether and
how parallel supercomputers such as the IBM Blue Gene and Cray XT can operate as parallel
visualization engines. Even though they do not have graphics capability, under certain conditions
of data size, algorithm, number of cores, and output display resolution, we have found that a
software rendering solution running in parallel on a supercomputing platform can actually be an
effective alternative to hardware-accelerated graphics cluster approaches that are the de facto
rendering method today.
Running visualization algorithms on a supercomputer also opens the door to in situ
visualization, where visualizations are rendered simultaneously with a running simulation (Ma,
2007; Tu, 2006; Yu, 2004). This permits scientists to “see into” their computations as they
proceed, even to steer them dynamically. There are many problems yet to be solved before in situ
visualization becomes a commonly used technique, but the ability to execute parallel
visualizations on the same architecture as simulations is one step in that direction.
Peterka and others are researching parallel visualization algorithms on leadership-class
systems, using the IBM Blue Gene/P system as a test environment. To date, a parallel volume
rendering algorithm has been successfully applied to over 32,000 BG/P compute cores. Datasets
of up to 335 billion elements were successfully volume rendered on BG/P to produce images up
to 16 million pixels.
Figure 14 shows an output image from volume rendering the entropy within a core-collapse
supernova using this approach. Through computational simulation, Blondin and Mezzacappa
(Blondin, 2003) are studying the physics of the shock wave that forms during the death of some
of the largest stars.
Figure 15 shows the end-to-end performance and scalability for a range of data, image, and
system sizes. Because the data are time-varying, each time step must be read from storage prior to
rendering. The size of a single time step can be gigabytes or even hundreds of gigabytes, and
Figure 15 shows that most of the total frame time (time to render one time step from start to
finish) is spent on I/O, reading the data from storage. Various optimizations, as already discussed
earlier, can help to hide the high cost of I/O (Peterka, 2008).
Excluding I/O, Figure 16 also shows that compositing is the next most expensive operation.
Compositing is a many-to-many communication step among all of the compute cores to exchange
partial results and merge these into a final image. Like I/O, this is a data movement operation.
This example demonstrates that large scale visualization performance is dominated by the cost of
data transfers, both from storage and across the interconnect between compute cores.
Returning to the original question, whether parallel volume rendering can be done within such
a supercomputing architecture, the answer is yes. The strengths of Argonne’s IBM Blue Gene are
a large capacity parallel storage system, low-latency high-bandwidth interconnection network,
and a large number of relatively low power compute nodes. Since the visualization algorithm (in
this case volume rendering) is bound by data movement and not rendering speed, the algorithm
properties and machine characteristics align. Data movement between supercomputer and
separate visualization cluster is eliminated and future in situ visualization is enabled by
performing visualization directly on the supercomputer.
ADVANCED DISPLAYS FOR SUPPORTING DISTRIBUTED VISUALIZATION
Traditionally data rendering and display occur together- i.e. the same computer that does the
rendering, displays the result. In distributed visualization, especially as necessitated by the sheer
massiveness of the data that needs to be rendered, it is ultimately more economical to separate
rendering from display. This Thin Client model is actually widely used in practice today – for
example every time we query Google, the “heavy lifting” of generating the result or visualization
is performed by massive compute clusters at Google, and the viewer simply sees the result on
their desktop computer, laptop or smartphone. This model is attractive because the large-scale
(and therefore expensive) computing infrastructure that supports such services can be shared
amongst large populations of users, and can be located near reliable and renewable power
sources.
As the types of problems we tackle grow in scale and complexity our notion of what
constitutes a Thin Client must change. The new Thin Client is a large ultra-high-resolution
display interfaced with an ultra-high-speed network. Increased screen real-estate and resolution
enables users to display and juxtapose more data simultaneously and thereby enhances the users’
ability to derive insight from data.
Until the mid-90s image creation for large displays (often called Power Walls) primarily used
Cathode Ray Tube-based projectors. The main advantage of CRT projectors is that they maintain
good brightness to 10,000 hours, are capable of generating very high resolution images (up to
1920x1200), and are able to refresh at high rates. In the mid to late 90s, LCD and DLP projectors
were rapidly declining in cost as well as physical size. This was driven largely by the business
graphics market which required low cost projectors for projecting presentation slides in board
rooms. Tilings of LCD or DLP projectors were used to create larger walls (Funkhouser, 2000).
While these projectors were relatively inexpensive it was difficult to align the geometry and color
of the screens. Techniques have been developed for automatic alignment through the use of
computer-vision. As a result commercial turnkey solutions are now available- but at a relatively
high cost. Today DLP projectors such as Sony’s SXRD are capable of 8-megapixel resolution
(4096x2160) and are largely used in movie theaters. In prototype form is NHK’s Super Hi-Vision
project (Kanazawa, 2003) which is capable of 7680x4320.
Currently the most economical way to build scalable ultra-high-resolution displays is by using
tilings of LCD panels rather than projectors due to the LCD’s long life, low power consumption,
high resolution and low cost. Other benefits include the fact that LCD panels are quite well color
calibrated and apart from the physical alignment of the panels during tiling, require no further
alignment, unlike projectors. Tilings of these have been used to create extremely high resolution
walls such as the 100 Megapixel LambdaVision display (Renambot, 2005). The main drawback
of LCD panels is that they have mullions that prevent them from producing truly seamless
displays. Mullions do not adversely impact the viewing of an image if they are taken into account
in the rendering of the image. However mullions do make the reading of text difficult when it
either occludes an entire sentence or words. Driving these tilings of displays requires a cluster of
computers, often each computer drives anywhere between 1 and 4 displays, and a master
computer coordinates the entire cluster. By building these tiled displays using small low cost
computers equipped with gigabit networking- such as Mac Minis we can leverage the Thin Client
paradigm and utilize remote shared computing infrastructure to generate and stream the
visualizations to the tiled display.
The major challenges in supporting tiled display environments then is how to scale graphics
rendering algorithms to be able to handle the exponentially growing data volumes that are
accumulated in scientific research, and how to best enable users to interact with information and
visualizations in these environments.
Middleware for Scalable Displays
The traditional model for scalable rendering has assumed that faster graphics cards will be
more than capable of supporting rendering to keep pace with the exponential growth of data size.
Middleware such as WireGL, and its enhanced version, Chromium manages a high resolution
scene by distributing polygons only to the computers responsible for rendering the particular
viewport into the overall scene (Humphreys, 2001). Termed “sort-first rendering” the advantage
of this scheme was that if an image filled a significant portion of the tiled display much of the
geometry would be evenly distributed across all the computers. However if the image fell on only
a small portion of the wall a load imbalance would result and overall rendering rate would
plummet.
An alternative approach replicates all the data across all the computers and simply uses raw
graphics power to process the data to generate the images for the individual viewports of the
display tiles. Used in middleware such as Conduit (by VRCO) and the Cross-Platform Cluster
Graphic Library (CGLX), the advantage of this approach is that many existing OpenGL
applications can be easily ported without modification of the code. The primary disadvantage is
that the amount of data that can be rendered is limited by the capabilities of the individual
graphics card.
Evolving out of the Thin Client paradigm, a third and most scalable approach delegates the
rendering to a remote cluster of computers and instead treats the tiled display simply as a large
frame buffer connected via a high speed network. This approach was first pioneered in SAGE
(the Scalable Adaptive Graphics Environment) (Jeong, 2006). The scheme has greater scalability
than previous approaches because it allows multiple cluster computers that are potentially
rendering different and very large data sets, to work concurrently and stream visualizations to be
displayed on the wall as individual windows as if on an enormous desktop. This allows users to
arbitrarily position and resize these windows on the wall and therefore enabling them to work
with multiple visualizations simultaneously- in effect creating a Cyber-Mashup (Leigh, 2007).
When the pixels of a visualization are streamed in parallel from a cluster of rendering nodes to
SAGE, they are intelligently routed to the correct display cluster nodes that are managing the
respective portions of the tiled display. As a window moves from one portion of the tiled display
to another, SAGE correctly re-routes the pixels in much the same way a network router routes
network packets. This approach therefore enables users to run multiple visualizations on
distributed rendering clusters and display them in a tiled display environment each in its own
separate window that can be independently controlled.
By contrast, the techniques used in WireGL/Chromium and CGLX require that the entire wall
be used to display only one visualization application at a time. This is acceptable for small tiled
displays but becomes impractical for very large display walls, especially those that can in the
future, potentially cover all the walls of a room. The SAGE model also has the advantage that far
less powerful graphics cards and computers can be used to drive the display walls thereby making
it more affordable and cost effective to maintain. These display walls can then connect into global
high performance networks to take advantage of large-scale computing resources that are
deployed at supercomputing centers around the world.
Supporting Distance Collaboration in Scalable Display Environments
Today, remote application sharing and video conferencing using tools such as WebEx, Skype,
and iChat, are making it routine for distributed users to work with each other and to work with
data from their desktop computers. These solutions however do not scale to ultra-high-resolution
display environments, which need to be able to share high-resolution visualizations and high-
definition video and audio in real-time to facilitate group-to-group collaboration.
Visualcasting is a novel image broadcasting scheme that extends SAGE to support distance
collaboration with multiple end-points without the need for traditional router-based support for
multicasting which have always been difficult to deploy over wide-area networks (Renambot,
2009). Visualcasting is a scalable real-time image replication service for multiple high-resolution
tiled displays. A visualization application streamed to multiple Visualcasting end-points (tiled
displays) can have different window layouts on each tiled display. The window operations
(moving or resizing) on each tiled display can occur independently. As windows on the tiled
display are resized or repositioned, SAGE performs the non-trivial task of reconfiguring the
multiple streams from the rendering source to the PC nodes that drive potentially differing tiled
displays.
Visualcasting is realized via the SAGE Bridge- a high-speed intelligent bridging system that
duplicates and splits pixel streams received from rendering clusters for each end-point. This
allows each rendering node to stream whole images without having to be cognizant of the
window layouts and tiled display configurations of multiple end-points. A single SAGE Bridge
node is built from a high-performance PC equipped with 10-gigabit network interfaces. Multiples
of these can be ganged together to facilitate the sharing of even higher resolution content or more
varieties of content. For example one bridge could be dedicated to high-definition audio/video
conferencing while another could be dedicated to sharing visualizations.
SAGE also supports audio streaming to facilitate distance collaboration. The design goal has
always been to treat sound data in SAGE in a similar fashion to pixel data potentially allowing for
sounds to be spatially collocated with the displays that show the visualizations. Sound buffers are
captured at various sources (microphones, high-definition video capture card, audio files, or
application-provided buffers), then streamed in an uncompressed form to minimize latency, and
finally sent to a sound card for playback.
Figure 18 and Figure 19 show a Visualcasting trial conducted over an international network
testbed called the Global Lambda Integrated Facility. In the trial ultra-high-resolution tiled
displays at the SC08 conference in Austin, Texas (Figure 18), Electronic Visualization
Laboratory at the University of Illinois at Chicago (Illinois), the School of Information at the
University of Michigan, and Masaryk University (Czech Republic) (Figure 19) were linked to
Visualcasting servers in Chicago so that they are all able to share a 4K (4096x2048) pixel
visualization stream as well as communicate over high-definition video conferencing.
CONCLUSION
The World Wide Web has been a transformative technology that has afforded its users the
ability to publish and retrieve an unprecedented amount of data. Making data massively available
and providing facilities for combining them stimulates the production of new insight. A crucial
component to the derivation of insight is visualization- the production of a meaningful visual
representation of data.
As data size continues to grow exponentially, the long-established technique of first
downloading data to a desktop computer and then creating a plot on a spreadsheet has given way
to distributed models of data retrieval, visualization and analysis. The majority of techniques used
on the Web today have their roots in the field of High-Performance Computing (HPC). For
almost two decades, the HPC community has developed innovative approaches for deriving
insight from large-scale distributed data.
Large-scale data visualization is conducted via a pipeline consisting of data retrieval, filtering,
rendering, and finally display. The motivation to collocate or distribute the steps of the pipeline is
driven largely by the economics of meeting user demand and managing scalability. Traditionally
when data was small, all the steps of the pipeline could reside on the same computer.
Visualization was interactive and users could adjust the visualizations in real-time. As data sizes
grew, visualization on single computers had to expand to utilize parallel processing on multiple
computers in order to keep pace. However, parallel computing was still largely collocated
computing because high speed wide-area networks were scarce. Furthermore, parallel computers
often worked in batch mode, and therefore the results of parallel visualization algorithms
consisted mainly of animated movies rather than interactive graphics. Viewing the visualization
from a different point of view was therefore not possible without time consuming re-computation.
As data volumes continued to grow, the economics of providing data management and data
processing services necessitated greater wide-area distribution of resources. Serendipitously, this
led to an explosion of new applications beyond scientific computing that includes e-commerce
and even video gaming.
The key challenge in data visualization today is how to provide users with the interactive
experience that they have grown accustomed to on their desktop computers, while offering the
scalability to handle the exponential explosion of data. Experiments show that large-scale
visualization is an Input/Output-bound problem. That is, the rate at which visualizations can be
generated is bound by the rate at which data can be sent through the visualization pipeline. If one
cannot move the data through the pipeline fast enough then one cannot process the data fast
enough. When interactivity is required, data access latency becomes another major problem-
especially when the data is distributed over wide-area networks. However as high-speed networks
such as the National LambdaRail and Internet-2 grow in capacity, the limitation of bandwidth
between distributed resources diminishes. Furthermore, through the aggressive use of bandwidth
to presend and prefetch data, it becomes possible to significantly reduce data access times to the
point where, surprisingly, it can be faster to access data from a remote computer than from one’s
local disk drive. Some researchers would argue that there will never be enough wide-area
bandwidth available to keep pace with both the growth of data and the multi-core and many-core
systems that are needed to process the data. In this case, researchers are examining the notion of
in situ rendering, where the parallelized visualization and computational algorithms are collocated
so that rendering the data can be done during the generation of the data. In in situ visualization,
the computation and visualization could be either collocated on the same supercomputer or run on
separate specialized resources that are collocated at the same site. In the first case, the data
transfer between computers is reduced though at a cost of sharing compute resources of the
supercomputer for visualization. In the latter approach, a visualization cluster, typically a GPU-
based cluster, needs to be interconnected to the supercomputer via high-speed local network
interconnects.
The notion of in situ visualization however is not a silver bullet. The rate at which a frame of
visualization is generated is bound by the rate of a simulation timestep- which can be on the order
of seconds to even hours. Therefore, in situ visualization does not lend itself to interactive
visualization. In situ visualization’s greatest value is in allowing the user to catch a glimpse of
how the computation is evolving in order to determine whether it is worth allowing it to continue
for days to weeks. This approach can enable computational steering where the user would directly
modify the parameters of the computation. Then once the computation is complete, techniques for
low-latency real-time distributed visualization can be brought to bear to provide users with a
means to interactively explore the results.
ACKNOWLEDGMENTS
This publication is based on work supported in part by Award CNS-0935919 and 0959053, made
by the National Science Foundation, and U.S. Dept. of Energy under Contract DE-AC02-
06CH11357.
REFERENCES
Bajaj C., Ihm I., Park S. & Song D. (2000). Compression-based ray casting of very large volume
data in distributed environments. Proceedings of the Fourth International Conference on High-
Performance Computing in the Asia-Pacific Region (HPC00),2, 720–725.