Prof. Dr.-Ing. Stephan Olbrich -- Direktor --
Regionales Rechenzentrum der Universität Hamburg (RRZ) /
Scientific Visualization & Parallel Processing (VPP)

Scalable In-Situ Data Extraction and Distributed Visualization

In the last few years, the data analysis and visualization aspect has
dramatically gained in importance, since this part of the complete
process chain is much more difficult to scale than the numerical cores
of simulation models. 3D presentation of results of scientific computing?
especially taking advantage of highly interactive virtual reality
environments ? has become feasible using low-cost equipment such as 3D
monitors or TV sets and advanced 3D graphics cards, where the
development was driven from the consumer market. In computational fluid
dynamics typically 3D grids consisting of up to 10^11 data points on
4000 cores can be simulated, which results in a non-stationary scenario
(~10^4 time steps) in ~10 Petabyte raw result data. Since such an amount
of data cannot be transferred or stored or explored using traditional
approaches of separate post-processing, one topic of world-wide research
is the development of tools to integrate data extraction in the
simulation software, so-called ?in-situ data extraction?, and to take
advantage of distributed systems for remote visualization.

We have developed a visualization middleware, which implements parallel
in-situ data extraction by providing a programming library in order to
minimize the sequential bottlenecks by parallelization of visualization
mapping methods and to reduce the data volume by storing polygons and
lines instead of raw data. Supporting synchronous, on-demand 3D
presentation and interaction scenarios under bandwidth and rendering
performance constraints, and nevertheless limiting the frame update time
to get interactive rates, requires flexible and efficient reduction and
post-filtering techniques. For this purpose, our data extraction library
supports MPI-based computing environments and encapsulates a parallel
implementation of vertex cluster based simplified isosurfaces, and
parallel extraction of property-enhanced pathlines. These pathlines can
be interactively post-filtered as part of a specialized, so-called ?3D
streaming server?, which combines storage, filtering, and play-out of
sequences of 3D scenes as a 3D movie, which can be navigated in
real-time.