AMR meshing of a circular region.

For both experimentalists and computational scientists, doing new science sometimes means building your own equipment. I have been one of the developers of the FLASH code, a multi-physics, highly parallel, astrophysical fluids code that has won a Gordon Bell prize, and have published work on techniques involving improving the efficiency in AMR codes and developing techniques for the smooth (transient-free) evolution of near-hydrostatic atmospheres using Godunov-type codes. I have also been involved with projects rigorously testing the accuracy of astrophysical codes, by verifying their solutions (ensuring that the correct solution to the PDEs is found) and validating them (comparing them directly to experiment).

But doing the next big experiment always requires adding one more tool. In parallel with computational projects using tools already at my disposal, over the next five years I plan to develop ‘all-speed’ hydrodynamic solvers (already in use in other disciplines) for use in publicly available codes, to make accessible regimes unavailable to the current generation of astrophysical solvers. In testing such a solver, collaboration with experimentalists will be essential, and can generate interesting science in its own right.

[ Past | Present | Future | Papers | Talks ]

Scaling results for FLASH code up to 2k processors

My past work on computational techniques for astrophysics has focused on:

1. Efficient large-scale parallelization of astrophysical codes;
2. New methods for accurate hydrodynamical simulations;
3. and Validation and Verification of those simulations, by comparing to semi-analytical results (verification) or laboratory experiments (validation)

My work with the FLASH code included improving performance of the already robust and efficient PARAMESH AMR package and the individual physics packages, helping FLASH win a Gordon Bell award. This is work I intend to continue, as efficient parallel computation is necessary for many computational investigations.

Sometimes looking into efficiency increases leads to counter-intuitive results, such as our results in Dursi, Zingale, et al. 2005 which showed that one common AMR technique, `time subcycling', while introducing a great deal of algorithmic complexity, is often of very little use in reducing CPU time in an AMR calculation, at least in geometrically complex explicit calculations. Time subcycling allows you to use fewer timesteps on coarse regions of the grid; however, except for special cases (including many cosmological simulations), a significant fraction of the grid is filled with the finest meshing, which means by far the majority of zones are at high resolutions. In this case, reducing the amount of work on course zones can have only very modest effect on total compute time. With implicit solvers, however, the answer can depend very sensitively on how and when the implict solves are done.

Modification of the PPM reconstruction to take into account HSE; from Zingale, Dursi, et al (2002)

Although efficiency is important, for simulations to be useful, calculating them quickly isn't enough; they must also be accurate. With Mike Zingale and John ZuHone, I have developed a set of techniques including boundary conditions and modifications to the standard Gudonov-type hydrodynamics solver to increase the accuracy of simulations of near-static stratified atmospheres. We showed that this can make the difference between seeing real physics in the stratified atmosphere, or seeing only numerical transients caused by re-settling.

Testing and ensuring numerical accuracy of an algorithm, however, is only the first level of building confidence in a simulation result. To make sure that simulations accurately reflect reality, contact must continually be made with experimental results. This is as true of established algorithms as they are applied to new problems as it is true of new hydrodynamic methods.

Simulated radiographs from a validation simulation for comparison to a laser experiment; from Calder et al (2002)

I have also been closely involved in the comparision of hydrodynamic simulations to laboratory experiments, and have been involved in the data analysis of one such experiment. This provides not only a way to test simulation codes, but to do new science. An excellent source of astrophysically-relevant laboratory test problems for codes are fluid instabilities. In this case, working with experimentalists not only provides challenging tests for a simulator, but can provide an opportunity to do real, detailed science on instabilities whose complex nonlinear evolution remains an unsolved problem.

Experimental results of the Rayleigh-Taylor instability, for comparison to simulations; from Calder et al (2002)

An easier, but still valuable, test is to do code-code comparisons, to try to understand the uncertainties in numerical results that come from using different numerical methods (or different implementations of the same methods.) I was recently involved in a very large code comparison project aimed both at understanding the nonlinear evolution of the Rayleigh-Taylor instability and understanding the differences in results between commonly-used codes by application to this problem.

Efficient Parallel Computation

Doing large-scale computation, such as the simulations sketched here, often involves parallel computation, which introduces great possibilities but also great complexities. Many of my contributions to the Flash code involved improving the efficiency of the parallel computation.

An important stage in this development is the implementation of efficient elliptic solvers. FLASH, which I intend to use as the framework in which to implement this solver, has a multigrid solver which is widely known to be unnecessarily slow. Over the next months I plan to re-write the multigrid solver to greatly improve its performance. This will improve the performance of the FLASH self-gravity module, but will also be an input into other physics solvers.

Benchmarking

In choosing

Understanding Dissipation in Compressible Hydrocodes

I'm also involved in a code comparison project investingating the decay of compressible (Mach 3) turbulence in a number of codes, using a variety of techinques. While it is often said (for instance) that PPM has very low diffusivity, this `lore' comes from considering simple, smooth flows; the behavior in complex, compressible flows is far from clear. Some of my work with this project involves looking at the effects of turning on and off features with unclear effects in multidimensions, such as PPMs contact steepening.

Hydrodynamic Solvers

Many astrophysical problems involve long periods of relatively quiet, secular evolution followed by a rapid stage of high-speed evolution. Generally, the current generation of multidimensional general-purpose astrophysical hydrodynamic solvers are quite capable at high-speed flows, but timestep restrictions render low speed evolutionary stages inaccessible. Some of my previous work has modestly extended the range of applicability of these methods by eliminating transients and improving accuracy, but many problems remain inaccessible.

Other disciplines (such as within the combustion research community) have problems with similar natures, and they have developed hydrodynamic solvers capable of dealing with either low-speed flows or even switching smoothly between low- and high-speed flows. In some cases, these methods can be directly used for astrophysical problems; in other cases, fairly significant generalizations must be made because of the highly nonlinear equations of state that occur.

Although there are a variety of methods, they fall largely into two sets of techniques; projection methods and dual-timestepping, representing extending different sorts of techniques into this regime. Projection methods, in particular, use methods similar to those used in other sorts of astrophysical codes, such as for solving for self-gravity, or implicit flux-limited diffusion. It is this class of solvers, then, that I propose to implement and eventually make publicly available.

Validation and Verification

As simulations become more complex, and more important in the understanding of astrophysical phenomenon, it becomes increasingly vital to have a verification and validation strategy for testing a new method. Slowly, more sophisticated approaches are being used in the astrophysical community for verification (ensuring that the model equation are being solved correctly), including the method of manufactured solutions. Validation, however, requires comparison with experimental results.

Low-speed flows in particular lend themselves to comparing simulations with table-top hydrodynamics experiments. For the comparison to be evidence for trusting the astrophysical aspects of the code, the problem being investigated must be a significant part of the target astrophysical problem, but still be simple enough to make a detailed understanding of the results possible. To get experimentalists interested in collaborating, the problem being studied must be interesting in and of itself. These considerations lead naturally to the examination of fluid instabilities, which are important in many astrophysical situations, and generally remain quite poorly understood in the fully nonlinear regime. By pursuing collaborations with experimental groups for investigations of fluid instabilities, one can both test a new solver and do real science with astrophysical application.

Papers

A. C. Calder, L J. Dursi, B. Fryxell, T. Plewa, V. G. Wiers, T. Dupont, H. F. Robey, R. P. Drake, B. A. Remmington, G. Dimonte, J. Hayes, J. M. Stone, P. M. Ricker, F. X. Timmes, M. Zingale, and K. Olson. Issues with Validating an Astrophysical Simulation Code. CiSE, 6(5):10--20, September 2004.

Guy Dimonte, et al.. A comparitive study of the turbulent Rayleigh-Taylor instability using high-resolution three-dimensional numerical simulations: The Alpha-Group collaboration. Physics of Fluids, 16(5):1668-1693, May 2004.

M. Zingale, L. J. Dursi, J. ZuHone, A. C. Calder, B. Fryxell, T. Plewa, J. W. Truran, A. Caceres, K. Olson, P. Ricker, K. Riley, R. Rosner, A. Siegel, F. X. Timmes, and N. Vladimirova. Mapping Initial Hydrostatic Models in Godunov Codes. ApJSS , 143(2):539-566, December 2002.

A. C. Calder, B. Fryxell, T. Plewa, R. Rosner, L. J. Dursi, V. G. Weirs, T. Dupont, H. F. Robey, J. O. Kane, B. A. Remington, R. P. Drake, G. Dimonte, M. Zingale, F. X. Timmes, K. Olson, P. Ricker, P. MacNeice, and H. M. Tufo. On Validating an Astrophysical Simulation Code. ApJSS , 143:201-229, November 2002.

R Rosner, A. Calder, L. J. Dursi, B. Fryxell, D. Q. Lamb, J. C. Niemeyer, K. Olson, P. Ricker, F. X. Timmes, J. W. Truran, H. Tufo, Y.-N. Young, M. Zingale, E. Lusk, and R. Stevens. Flash Code: Studying Astrophysical Thermonuclear Flashes. Computing in Science and Engineering, 2, 2000.

L. J. Dursi, M. Zingale. Efficiency Gains from Time Refinement on AMR Meshes and Explicit Timestepping. In Adaptive Mesh Refinement — Theory and Applications, Springer-Verlag, 2005.

Talks

• Simulating Astrophysical Combustion with the FLASH code
  CAIMS/MITACS Joint Annual Conference, June 2006
  [PDF] [OpenDocument]
  An invited talk to the Scientific Computing session of the annual Canadian applied mathematics conference discussing simulating combustion in an astrophysical context, and the computational choices that this regime suggests.
• Local Ignition in Carbon-Oxygen White Dwarfs: One Zone Ignition, and Spherical Shock Ignition of Detonations
  208^th AAS meeting, June 2006
  [PDF] [OpenDocument]
  A presentation describing recent work on the physics of local ignition in Type Ia supernovae.
• Towards Understanding some Astrophysical Flows using Multiscale Approaches with the FLASH code
  HPCS 2006, May 2006
  [PDF] [OpenDocument]
  A discussion of some of our attempts to simulate turbulent astrophysical flows with interesting physics on disparate scales using simple multiscale approaches.
• Lagrangian Methods and SPH
  Stony Brook University, Apr 2006
  [PDF]
  An introductory lecture to a class on grid methods for hydrodynamics on SPH methods.
• Hydrodynamics in the FLASH code
  CITA FLASH workshop, March 2005
  [PDF]
  A partial overview of astrophysical CFD generally, and a discussion of the hydrodynamical methods implemented in the FLASH code in particular.
• High Performance Reactive Fluid Flow Simulations Using Adaptive Mesh Refinement on Thousands of Processors
  Supercomputing 2000, Nov 2000
  [PDF]
  This was the talk I gave (on behalf of the whole FLASH team) for the Gordon Bell Award.