Stanford University HomeContactStanford University Home
Center for Integrated Turbulence Simulations Advanced Simulation and Computing ASC logo
 About CITS Research Simulation Publications Software
Research Projects
RANS
LES
Overview
CDP
Applications
Publications
People
Integration
Merrimac



 CITS Home

LES Group

Overview | CDP: Unstructured LES Code | Applications | Publications | People

Applications

Particle/Mesh Load Balancing

Particle/Mesh load balancing more than doubles the speed of CDP for certain problems. While it is theoretically possible to dynamically repartition and redistribute the problem in memory, rebalancing in CDP is performed infrequently, and has thus been implemented to look like a restart: i.e. a set of grid and solution files are written for a new partition, the code is stopped and then restarted using the new partition.

Background

The partitioning of an unstructured grid is a complex optimization problem that attempts to balance the number of control volumes (and thus work) on each partition while minimizing the "edge-cut" – related to the cost of communication between neighboring partitions. In CDP, the partitioning problem is made more complex because the fuel spray uses Lagrangian particle tracking, and the geometric localization of the spray near the fuel injectors can lead to load imbalance.

particles

To solve this problem, the partitioning used in CDP is based on the dual constraint of balancing the number of control volumes and the number of Lagrangian particles on each partition. This can only be done at the cost of a slight increase in edge-cut, as shown in the following example.

The above figure shows a cut through a Pratt & Whitney injector and the associated spray of atomized fuel droplets. Using the ParMETIS graph partitioning software, a single constraint partition was produced on 96 processors. Details of this single-constraint partition are given in the following figure. The edge cut reported by ParMETIS was 124170.


load load balance

Notice that most of the particles are contained in just a few of the partitions. Because of synchronization points in the time advancement, this means that the majority of the processors will be idle during the particle ODE integration.

Using the multi-constraint capability of ParMETIS, a dual-constraint partition for the same problem can be constructed.


load 2 load balance 2

The dual constraint partition attempts to balance both particles and control volumes on each processor/partition. The edge-cut reported by ParMETIS was 176686, or 42% larger than the single-constraint partition. The complexity (the number of nearest neighbors for each partition) has also increased significantly. Despite these increases, the dual constraint partition is nearly twice as fast per time step than the single-constraint partition. This speedup is expected to be even more significant when more complex spray models are used.



Stanford Seal

Back to top

Copyright © 2004 Center for Integrated Turbulence Simulations, Stanford University. All rights reserved.