LES Group
Overview |
CDP: Unstructured LES Code |
Applications |
Publications |
People
Applications
Particle/Mesh Load Balancing
Particle/Mesh load balancing more than doubles the speed of CDP for certain problems. While it is theoretically possible to dynamically repartition and redistribute the problem in memory, rebalancing in CDP is performed infrequently, and has thus been implemented to look like a restart: i.e. a set of grid and solution files are written for a new partition, the code is stopped and then restarted using the new partition.
Background
The partitioning of an unstructured grid is a complex optimization problem that attempts to balance the number of control volumes (and thus work) on each partition while minimizing the "edge-cut" – related to the cost of communication between neighboring partitions. In CDP, the partitioning problem is made more complex because the fuel spray uses Lagrangian particle tracking, and the geometric localization of the spray near the fuel injectors can lead to load imbalance.
To solve this problem, the partitioning used in CDP is based on the dual constraint of balancing the number of control volumes and the number of Lagrangian particles on each partition. This can only be done at the cost of a slight increase in edge-cut, as shown in the following example.
The above figure shows a cut through a Pratt & Whitney injector and the associated spray of atomized fuel droplets. Using the ParMETIS graph partitioning software, a single constraint partition was produced on 96 processors. Details of this single-constraint partition are given in the following figure. The edge cut reported by ParMETIS was 124170.
Notice that most of the particles are contained in just a few of the partitions. Because of synchronization points in the time advancement, this means that the majority of the processors will be idle during the particle ODE integration.
Using the multi-constraint capability of ParMETIS, a dual-constraint partition for the same problem can be constructed.
The dual constraint partition attempts to balance both particles and control volumes on each processor/partition. The edge-cut reported by ParMETIS was 176686, or 42% larger than the single-constraint partition. The complexity (the number of nearest neighbors for each partition) has also increased significantly. Despite these increases, the dual constraint partition is nearly twice as fast per time step than the single-constraint partition. This speedup is expected to be even more significant when more complex spray models are used.