1. AstroBEAR Overview Road to Parallelization

2. Current Limitations

3. Optimized Communication ● For hyperbolic systems, only boundary cells between grids need to be communicated. If grids in proximity to each other are placed on the same processor, much of this communication can be eliminated. ● The way the time stepping is handled enforces an ordering of grid updates – level 0 cannot take a second step until level 1 has taken two etc... The earliest updates can be done are in the following sequence: 0123 – 3 – 23 – 13 – 23 – 3 – 23 – 3 ● Or as in Simult_AMR 3 – 23 – 13 – 23 – 03 – 23 – 13 – 23 ● To achieve perfect load balancing throughout an update step, each processor would need an equal load on each level independently ● The combination of equal load balancing conflicts with physically disjoint processor regions ● Since the cost for updating a region at any given refinement level is proportional to, moderately low filling ratios are required for the highest level to be as time consuming as the next lower level. (ie 12.5 % in 2D and 6.25% in 3D) – incidentally, once a filling ratio reaches 87.5% (in 2D), you have lost the benefit of AMR and might as well completely refine the lower level... ● One solution is to separate the dependencies between grids near processor boundaries through redundant calculation ● This then frees each processor to perform updates across all levels reducing the need for load balancing.

4. AMR Algorithm ● Each grid needs to do the following: 1) Prolongate the coarse data from it's parent grid (out to rmbc ghost zones) (childmask=parent%childmask-1) 2) Receive better data from old overlapping grids (including ghost regions) (not child mask!) 1) Determine where to place children (while avoiding improper nesting) 2) Communicate location of children to neighbors ghost zones. 3) Create children 4) Calculate and apply fluxes* 5) Wait for children to take two sub-steps 6) Receive coarsened data and/or fluxes from children 7) Synchronize fluxes at boundaries with neighbors 8) Complete conservative update using coarsened data and/or fluxes. 9) Update coarsened data in ghost regions from neighbors (First Time Only) 3) Coarsen data and fixup fluxes to the parent grid. 2x

5. * Since different AMR levels will not necessarily calculate consistent fluxes (or source terms), there needs to be some way of synchronizing finer level data/fluxes with the coarser level to keep the code conservative and consistent. For cells that are refined, it is easiest to simply replace the coarse data with the finer level data. (These cells aren't part of the 'Finest Level Data Representation (FLDR) 'and don't need to undergo a final update for that matter). For coarse cells that are adjacent to refined cells, the fluxes used should be consistent – and the coarser grid should ultimately use the more accurate finer grid's fluxes. Fortunately these boundaries are always located at the edges of the finer grids, so it is relatively straightforward for each refined grid to store its fluxes along its outer boundary. The coarser grids, however, still need a way to correctly apply these finer grid fluxes – and there are two different approaches. 1) Don't apply the coarse fluxes at these FLDR coarse-fine boundaries in the first place – just apply the finer grid fluxes later 2) Apply the coarse fluxes, but store them at the coarse-fine boundaries to be differenced against the finer grid fluxes later. The first approach is the easiest to implement, although some coarse cells in the FLDR will not have valid data until they have received and applied the finer grid fluxes. The second approach updates the entire FLDR (though not entirely conservatively) at first, and then makes small adjustments later after receiving the fixup fluxes. Currently AstroBear uses the first approach. When the FLDR is required before the final coarsening (as in hypre), it uses qfix as an approximately conservative solution.

6. 1) Since Astrobear uses nested grids, most of these coarse-fine boundaries will be at parent-child interfaces. Occasionally, however, a child will share a boundary with its parent – in which case the coarse-fine boundary will be at a child-uncle interface. Currently AstroBear has no solution for this problem - since their is no protocol for children to communicate with their uncles. 2) One solution would be for children to first communicate with their parents, and have parents then communicate with their siblings. If a child shares a boundary with it's parent, then the parent will be storing its own fluxes at the same boundary. If the parent then stores the common-boundary refined fluxes from its child (which it currently does), it could then synchronize the fluxes with those of its sibling. Both the parent and the sibling would need to know that the parent had the refined and therefore correct value for the flux at the boundary. Ghosting childmask, however, allows this to be the case. 3) Having grids on the same level synchronize their fluxes is in general a good idea anyway. Ideally, both grids would calculate the same fluxes at the boundary – but sometimes approximations are made to reduce the number of ghost cells needed resulting in slight differences. And when one grid is refined but not the other, these differences will exist anyways. So if each pair of neighboring grids either averaged, or copied there fixupfluxes at the boundary (depending on childmask) while updating their outermost row of cells, the code would be completely consistent and conservative. 4) This all may seem like a lot of work to keep the hydrodynamic variables completely conservative... But when it comes to maintaining divB across different grids and AMR levels, these small errors can quickly grow. 5) Instead of synchronizing fluxes between neighbors and between children and parents, we need to synchronize emf's. And instead of restricting and prolongating density etc..., we are restricting and prolongating the magnetic fields. The fact that the bfields are now edge/face centered in 2D/3D and that emf's are corner/edge centered makes this a bit trickier. Face centered bfields leads to a double representation in fixed grid that then requires neighboring grids to stay synchronized. In addition edge centered emf's now require synchronization between anywhere from two to four grids neighboring grids

7. SetGhosts & 12 aux fields to the rescue!!! 1) Currently AstroBear stores the 'fluxes' everywhere – which fortunately isn't too memory intensive since you only need 3 'fluxes' per cell to update 3 variables in 3D or 1 'flux' per cell to update 2 variables in 2D (as opposed to the 3 'fluxes' per cell for each variable. 2) Of course since each coarse grid needs to store it's own current coarse fluxes, as well as restricted fluxes received from the grid below, and accumulated fluxes headed to the grid above, you end up needing to store 3 representations of the 'fluxes' – leading to the 12 aux fields in 3D (and 5 aux fields in 2D). 3) Each level synchronizes its emf's along the edge by using an extra call to SetGhost. Where none of the neighboring grids are refined, each grid uses whichever 'flux' is the smallest. Ideally, an average would be more accurate, but since these edges can be shared by two, three, or four grids, averaging becomes difficult... 4) The child-uncle problem is also solved by having parent grids synchronize child 'fluxes'. The child 'fluxes' don't need to be averaged since the children's 'fluxes' should already be synchronized 5) w

8. AMR Algorithm ● Each grid needs to do the following: 1) Prolongate the coarse data from it's parent grid (out to rmbc ghost zones) (childmask=parent%childmask-1) 2) Receive better data from old overlapping grids (including ghost regions) (not child mask!) 3) Apply the prolongation fixup to the aux fields 1) Determine where to place children (while avoiding improper nesting) 2) Communicate location of children to neighboring ghost zones. (This can happen in step 7) 3) Create children 4) Calculate fluxes and emf's and apply fluxes everywhere but at coarse fine boundaries (what to do with ghost regions?) 5) Wait for children to take two sub-steps 6) Receive and apply coarsened data and fluxes from children 7) Synchronize fixup fluxes at boundaries with neighbors using childmask and synchronize emf's at boundaries 8) Complete conservative update using coarsened data and/or fluxes and emf's. 9) Update coarsened data in ghost regions from neighbors (First Time Only) 4) Coarsen data and fixup fluxes to the parent grid. Parents send and receive coarsened data to children once per every two child steps. Children communicate overlapping data once per every two steps. Children communicate ghost cell centered data once per every two steps – and emf's are synchronized every step. 2x