1. H.H. Bayraktar, T.M. Keaveny, P. Papadopoulos and Atul Gupta
Ultrascalable Implicit Finite Element Analyses in Solid Mechanics with over a Half a Billion Degrees of Freedom (excerpts)
Mark F. Adams
H.H. Bayraktar, T.M. Keaveny, P. Papadopoulos and Atul Gupta

2. Trabecular Bone
Cortical bone
Trabecular bone
5-mm Cube

3. Micro-Computed Tomography
Methods: FE modeling
Micro-Computed Tomography
22 m resolution
3D image
Mechanical Testing
E, yield, ult, etc.
2.5 mm cube
44 m elements
FE mesh

4. the vertebral body you are showing is pretty healthy from a 80 year old female and it is a T-10 that is thoracic. So it is pretty close to the mid-spine. Usually research is done from T-10 downward to the lumbar vertebral bodies. There are 12 thoracic VB's and 5 lumbar. The numbers go up as you go down.

5. 1 mm slice from vertebral body

6. Vertebral Body With Shell
80 µm w/ shell
Large deformation elast.
6 load steps (3% strain)
Scaled speedup
~131K dof/processor
7 to 537 million dof
4 to 292 nodes
IBM SP Power3
14 of 16 procs/node

7. Computational Architecture
FE Mesh Input File
ParMetis
Athena
Partition to SMPs
FE input file (in memory)
FE input file (in memory)
Athena: Parallel FE
ParMetis
Parallel Mesh Partitioner (Univerisity of Minnesota)
Prometheus
Multigrid Solver
FEAP
Serial general purpose FE application (University of California)
PETSc
Parallel numerical libraries (Argonne National Labs)
ParMetis
Athena
Athena
File
File
File
File
FEAP
FEAP
FEAP
FEAP
Material Card
pFEAP
Silo DB
Olympus
METIS
Prometheus
METIS
METIS
METIS
Visit
ParMetis
PETSc

8. ParMetis partitions

9. 131K dof / proc .47 Teraflops - 4088 processors