H.H. Bayraktar, T.M. Keaveny, P. Papadopoulos and Atul Gupta Ultrascalable Implicit Finite Element Analyses in Solid Mechanics with over a Half a Billion Degrees of Freedom (excerpts) Mark F. Adams H.H. Bayraktar, T.M. Keaveny, P. Papadopoulos and Atul Gupta
Trabecular Bone Cortical bone Trabecular bone 5-mm Cube
Micro-Computed Tomography Methods: FE modeling Micro-Computed Tomography CT @ 22 m resolution 3D image Mechanical Testing E, yield, ult, etc. 2.5 mm cube 44 m elements FE mesh
the vertebral body you are showing is pretty healthy from a 80 year old female and it is a T-10 that is thoracic. So it is pretty close to the mid-spine. Usually research is done from T-10 downward to the lumbar vertebral bodies. There are 12 thoracic VB's and 5 lumbar. The numbers go up as you go down.
1 mm slice from vertebral body
Vertebral Body With Shell 80 µm w/ shell Large deformation elast. 6 load steps (3% strain) Scaled speedup ~131K dof/processor 7 to 537 million dof 4 to 292 nodes IBM SP Power3 14 of 16 procs/node
Computational Architecture FE Mesh Input File ParMetis Athena Partition to SMPs FE input file (in memory) FE input file (in memory) Athena: Parallel FE ParMetis Parallel Mesh Partitioner (Univerisity of Minnesota) Prometheus Multigrid Solver FEAP Serial general purpose FE application (University of California) PETSc Parallel numerical libraries (Argonne National Labs) ParMetis Athena Athena File File File File FEAP FEAP FEAP FEAP Material Card pFEAP Silo DB Olympus METIS Prometheus METIS METIS METIS Visit ParMetis PETSc
ParMetis partitions
131K dof / proc .47 Teraflops - 4088 processors