1. Diego d’Aulignac GRAVIR/INRIA Rhone-Alpes France
Modélisation de l’interaction avec objets déformables en temps-réel pour des simulateurs médicaux
Diego d’Aulignac
GRAVIR/INRIA Rhone-Alpes
France

2. Medical Simulators Motivations Objectives danger to patients cost
certification
Objectives
Geometric Models
Physical Models
deformation
interaction

3. Problems Simulation MUST be real-time! Simulation MUST be realistic!
deformation
resolution
Simulation MUST be realistic!
model
identification of parameters
Simulation MUST be interactive!
collision detection
haptic interaction

4. Plan Deformation Models Real-time Resolution Techniques
Mass-Spring vs. FEM
Real-time Resolution Techniques
Static
Dynamic
Echographic Simulator
parameter identification
Liver Model
interactive deformation

5. Deformable Object Geometry Elements Comparison Springs [TW90]
Tetrahedra FEM [OH99]
Comparison
Realism
Speed

6. Geometrical Model 56 surface points 108 triangles 57 total points
120 tetrahedra
230 edges

7. Mass-Spring Model
Initial length
Deformed length

8. Finite Element Method (FEM)
Deformed configuration
Deformation tensor:
Initial configuration x a
Green’s strain
displacements
Small strain
Cauchy Strain:

9. Strain-Stress
Deformation Energy
Lamé coefficients
force per unit area

10. Mass-Spring Model Springs are placed along the edges (230)
Not very realistic: modeling a volume with springs!
The force of each spring relatively cheap to evaluate
globally fast

11. Finite Element Method (FEM)
120 tetrahedra using Green’s strain tensor
Continuum is modeled with volumetric element.
Dilatation may be controlled
Approximately four times slower than mass-spring network

12. Deformable Models (conclusions)
Mass-Spring
One dimentional elements
Unrealistic to model volume
Tetrahedral FEM
Good realism for 3D continuum
Control of dilatation
Approximately 4 times slower to evaluate forces

13. Contributions
Quantitative and qualitative comparison of mass-springs and tetrahedral elements
Interactive non-linear static resolution
Formal analysis of the real-time stability of integration methods
based on parameters
Identification of the parameters of a model from experimental data
Relevant medical applications

14. Plan Deformation Models Real-time Resolution Techniques
Mass-Spring vs. FEM
Real-time Resolution Techniques
Static
Dynamic
Echographic Simulator
parameter identification
Liver Model
interactive deformation

15. Real-time Resolution Static Resolution Dynamic resolution
linear resolution [Cotin97]
small displacements
Our approach: non-linear resolution
large displacements
Dynamic resolution
explicit [Picinbono01]
implicit [BW98]

16. Linear Static Resolution
Linear case:
Pre-inversion (if enough space)
No large strain
No rotation
No material non-linearity
Principle of virtual work:
internal and external forces are balanced

17. Nonlinear Static Resolution
Non-linear case:
Stiffness matrix changes with displacement:
geometric
material

18. Newton Iteration Full Newton-Rapson method: Reevaluation of Jacobian
Faster convergence
Modified Newton-Rapson method:
Constant Jacobian
Slower Convergence

19. Dynamic Analysis 2nd order non-linear differential equation Convert to
1st order
system

20. Explicit Integration Runge-Kutta method with s stages s
Order of consistency (accuracy) vs. stages
precision

21. Explicit Integration Stability
linearizing Im
Timestep is limited by the
the physical parameters! Re

22. Implicit Integation B-stable implicit euler: linearisation
If you know your history, then you would know where you are coming from.
Bob Marley
Implicit Integation
Over-damped case
B-stable implicit euler:
linearisation
Semi-implicit euler
Stable for linear case (A-stable)
any timestep
any physical parameters

23. Resolution (conclusions)
Static analysis
non-linear resolution for large displacements
Dynamic
explicit
strict stability criteria
implicit
no limit on timestep, but resolution of non-linear system

24. Contributions
Quantitative and qualitative comparison of mass-springs and tetrahedral elements
Interactive non-linear static resolution
Formal analysis of the real-time stability of integration methods
based on parameters
Identification of the parameters of a model from experimental data
Relevant medical applications

25. Plan Deformation Models Real-time Resolution Techniques
Mass-Spring vs. FEM
Real-time Resolution Techniques
Static
Dynamic
Echographic Simulator
parameter identification
Liver Model
interactive deformation

26. Thigh Echography

27. Echographic Simulator
Data Acquisition
Model of the thigh
Mass-Spring
Neural
Interaction
collision
haptics
Generation of echographic image

28. Data Acquisition
(at LIRMM, Montpellier)
64 sample points are marked on the thigh. For each, the forces for some given penetrations are measured
Two different probes
(a) Indentor shaped probe for punctual force-penetration data
(b) Probe with surface equal to that of a typical echographic probe
1- The end effector advances in small steps (2mm) in the direction normal to the surface of the thigh.
2- The force depending on the penetration distance is measured

29. Data Acquisition: Experimental Results
[d’Aulignac et al.
MICCAI 99]
Data Acquisition: Experimental Results
displacement
Force
displacement
Force
Indentor probe
Surface probe
The two probes do not offer the same resistance
difference in surface area
Different curves for different points
different depth of soft tissue
Highly non-linear behaviour

30. Echographic Simulator
Data Acquisition
Model of the thigh
Mass-Spring
Neural
Interaction
collision
haptics
Generation of echographic image

31. Dynamic Model of the thigh
Incompressibility
of the tissue
Elasticity of the
epidermis
Why mass-spring model?
computationally efficient
interior NOT discretized into tetrahedra

32. Identification of the Parameters of a Dynamic Model
Optimization Algorithm
New parameters (elasticity, plasticity, collision stiffness ...)
Error -
Behaviour
Model
Resolution
Desired behaviour
Measurements
For each sample point, deformation/force values with each probe
=> Total of ~1200 measurements.

33. Distribution of Nonzero Error Values
(in collaboration with UC Berkeley)
[d’Aulignac et al., IROS 99]
Parameter Estimation
Least-squares minimisation:
1. find (a,b) for each non-linear spring
2. find (a,b) for each non-linear spring, and (a) for all linear springs
=> Avoid local minima
Distribution of Nonzero Error Values
Error of the model with respect to the experimental data
=> Overall error less than 5%
Error (N)

34. Dynamic Analysis Explicit integration Implicit integration
Euler stability
too small timesteps
no real-time
...or large mass
slow movement
no gravity
Implicit integration
Semi-Implicit Euler
constant Jacobian
100 steps per second
h=1/100 (i.e. real time)

35. Dynamic Resolution
100 Hz using semi-implicit integration

36. Neural Networks Displacement of particles: u Static Analysis
Multi-layer perceptron is a general approximizer
Network is trained directly on experimental data
back-propagation
Forces acting on particles: f
64 inputs and outputs

37. Neural Networks Neural Model Experimental data Displacement (mm)
Force (N)
Neural Model
Experimental data

38. Mass-Spring vs. Neural Model
topology chosen
based on measurements
dynamic resolution
semi-implicit (100 Hz)
Neural model
no assuption on topology
static resolution
very fast
no change of topology

39. Echographic Simulator
Data Acquisition
Model of the thigh
Mass-Spring
Neural
Interaction
collision
haptics
Generation of echographic image

40. Interaction
Collision Detection
Collision Response
Force Feedback

41. Collision Detection Finds polygons in the OpenGL viewing frustrum
Detects collision between simple rigid body and any other object quickly

42. Collision Response Inter-penetration distance must be computed
Penalty forces [Hunt and Crossley 1975]
Inter-penetration distance must be computed
Generates large forces (bad for haptics)

43. Haptics Haptic devices require high update frequency
typically around 1kHz
….which the simulation normally can’t meet
100 Hz (dynamic model)

44. Haptic Interaction Local approximation of the contact
simple local model running in a separate thread
fast collision detection
fast force computation
Haptic loop (1kHz):
collision detection and response with local model
[Balaniuk 99]
Local model update
position
Simulation Loop (100Hz):
deformation
global collision detection and response

45. Haptic Feedback With local model force time Without local model
[d’Aulignac et al. ,
ICRA, 2000]
Haptic Feedback
With local model
force
time
Without local model

46. Echographic Simulator
Data Acquisition
Model of the thigh
Mass-Spring
Neural
Interaction
collision
haptics
Generation of echographic image

47. Echographic Image Generation
[Vieira01] (in collaboration with TIMC-IMAG, France)
64 images aquired
on each sample point
Voxel Map
120 Mb
Interpolation
fill in the blanks
Provide image
any rotation
any position

48. Echographic Image Deformation
Problem
structures deform differently
vein
bone, etc.
segmentation
Linear deformation
Possible extension: precalculated deformation maps [Troccaz et al, 2000]

49. A first Prototype

50. Echographic Simulator (conclusions)
Data Acquisition
Model of the thigh
Mass-Spring
Neural
Interaction
local model
Generation of echographic image
linear deformation

51. Contributions
Quantitative and qualitative comparison of mass-springs and tetrahedral elements
Interactive non-linear static resolution
Formal analysis of the real-time stability of integration methods
based on parameters
Identification of the parameters of a model from experimental data
Relevant medical applications

52. Plan Deformation Models Real-time Resolution Techniques
Mass-Spring vs. FEM
Real-time Resolution Techniques
Static
Dynamic
Echographic Simulator
parameter identification
Liver Model
interactive deformation

53. Keyhole Surgery Surgery involves soft tissues simulation
Need to model deformation
…in real-time!

54. Human Liver Interior composed of parenchyma
Surounded by elastic skin or Glisson’s capsule
Venous network
Approximate weight: 1.5 kg

55. Liver Model Geometry Physical Model Dynamic Analysis Static Analysis
explicit integration stability
Static Analysis
non-linear resolution

56. Geometrical Model 187 Vertices 370 Triangles 299 Particles GHS3D
1151 Tetrahedra
1634 Edges
GHS3D

57. Physical Model Heterogenous Non-linear: Stress Strain
[Boux et al., ISER, 2000]
Heterogenous
Non-linear:
Stress
Strain
skin
Parenchyma
Weight distributed equaly on all particles (i.e. approximately 5g each)

58. Explicit Integration
280 steps per second
mass 5 grams

59. Stability Analysis Im Re

60. Simulation Achitecture SGI Onyx2 Compexity 370 facets 1151 tetrahedra
3399 springs
Frequency
150Hz
Explicit not stable!
...large mass

61. Static Resolution
The large deformations of the organ during operation require non-linear resolution techniques.

62. Iterative Solution Calculate forces on nodes
Evaluate stiffness matrix K? (analytically)
Iteratively solve linear system for displacements u
Ku = f
by successive over- relaxation (SOR)
until residual forces < epsilon through Newton-Rapson iteration

63. Modified Newton-Raphson
Accurate solution (many SOR iterations) does not allow faster solution
Inexact Jacobian limits convergence speed
Of special importance for strong nonlinearities
residual
iterations

64. Newton-Raphson Less iteration to converge then modified NR
Exact Jacobian allows faster convergence
Global time gain when solving linear system accurately
residual
iterations

65. Pseudo-Dynamic
Interactive resolution of the non-linear system.

66. Result Pseudo-dynamic 1157 tetrahedra Iterative non-linear resolution
Rotational invarience
(N.B. Real-time animation)
60 NR iterations/sec on SGI Octane 175Mhz

67. Liver Model (conclusions)
Physical Model
mass-springs
Dynamic Analysis
explicit integration unstable
Static Analysis
interactive non-linear resolution

68. Summary Physical Models Resolution Medical Simulators
Mass-Spring or FEM?
Resolution
Static
linear or non-linear?
Dynamic
explicit or implicit?
Medical Simulators
The choice of numerical methods must be guided by the application!

69. Contributions
Quantitative and qualitative comparison of mass-springs and tetrahedral elements
Interactive non-linear static resolution
Formal analysis of the real-time stability of integration methods
based on parameters
Identification of the parameters of a model from experimental data
Relevant medical applications

70. Local Model

71. Explicit Integration Dynamic equations solved by Euler’s method
Linearizing by assuming constant matrices we can calculate derivative analytically
The absolute value of (1+z) must be smaller than 1

72. Backwards engineering
Geometrical description
Geometrical description
elasticity
elasticity
Physical Model
Physical Model
forces
displacement
forces
displacements
Results
Results