1. Theory and Case Studies
(Active) 3D Scanning
Theory and Case Studies

2. Range Acquisition Taxonomy
Mechanical (CMM, jointed arm)
Inertial (gyroscope, accelerometer)
Contact
Ultrasonic trackers
Magnetic trackers
Industrial CT
Range acquisition
Transmissive
Ultrasound
MRI
Radar
Non-optical
Sonar
Reflective
Optical

3. Range Acquisition Taxonomy
Shape from X:
stereo
motion
shading
texture
focus
defocus
Passive
Optical methods
Active variants of passive methods
Stereo w. projected texture
Active depth from defocus
Photometric stereo
Active
Time of flight
Triangulation

4. Active Optical Methods
Advantages:
Usually can get dense data
Usually much more robust and accurate than passive techniques
Disadvantages:
Introduces light into scene (distracting, etc.)
Not motivated by human vision

5. Active Variants of Passive Techniques
Regular stereo with projected texture
Provides features for correspondence
Active depth from defocus
Known pattern helps to estimate defocus
Photometric stereo
Shape from shading with multiple known lights

6. Pulsed Time of Flight
Basic idea: send out pulse of light (usually laser), time how long it takes to return

7. Pulsed Time of Flight Advantages: Disadvantages:
Large working volume (up to 100 m.)
Disadvantages:
Not-so-great accuracy (at best ~5 mm.)
Requires getting timing to ~30 picoseconds
Does not scale with working volume
Often used for scanning buildings, rooms, archeological sites, etc.

8. AM Modulation Time of Flight
Modulate a laser at frequencym , it returns with a phase shift 
Note the ambiguity in the measured phase!  Range ambiguity of 1/2mn

9. AM Modulation Time of Flight
Accuracy / working volume tradeoff (e.g., noise ~ 1/500 working volume)
In practice, often used for room-sized environments (cheaper, more accurate than pulsed time of flight)

10. Triangulation

11. Triangulation: Moving the Camera and Illumination
Moving independently leads to problems with focus, resolution
Most scanners mount camera and light source rigidly, move them as a unit

12. Triangulation: Moving the Camera and Illumination

13. Triangulation: Moving the Camera and Illumination

14. Triangulation: Extending to 3D
Possibility #1: add another mirror (flying spot)
Possibility #2: project a stripe, not a dot
Laser
Object
Camera

15. Triangulation Scanner Issues
Accuracy proportional to working volume (typical is ~1000:1)
Scales down to small working volume (e.g. 5 cm. working volume, 50 m. accuracy)
Does not scale up (baseline too large…)
Two-line-of-sight problem (shadowing from either camera or laser)
Triangulation angle: non-uniform resolution if too small, shadowing if too big (useful range: 15-30)

16. Triangulation Scanner Issues
Material properties (dark, specular)
Subsurface scattering
Laser speckle
Edge curl
Texture embossing

17. Multi-Stripe Triangulation
To go faster, project multiple stripes
But which stripe is which?
Answer #1: assume surface continuity

18. Multi-Stripe Triangulation
To go faster, project multiple stripes
But which stripe is which?
Answer #2: colored stripes (or dots)

19. Multi-Stripe Triangulation
To go faster, project multiple stripes
But which stripe is which?
Answer #3: time-coded stripes

20. Time-Coded Light Patterns
Assign each stripe a unique illumination code over time [Posdamer 82]
Time
Space

21. Applications of 3D Scanning – Scanning Sculptures
The Pietà Project
IBM Research
The Digital Michelangelo Project
Stanford University
The Great Buddha Project
University of Tokyo

22. Why Scan Sculptures? Sculptures interesting objects to look at
Introduce scanning to new disciplines
Art: studying working techniques
Art history
Cultural heritage preservation
Archeology
High-visibility projects

23. Why Scan Sculptures? Challenging But not too challenging
High detail, large areas
Large data sets
Field conditions
Pushing hardware, software technology
But not too challenging
Simple topology
Possible to scan most of surface

24. Issues Addressed Resolution Coverage Type of data
Theoretical: limits of scanning technologies
Practical: physical access, time
Type of data
High-res 3D data vs. coarse 3D + normal maps
Influenced by eventual application
Intellectual Property

25. IBM’s Pietà Project Michelangelo’s “Florentine Pietà”
Late work (1550s)
Partially destroyed by Michelangelo, recreated by his student
Currently in the Museo dell’Opera del Duomo in Florence

26. Who?
Dr. Jack Wasserman, professor emeritus of art history at Temple University
Visual and Geometric Computing IBM Research:
Fausto Bernardini
Holly Rushmeier
Ioana Martin
Joshua Mittleman
Gabriel Taubin
Andre Gueziec
Claudio Silva

27. Scanner Visual Interface “Virtuoso” Active multibaseline stereo
Projector (stripe pattern), 6 B&W cameras, 1 color camera
Augmented with 5 extra “point” light sources for photometric stereo (active shape from shading)

28. Data Range data has 2 mm spacing, 0.1mm noise
Each range image: 10,000 points, 2020 cm
Color data: 5 images with controlled lighting, 1280960, 0.5 mm resolution
Total of 770 scans, 7.2 million points

29. Scanning Final scan June 1998, completed July 1999
Total scanning time: 90 hours over 14 days (includes equipment setup time)

30. Postprocessing
Use 1111 grid of projected laser dots to help with pairwise alignment
Align all scans to each other, then apply nonrigid “conformance smoothing”
Reconstruct surface using BPA
Compute normal and albedo maps, align to geometry

31. Results

32. The Digital Michelangelo Project

33. Goals Scan 10 sculptures by Michelangelo
High-resolution (“quarter-millimeter”) geometry
Side projects: architectural scanning (Accademia and Medici chapel), scanning fragments of Forma Urbis Romae

34. Why Capture Chisel Marks?
0:45
So why capture chisel marks?
Two reasons…first…
5 of the statues we scanned are unfinished
this is the slave called Atlas, in the Galleria dell'Accademia
On these unfinished statues
you can see Michelangelo’s working methods, and
and, since we know what kinds of chisels Michelangelo used
we might be able to use our models to make hypotheses about which chisels he used where
(skip) whether we can make and prove such hypotheses or not is still unclear
By the way, look at the whitish stripes indicated by arrows in the middle image
that whitening is caused by the crushing of marble crystals under the impact of Michelangelo’s chisel
we wanted to capture that whitening in our color data
you’ll see it again later in the talk
ugnetto ?
Atlas (Accademia)

35. Why Capture Chisel Marks as Geometry?
0:50
On Michelangelo’s (more or less) finished statues
like Day, in the Medici Chapel
there’s a different reason for capturing chisel marks
Look at a closeup of his back
note the modeling of the muscles.
Well,
the carving is not nearly as deep as it appears in this image
Michelangelo is playing an optical trick on us
If we zoom even closer, we see that
the surface is covered with tiny grooves
made using a multi-tooth chisel called a gradina
These grooves cast tiny shadows
as the surface curves away from the light
This self-shadowing darkens the surface near grazing angles (relative to the light),
making the surface relief seem deeper than it really is!
If we want to reproduce this effect
we need to capture these grooves…
and as geometry, not as texture!
2 mm
Day (Medici Chapel)

36. Who? 0:30 to end 0:15 Let me finish by reminding you that
this is not the work of 2 people
or even of 12
but rather of 60 or 70
some of whom are acknowledged here
some in the paper
Faculty and staff
Prof. Brian Curless John Gerth
Jelena Jovanovic Prof. Marc Levoy
Lisa Pacelle Domi Pitturo
Dr. Kari Pulli
Graduate students
Sean Anderson Barbara Caputo
James Davis Dave Koller
Lucas Pereira Szymon Rusinkiewicz
Jonathan Shade Marco Tarini
Daniel Wood
Undergraduates
Alana Chan Kathryn Chinn
Jeremy Ginsberg Matt Ginzton
Unnur Gretarsdottir Rahul Gupta
Wallace Huang Dana Katter
Ephraim Luft Dan Perkel
Semira Rahemtulla Alex Roetter
Joshua Schroeder Maisie Tsui
David Weekly
In Florence
Dottssa Cristina Acidini Dottssa Franca Falletti
Dottssa Licia Bertani Alessandra Marino
Matti Auvinen
In Rome
Prof. Eugenio La Rocca Dottssa Susanna Le Pera
Dottssa Anna Somella Dottssa Laura Ferrea
In Pisa
Roberto Scopigno
Sponsors
Interval Research Paul G. Allen Foundation for the Arts
Stanford University
Equipment donors
Cyberware Cyra Technologies
Faro Technologies Intel
Silicon Graphics Sony
3D Scanners

37. Scanner Design Flexibility Accuracy
1:40 through closeup of St. Matthew
0:50
Here’s the scanner we built for the project
It has four degrees of freedom
a horizontal arm that moves up and down
a carriage that travels left and right, and
a pan-tilt assembly that sits on top of the carriage
This assembly holds a scan head containing
a laser and digital video camera for acquiring range, and a
white light and digital still camera for acquiring color
People often comment on the
unusual shape of our scanner
it incorporates a number of unique features to make it
flexible – in particular reconfigurable, and
accurate – in particular by minimizing deflections during motion
I don’t have time to go through these features now
in the paper we try to sketch out the
whole decision tree we followed when designing the scanner
including the branches we didn’t take
4 motorized axes
Flexibility
outward-looking rotational scanning
16 ways to mount scan head on arm
Accuracy
center of gravity kept stationary during motions
precision drives, vernier homing, stiff trusses
laser, range camera,
white light, and color camera

38. Scanning a Large Object
In order to scan large objects, we have to move the scanhead in various ways. For a single sweep, we use a pitching motion that scans surfaces within the curved tube illustrated in yellow.
To make wider scans, we employ the panning motion by rotating the scanhead so the next sweep slightly overlaps the previous one. All these sweeps form a shell, which is illustrated in blue.
Finally, we can make the effective working volume thicker by horizontally translating the scanhead, thus collecting overlapping shells into a scan.
In order to scan more of the object, we need to relocate the scanhead using a wider, uncalibrated motion. Such motions include vertical translation, rolling the whole gantry to a new position, and removing and reattaching the scanhead in a different orientation, such as below the horizontal translation stage, or even sideways for horizontal scanning. Finally, we could extend the scanner's vertical reach by adding extra segments to the gantry.
Calibrated motions
pitch (yellow)
pan (blue)
horizontal translation (orange)
Uncalibrated motions
vertical translation
rolling the gantry
remounting the scan head

39. Single Scan of St. Matthew
0:30
Here’s the range image acquired during one sweep of the laser stripe
the texture you see… are Michelangelo’s chisel marks
here’s a plot through his cheek
each valley is the groove left by one tooth of a multi-tooth chisel
possibly this two-tooth chisel called a Dente di cane ("dog-tooth")
The spacing between our samples points (in X and Y) is ¼ mm
our depth resolution is 50 microns, but
(skip) how good is that 50 microns?…
1 mm
Single Scan of St. Matthew
0:30

40. Postprocessing Manual initial alignment
Pairwise ICP, then global registration
VRIP (parallelized across subvolumes)
Use high-res geometry to discard bad color data, perform inverse lighting calculations

41. Statistics About the Scan of David
0:20
So here’s a rendering of our computer model of the David
we rolled the gantry, which now weighed about a ton, 480 times
2 billion polygons
you can read the rest
It took 30 days to scan the statue
but we were only allowed to scan at night, when the museum was closed
so it was basically 30 all-nighters in a row
480 individually aimed scans
0.3 mm sample spacing
2 billion polygons
7,000 color images
32 gigabytes
30 nights of scanning
22 people
0:30

42. Head of Michelangelo’s David
0:20
We haven’t assembled the David at full-resolution yet
Here’s a 1 mm model of the head
watertight
with color
The photograph at the left was taken with uncalibrated lighting
so the two images don’t match exactly
but hopefully you agree that we’ve basically captured the statue’s appearance
Photograph
1.0 mm computer model

43. Side project: The Forma Urbis Romae
1:20
Throughout the project
we looked for interesting acquisition opportunities
here's one we found
The Forma Urbis Romae
a giant marble map of ancient Rome
60 feet wide, 45 feet high, 4 inches thick
carved in ancient times
now lying in fragments – 1,163 of them
Here’s one fragment
about 4 feet wide, 250 pounds
If we zoom in on it
we can see, incised on its surface
every room, every door, every staircase in the city
a feat of mapmaking that has never been matched
To make a long story short
solving this jigsaw puzzle is one of the key open problems of classical archeology
our idea was to scan the fragments…

44. Forma Urbis Romae Fragment
and then
search among their side faces for matches
It’s essentially
a large geometric alignment problem
of the kind we’ve been talking about
but this time with no initial guesses
side face
0:45

45. Here’s the whole puzzle
We don’t expect to fit all these pieces together
but if we fit even a few together
we might be able to locate monuments
described in the histories of Pliny the Elder
that have been lost since antiquity
so we’re pretty excited about this side project

46. Hard Problems
Keeping scanner calibrated is hard in the lab, really hard in the museum
Dealing with large data sets is painful
Filling all the holes converges only asymptotically (if it converges at all…)

47. The Great Buddha Project
Great Buddha of Kamakura
Original made of wood, completed 1243
Covered in bronze and gold leaf, 1267
Approx. 15 m tall
Goal: preservation of cultural heritage

48. Who? Institute of Industrial Science, University of Tokyo
Daisuke Miyazaki
Takeshi Ooishi
Taku Nishikawa
Ryusuke Sagawa
Ko Nishino
Takashi Tomomatsu
Yutaka Takase
Katsushi Ikeuchi

49. Scanner Cyrax range scanner by Cyra Technologies
Laser pulse time-of-flight
Accuracy: 4 mm
Range: 100 m

50. Processing 20 range images (a few million points)
Simultaneous all-to-all ICP
Variant of volumetric merging (parallelized)

51. Results