(Active) 3D Scanning Theory and Case Studies. 3D Scanning Applications Computer graphicsComputer graphics Product inspectionProduct inspection Robot navigationRobot.

Slides:



Advertisements
Similar presentations
3D Head Mesh Data Stereo Vision Active Stereo 3D Reconstruction 3dMD System 1.
Advertisements

TERRESTRIAL LASER SCANNING (TLS): APPLICATIONS TO ARCHITECTURAL AND LANDSCAPE HERITAGE PRESERVATION – PART 1.
Kawada Industries Inc. has introduced the HRP-2P for Robodex 2002
Structured Light principles Figure from M. Levoy, Stanford Computer Graphics Lab.
Jan Visualization with 3D CG Masaki Hayashi Digitization.
Structured light and active ranging techniques Class 11.
Computer Vision Spring ,-685 Instructor: S. Narasimhan Wean 5403 T-R 3:00pm – 4:20pm Lecture #17.
Vision Sensing. Multi-View Stereo for Community Photo Collections Michael Goesele, et al, ICCV 2007 Venus de Milo.
Stereo Many slides adapted from Steve Seitz. Binocular stereo Given a calibrated binocular stereo pair, fuse it to produce a depth image Where does the.
Structured Light + Range Imaging Lecture #17 (Thanks to Content from Levoy, Rusinkiewicz, Bouguet, Perona, Hendrik Lensch)
SA-1 Probabilistic Robotics Probabilistic Sensor Models Beam-based Scan-based Landmarks.
Stereo.
December 5, 2013Computer Vision Lecture 20: Hidden Markov Models/Depth 1 Stereo Vision Due to the limited resolution of images, increasing the baseline.
The Digital Michelangelo Project: 3D scanning of large statues Marc Levoy, Kari Pulli, Brian Curless, Szymon Rusinkiewicz, David Koller, Lucas Pereira,
The Digital Michelangelo Project Marc Levoy Computer Science Department Stanford University.
SIGGRAPH Course 30: Performance-Driven Facial Animation Section: Markerless Face Capture and Automatic Model Construction Part 2: Li Zhang, Columbia University.
Marc Levoy 1 Szymon Rusinkiewicz 1 Matt Ginzton 1 Jeremy Ginsberg 1 Kari Pulli 1 David Koller 1 Sean Anderson 1 Jonathan Shade 2 Brian Curless 2 Lucas.
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2005 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
Structured light and active ranging techniques Class 8.
A Laser Range Scanner Designed for Minimum Calibration Complexity James Davis, Xing Chen Stanford Computer Graphics Laboratory 3D Digital Imaging and Modeling.
Acknowledgement: some content and figures by Brian Curless
Introduction to Computer Vision 3D Vision Topic 9 Stereo Vision (I) CMPSCI 591A/691A CMPSCI 570/670.
Overview of 3D Scanners Acknowledgement: some content and figures by Brian Curless.
Real-Time 3D Model Acquisition
3D from multiple views : Rendering and Image Processing Alexei Efros …with a lot of slides stolen from Steve Seitz and Jianbo Shi.
Computer Science Department
3D Scanning. Computer Graphics Pipeline Human time = expensiveHuman time = expensive Sensors = cheapSensors = cheap – Computer graphics increasingly relies.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Stereo Guest Lecture by Li Zhang
Project 1 artifact winners Project 2 questions Project 2 extra signup slots –Can take a second slot if you’d like Announcements.
A Hierarchical Method for Aligning Warped Meshes Leslie Ikemoto 1, Natasha Gelfand 2, Marc Levoy 2 1 UC Berkeley, formerly Stanford 2 Stanford University.
The Digital Michelangelo Project Marc Levoy Computer Science Department Stanford University.
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2006 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry Topics: Basics of projective geometry Points and hyperplanes in projective space Homography.
3D Photography Szymon Rusinkiewicz Fall D Photography Obtaining 3D shape (and sometimes color) of real-world objectsObtaining 3D shape (and sometimes.
Theory and Case Studies
Computer Vision Spring ,-685 Instructor: S. Narasimhan WH 5409 T-R 10:30am – 11:50am Lecture #15.
3D Vision. 3D Perception: Illusions Block & Yuker.
Structured light and active ranging techniques Class 8
Digital Archive of Cultural Heritages - Three Dimensional Modeling of Cultural Heritages and its use for Educational Purposes - Atsushi Nakazawa, Takeshi.
1 Intelligent Robotics Research Centre (IRRC) Department of Electrical and Computer Systems Engineering Monash University, Australia Visual Perception.
Technology and Historical Overview. Introduction to 3d Computer Graphics  3D computer graphics is the science, study, and method of projecting a mathematical.
Sensors Ioannis Stamos. Perspective projection Ioannis Stamos – CSCI F08 Pinhole & the Perspective Projection SCREEN SCENE (x,y) Is there an image.
Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University.
Visual Perception PhD Program in Information Technologies Description: Obtention of 3D Information. Study of the problem of triangulation, camera calibration.
December 4, 2014Computer Vision Lecture 22: Depth 1 Stereo Vision Comparing the similar triangles PMC l and p l LC l, we get: Similarly, for PNC r and.
Stereo Many slides adapted from Steve Seitz.
A General-Purpose Platform for 3-D Reconstruction from Sequence of Images Ahmed Eid, Sherif Rashad, and Aly Farag Computer Vision and Image Processing.
1 Reconstructing head models from photograph for individualized 3D-audio processing Matteo Dellepiane, Nico Pietroni, Nicolas Tsingos, Manuel Asselot,
Spring 2015 CSc 83020: 3D Photography Prof. Ioannis Stamos Mondays 4:15 – 6:15
Lec 22: Stereo CS4670 / 5670: Computer Vision Kavita Bala.
Tutorial Visual Perception Towards Computer Vision
Lecture 16: Stereo CS4670 / 5670: Computer Vision Noah Snavely Single image stereogram, by Niklas EenNiklas Een.
Autonomous Robots Vision © Manfred Huber 2014.
Triangulation Scanner Design Options
1Ellen L. Walker 3D Vision Why? The world is 3D Not all useful information is readily available in 2D Why so hard? “Inverse problem”: one image = many.
Project 2 due today Project 3 out today Announcements TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAA.
Manufacturing Process II
Stereo CS4670 / 5670: Computer Vision Noah Snavely Single image stereogram, by Niklas EenNiklas Een.
Faculty of Sciences and Technology from University of Coimbra Generating 3D Meshes from Range Data [1] Graphic Computation and Three-dimensional Modeling.
CS4670 / 5670: Computer Vision Kavita Bala Lec 27: Stereo.
CSc 8820 Advanced Graphics Algorithms
3D Graphics Rendering PPT By Ricardo Veguilla.
3D Head Mesh Data Stereo Vision Active Stereo 3D Reconstruction
Common Classification Tasks
Acknowledgement: some content and figures by Brian Curless
Image Based Modeling and Rendering (PI: Malik)
--- Stereoscopic Vision and Range Finders
--- Stereoscopic Vision and Range Finders
Probabilistic Robotics
Presentation transcript:

(Active) 3D Scanning Theory and Case Studies

3D Scanning Applications Computer graphicsComputer graphics Product inspectionProduct inspection Robot navigationRobot navigation As-built floorplansAs-built floorplans Product design Archaeology Clothes fitting Art history

Industrial Inspection Determine whether manufactured parts are within tolerancesDetermine whether manufactured parts are within tolerances

Medicine Plan surgery on computer model, visualize in real timePlan surgery on computer model, visualize in real time

Medicine

Medicine

Medicine

Scanning Buildings Quality control during constructionQuality control during construction As-built modelsAs-built models

Scanning Buildings Quality control during constructionQuality control during construction As-built modelsAs-built models

Clothing Scan a person, custom-fit clothingScan a person, custom-fit clothing U.S. Army; booths in mallsU.S. Army; booths in malls

Range Acquisition Taxonomy Range acquisition Contact Transmissive Reflective Non-optical Optical Industrial CT Mechanical (CMM, jointed arm) Radar Sonar Ultrasound MRI Ultrasonic trackers Magnetic trackers Inertial (gyroscope, accelerometer)

Touch Probes Jointed arms with angular encodersJointed arms with angular encoders Return position, orientation of tipReturn position, orientation of tip Faro Arm – Faro Technologies, Inc.

Range Acquisition Taxonomy Optical methods Passive Active Shape from X: stereomotionshadingtexturefocusdefocus Active variants of passive methods Stereo w. projected texture Active depth from defocus Photometric stereo Time of flight Triangulation

Active Optical Methods Advantages:Advantages: – Usually can get dense data – Usually much more robust and accurate than passive techniques Disadvantages:Disadvantages: – Introduces light into scene (distracting, etc.) – Not motivated by human vision

Active Variants of Passive Techniques Regular stereo with projected textureRegular stereo with projected texture – Provides features for correspondence Active depth from defocusActive depth from defocus – Known pattern helps to estimate defocus Photometric stereoPhotometric stereo – Shape from shading with multiple known lights

Pulsed Time of Flight Basic idea: send out pulse of light (usually laser), time how long it takes to returnBasic idea: send out pulse of light (usually laser), time how long it takes to return

Pulsed Time of Flight Advantages:Advantages: – Large working volume (up to 100 m.) Disadvantages:Disadvantages: – Not-so-great accuracy (at best ~5 mm.) Requires getting timing to ~30 picosecondsRequires getting timing to ~30 picoseconds Does not scale with working volumeDoes not scale with working volume Often used for scanning buildings, rooms, archeological sites, etc.Often used for scanning buildings, rooms, archeological sites, etc.

AM Modulation Time of Flight Modulate a laser at frequency m, it returns with a phase shift Modulate a laser at frequency m, it returns with a phase shift  Note the ambiguity in the measured phase!  Range ambiguity of 1 / 2 m nNote the ambiguity in the measured phase!  Range ambiguity of 1 / 2 m n

AM Modulation Time of Flight Accuracy / working volume tradeoff (e.g., noise ~ 1 / 500 working volume)Accuracy / working volume tradeoff (e.g., noise ~ 1 / 500 working volume) In practice, often used for room-sized environments (cheaper, more accurate than pulsed time of flight)In practice, often used for room-sized environments (cheaper, more accurate than pulsed time of flight)

Triangulation

Triangulation: Moving the Camera and Illumination Moving independently leads to problems with focus, resolutionMoving independently leads to problems with focus, resolution Most scanners mount camera and light source rigidly, move them as a unitMost scanners mount camera and light source rigidly, move them as a unit

Triangulation: Moving the Camera and Illumination

Triangulation: Extending to 3D Possibility #1: add another mirror (flying spot)Possibility #1: add another mirror (flying spot) Possibility #2: project a stripe, not a dotPossibility #2: project a stripe, not a dot Object Laser CameraCamera

Triangulation Scanner Issues Accuracy proportional to working volume (typical is ~1000:1)Accuracy proportional to working volume (typical is ~1000:1) Scales down to small working volume (e.g. 5 cm. working volume, 50  m. accuracy)Scales down to small working volume (e.g. 5 cm. working volume, 50  m. accuracy) Does not scale up (baseline too large…)Does not scale up (baseline too large…) Two-line-of-sight problem (shadowing from either camera or laser)Two-line-of-sight problem (shadowing from either camera or laser) Triangulation angle: non-uniform resolution if too small, shadowing if too big (useful range: 15  -30  )Triangulation angle: non-uniform resolution if too small, shadowing if too big (useful range: 15  -30  )

Triangulation Scanner Issues Material properties (dark, specular)Material properties (dark, specular) Subsurface scatteringSubsurface scattering Laser speckleLaser speckle Edge curlEdge curl Texture embossingTexture embossing

Multi-Stripe Triangulation To go faster, project multiple stripesTo go faster, project multiple stripes But which stripe is which?But which stripe is which? Answer #1: assume surface continuityAnswer #1: assume surface continuity

Multi-Stripe Triangulation To go faster, project multiple stripesTo go faster, project multiple stripes But which stripe is which?But which stripe is which? Answer #2: colored stripes (or dots)Answer #2: colored stripes (or dots)

Multi-Stripe Triangulation To go faster, project multiple stripesTo go faster, project multiple stripes But which stripe is which?But which stripe is which? Answer #3: time-coded stripesAnswer #3: time-coded stripes

Time-Coded Light Patterns Assign each stripe a unique illumination code over time [Posdamer 82]Assign each stripe a unique illumination code over time [Posdamer 82] Space Time

Range Processing Pipeline StepsSteps 1. manual initial alignment 2. ICP to one existing scan 3. automatic ICP of all overlapping pairs 4. global relaxation to spread out error 5. merging using volumetric method

Range Processing Pipeline StepsSteps 1. manual initial alignment 2. ICP to one existing scan 3. automatic ICP of all overlapping pairs 4. global relaxation to spread out error 5. merging using volumetric method

Range Processing Pipeline StepsSteps 1. manual initial alignment 2. ICP to one existing scan 3. automatic ICP of all overlapping pairs 4. global relaxation to spread out error 5. merging using volumetric method

Range Processing Pipeline StepsSteps 1. manual initial alignment 2. ICP to one existing scan 3. automatic ICP of all overlapping pairs 4. global relaxation to spread out error 5. merging using volumetric method

Range Processing Pipeline StepsSteps 1. manual initial alignment 2. ICP to one existing scan 3. automatic ICP of all overlapping pairs 4. global relaxation to spread out error 5. merging using volumetric method +

Range Processing Pipeline StepsSteps 1. manual initial alignment 2. ICP to one existing scan 3. automatic ICP of all overlapping pairs 4. global relaxation to spread out error 5. merging using volumetric method

Applications of 3D Scanning – Scanning Sculptures The Pietà ProjectThe Pietà Project IBM Research The Digital Michelangelo ProjectThe Digital Michelangelo Project Stanford University The Great Buddha ProjectThe Great Buddha Project University of Tokyo

Why Scan Sculptures? Sculptures interesting objects to look atSculptures interesting objects to look at Introduce scanning to new disciplinesIntroduce scanning to new disciplines – Art: studying working techniques – Art history – Cultural heritage preservation – Archeology High-visibility projectsHigh-visibility projects

Why Scan Sculptures? ChallengingChallenging – High detail, large areas – Large data sets – Field conditions – Pushing hardware, software technology But not too challengingBut not too challenging – Simple topology – Possible to scan most of surface

Issues Addressed ResolutionResolution CoverageCoverage – Theoretical: limits of scanning technologies – Practical: physical access, time Type of dataType of data – High-res 3D data vs. coarse 3D + normal maps – Influenced by eventual application Intellectual PropertyIntellectual Property

IBM’s Pietà Project Michelangelo’s “Florentine Pietà”Michelangelo’s “Florentine Pietà” Late work (1550s)Late work (1550s) Partially destroyed by Michelangelo, recreated by his studentPartially destroyed by Michelangelo, recreated by his student Currently in the Museo dell’Opera del Duomo in FlorenceCurrently in the Museo dell’Opera del Duomo in Florence

Who? Dr. Jack Wasserman, professor emeritus of art history at Temple UniversityDr. Jack Wasserman, professor emeritus of art history at Temple University Visual and Geometric Computing IBM Research:Visual and Geometric Computing IBM Research: Fausto Bernardini Holly Rushmeier Ioana Martin Joshua Mittleman Gabriel Taubin Andre Gueziec Claudio Silva

Scanner Visual Interface “Virtuoso”Visual Interface “Virtuoso” Active multibaseline stereoActive multibaseline stereo Projector (stripe pattern), 6 B&W cameras, 1 color cameraProjector (stripe pattern), 6 B&W cameras, 1 color camera Augmented with 5 extra “point” light sources for photometric stereo (active shape from shading)Augmented with 5 extra “point” light sources for photometric stereo (active shape from shading)

Data Range data has 2 mm spacing, 0.1mm noiseRange data has 2 mm spacing, 0.1mm noise Each range image: 10,000 points, 20  20 cmEach range image: 10,000 points, 20  20 cm Color data: 5 images with controlled lighting, 1280  960, 0.5 mm resolutionColor data: 5 images with controlled lighting, 1280  960, 0.5 mm resolution Total of 770 scans, 7.2 million pointsTotal of 770 scans, 7.2 million points

Scanning Final scan June 1998, completed July 1999Final scan June 1998, completed July 1999 Total scanning time: 90 hours over 14 days (includes equipment setup time)Total scanning time: 90 hours over 14 days (includes equipment setup time)

Postprocessing Use 11  11 grid of projected laser dots to help with pairwise alignmentUse 11  11 grid of projected laser dots to help with pairwise alignment Align all scans to each other, then apply nonrigid “conformance smoothing”Align all scans to each other, then apply nonrigid “conformance smoothing” Reconstruct surface using BPAReconstruct surface using BPA Compute normal and albedo maps, align to geometryCompute normal and albedo maps, align to geometry

Results

The Digital Michelangelo Project

Goals Scan 10 sculptures by MichelangeloScan 10 sculptures by Michelangelo High-resolution (“quarter-millimeter”) geometryHigh-resolution (“quarter-millimeter”) geometry Side projects: architectural scanning (Accademia and Medici chapel), scanning fragments of Forma Urbis RomaeSide projects: architectural scanning (Accademia and Medici chapel), scanning fragments of Forma Urbis Romae

Why Capture Chisel Marks? Atlas (Accademia) ugnettougnetto ? ?

Why Capture Chisel Marks as Geometry? Why Capture Chisel Marks as Geometry? Day (Medici Chapel) 2 mm

Who? Faculty and staff Prof. Brian CurlessJohn Gerth Jelena JovanovicProf. Marc Levoy Lisa PacelleDomi Pitturo Dr. Kari Pulli Graduate students Sean AndersonBarbara Caputo James DavisDave Koller Lucas PereiraSzymon Rusinkiewicz Jonathan ShadeMarco Tarini Daniel Wood Undergraduates Alana ChanKathryn Chinn Jeremy GinsbergMatt Ginzton Unnur GretarsdottirRahul Gupta Wallace HuangDana Katter Ephraim LuftDan Perkel Semira RahemtullaAlex Roetter Joshua SchroederMaisie Tsui David Weekly In Florence Dottssa Cristina Acidini Dottssa Franca Falletti Dottssa Licia BertaniAlessandra Marino Matti Auvinen In Rome Prof. Eugenio La RoccaDottssa Susanna Le Pera Dottssa Anna SomellaDottssa Laura Ferrea In Pisa Roberto Scopigno Sponsors Interval ResearchPaul G. Allen Foundation for the Arts Stanford University Equipment donors CyberwareCyra Technologies Faro TechnologiesIntel Silicon GraphicsSony 3D Scanners

Scanner Design FlexibilityFlexibility – outward-looking rotational scanning – 16 ways to mount scan head on arm AccuracyAccuracy – center of gravity kept stationary during motions – precision drives, vernier homing, stiff trusses 4 motorized axes laser, range camera, white light, and color camera

Scanning a Large Object Calibrated motionsCalibrated motions – pitch (yellow) – pan (blue) – horizontal translation (orange) Uncalibrated motions – vertical translation – rolling the gantry – remounting the scan head

Single Scan of St. Matthew 1 mm

Postprocessing Manual initial alignmentManual initial alignment Pairwise ICP, then global registrationPairwise ICP, then global registration VRIP (parallelized across subvolumes)VRIP (parallelized across subvolumes) Use high-res geometry to discard bad color data, perform inverse lighting calculationsUse high-res geometry to discard bad color data, perform inverse lighting calculations

Statistics About the Scan of David 480 individually aimed scans480 individually aimed scans 0.3 mm sample spacing0.3 mm sample spacing 2 billion polygons2 billion polygons 7,000 color images7,000 color images 32 gigabytes32 gigabytes 30 nights of scanning30 nights of scanning 22 people22 people

Head of Michelangelo’s David Photograph 1.0 mm computer model

Side project: The Forma Urbis Romae

Forma Urbis Romae Fragment Forma Urbis Romae Fragment side face

Hard Problems Keeping scanner calibrated is hard in the lab, really hard in the museumKeeping scanner calibrated is hard in the lab, really hard in the museum Dealing with large data sets is painfulDealing with large data sets is painful Filling all the holes converges only asymptotically (if it converges at all…)Filling all the holes converges only asymptotically (if it converges at all…)

The Great Buddha Project Great Buddha of KamakuraGreat Buddha of Kamakura Original made of wood, completed 1243Original made of wood, completed 1243 Covered in bronze and gold leaf, 1267Covered in bronze and gold leaf, 1267 Approx. 15 m tallApprox. 15 m tall Goal: preservation of cultural heritageGoal: preservation of cultural heritage

Who? Institute of Industrial Science, University of TokyoInstitute of Industrial Science, University of Tokyo Daisuke Miyazaki Takeshi Ooishi Taku Nishikawa Ryusuke Sagawa Ko Nishino Takashi Tomomatsu Yutaka Takase Katsushi Ikeuchi

Scanner Cyrax range scanner by Cyra TechnologiesCyrax range scanner by Cyra Technologies Laser pulse time-of-flightLaser pulse time-of-flight Accuracy: 4 mmAccuracy: 4 mm Range: 100 mRange: 100 m

Processing 20 range images (a few million points)20 range images (a few million points) Simultaneous all-to-all ICPSimultaneous all-to-all ICP Variant of volumetric merging (parallelized)Variant of volumetric merging (parallelized)

Results