Computer Science Department

Name: Computer Science Department
Uploaded: 2017-09-30T21:42:10+00:00
Duration: PTM36S6
Description: Computer Science Department

Computer Science Department
0:45, 1:30 for intro (through list of problems), 32:15 for entire talk (yeah, right) I have given many talks about the Digital Michelangelo Project in past talks, I stressed positive results but… not everything went well on the project, and to date, we have not produced a complete archive of 3D models of the statues we scanned 3 years ago so… this will be an anti-Digital Michelangelo Project talk What went wrong during the scanning? What continues to go wrong as we process our data? Why is 3D scanning so hard? Why is 3D scanning hard? Marc Levoy Computer Science Department Stanford University 34:15 total + 30% = ~45 minutes Time =

The Digital Michelangelo Project
0:15 This is not intended to be a talk on the Digital Michelangelo Project so here is the extent of my executive summary of that project, as well as a proof that we did obtain some useful data Atlas Awakening Bearded Youthful Time =

Dusk Dawn Night Day

St. Matthew David Forma Urbis Romae Time =

Eight hard problems in 3D scanning
0:30 I’ll organize my talk around these eight problems not an exhaustive list – there are other hard problems my choices are clearly biased by our experiences in the DMP on the other hand, these problems are not unique to our project - so I hope you will find them interesting By necessity… I”ll be interleaving consideration of these problems with a description of our scanning pipeline, so let’s start off with that… optically uncooperative materials scanning in the presence of occlusions insuring safety for delicate objects scanning large objects at high resolution accurate scanning in the field filling holes in dense polygon models handling large datasets creating digital archives of 3D content Some games with range data Time =

Scanners used in the Digital Michelangelo Project
0:30, 2:30 through St. Matthew’s face Cyberware triangulation laser scanner custom design Faro + 3D Scanners also a triangulation laser scanner on an articulated arm off-the-shelf for tight spots but not too much - it was tiring to use Cyra time-of-flight prototype Cyberware - for statues Faro + 3D Scanners - for tight spots Cyra - for architecture 4:30 through Matthew Time = 0:45

Our triangulation-based scanner
0:30 truss extensions for tall statues 4 motorized axes laser, range camera, white light, and color camera Time = 0:30

Scanning St. Matthew working in the museum scanning geometry scanning
0:30 Let’s cut to the chase here we are in the Galleria dell’Accademia scanning Michelangelo’s unfinished statue of the apostle St. Matthew you can see the David in the background In the middle image we’re acquiring geometry you can see the laser stripe sweeping across the face of St. Matthew In the right image we’re acquiring color you can see our white spotlight on the statue’s neck working in the museum scanning geometry scanning color Time = 0:15

single scan of St. Matthew
0:30 Here’s the range image acquired during one sweep of the laser stripe the texture you see… are Michelangelo’s chisel marks here’s a plot through his cheek each valley is the groove left by one tooth of a multi-tooth chisel possibly this two-tooth chisel called a Dente di cane ("dog-tooth") The spacing between our samples points (in X and Y) is ¼ mm our depth resolution is 50 microns, but how good is that 50 microns?… 1 mm single scan of St. Matthew Time = 0:30

Hard problem #1: optically uncooperative materials
0:45, 3:00 for entire problem dirty -> dark -> dropouts in range data shiny -> false returns -> outliers in range data fuzzy – like hair scattering – i.e. translucent transparent – esp. objects that change ray paths, like lenses dark shiny fuzzy scattering transparent moving etc. Time =

How optically cooperative is marble. [Godin et al
How optically cooperative is marble? [Godin et al., Optical Measurement 2001] 2:00 including next slide When a laser beam strikes a block of marble, it scatters beneath the surface this gives marble its distinctive glow, which you can see here So how does this subsurface scattering affect range scanning? working with the National Research Council of Canada we've formed the following hypothesis Normally, when the laser beam strikes an object, it's reflected from the surface towards the camera this is the basis for laser triangulation When the beam strikes a marble surface it refracts, then forms a volume of scattered light beneath the surface the camera sees a refracted view of this volume Ignoring the refraction for a moment, the centroid of this volume is clearly displaced horizontally from where the laser struck the surface this displacement causes a systematic bias in the computed depth we’ve observed 40 microns More importantly, the shape of the volume varies randomly across the surface with the marble crystal structure giving rise to noise in the range data – almost ¼ mm this noise fundamentally limits the accuracy with which we can scan marble objects systematic bias of 40 microns noise of 150 – 250 microns worse at oblique angles of incidence worse for polished statues Time =

Laser scattering noise on David’s toe
In this case, the noise is about 0.1 mm twice this bad, or worse, on the highly polished statue of Night Fortunately, the noise changes randomly also with direction of incidence of the laser, so we can reduce the noise to some extent by taking multiple scans from different directions and averaging them 50 micron scan by NRC/Canada 0.1 mm Time =

Some marbles scatter more than others
0:15 In addition to the level of polish some marbles are naturally more translucent than others, and hence scatter more Michelangelo’s Pieta is famous for its extraordinary translucency this statue may be unscannable Of course, this statue may be unscannable for another reason… Michelangelo’s Pietà Time =

Hard problem #2: scanning in the presence of occlusions
1:30 + 1:15 movie excerpt (skip to 2/3 point), 3:00 for entire problem fig 4 in the DMP, we could roll our scanner– this helped us scan grooves of different orientation < 30° ? 20° ? ? occlusions (& self-occlusion) force grazing scans, which lead to holes a scanner with a fixed triangulation angle cannot circumvent all occlusions a hammer & chisel can reach places a triangulation scanner cannot Time =

Some statues may be “unscannable” (using optical methods)
0:10 Laocoon Time =

Hard problem #3: insuring safety for delicate objects
0:45, 2:45 for entire problem Although it may not be immediately apparent, the problem of reaching all surfaces on an object is intimately related to the problem of insuring safety for the object during scanning There are actually several safety issues… not a problem for marble statues we also scanned some old violins – and we had to “run a calculation” in general energy deposition is not a problem for rangefinders that using scanning (i.e. moving) laser beams the spotlight we used for color acquisition deposits more energy on the object than the moving laser beam avoiding collisions… yes, collisions - between the scanner and the object being scanned energy deposition not a problem for marble statues avoiding collisions Time =

Why are collisions hard to avoid?
1:30 fig1 circumscribe it with a circle think of this circle as a conservative convex hull fig 2 to scan this part of the object reasonably perpendicularly requires a standoff equal in size to the diameter of the circle since most triangulation scanners have a fixed standoff, this is a large and cumbersome scanner you could move the scanner inside the circle – allowing a smaller standoff, but this introduces the possibility of collision fig 3 of course, sometimes we can’t avoid moving inside the circle this greatly increases the chances of a collision summarizing cannot completely scan an arbitrary statue from outside its convex hull eventually something – a scan head, or a mirror – must get close to the statue to insure safety, scan head should stay outside the convex hull of the object in the worst case, required standoff may equal diameter of convex hull a complicated object cannot be scanned entirely from outside its convex hull Time =

Hard problem #3: insuring safety for delicate objects
0:30 How do you avoid colliding with a statue? we used a variety of techniques, which I won’t describe in detail we were lucky in 5 months of around-the-clock scanning, we didn't damage anything but there's no silver bullet for this problem it will always be with us and there will be accidents energy deposition not a problem for marble statues avoiding collisions manual motion controls automatic cutoff switches one person serves as spotter avoid time pressure get enough sleep surviving collisions pad the scan head Time =

Hard problem #4: scanning large objects at high resolution
1:15, 2:30 for entire problem So far we’ve been talking mainly about problems at the scale of individual range samples let’s switch scales There are many geometric configurations of scanners that we could have built for the DMP can digitize a tall statue, and can capture chisel marks the ratio between these two scales what one might call the geometric dynamic range of a scanning system is what makes the design problem hard Why is it hard to provide a 20,000:1 dynamic range? scanning at this resolution, using today’s triangulation technology, implies a small working volume given a 14cm stripe, and overlapping the stripes, it would take 30 stripes to get around the David for each stripe, the scanner must be repositioned, and not every one will capture useful geometry David is 5 meters tall chisel marks need 1/4mm dynamic range of 20,000:1 20,0002 = 1 billion polygons 14cm wide working stripe David was ~30 stripes around Time =

Digital Michelangelo Project reconfigurable robotic gantry
0:30 It’s surprisingly difficult to design a system with this much flexibilty that is also field-deployable – you can carry it into a museum in pieces and assemble it there Our Siggraph 2000 paper talks at length about the design of our gantry… calibrated motions pitch (yellow) pan (blue) horizontal translation (orange) uncalibrated motions vertical translation remounting the scan head moving the entire gantry Time =

Scanning St. Matthew 104 scans 800 million polygons 4,000 color images
0:15 we made 104 uncalibrated moves to scan St. Matthew – a relatively simple statue 104 scans 800 million polygons 4,000 color images 15 gigabytes 1 week of scanning Time = 0:15

Scanning the David 480 individually aimed scans 2 billion polygons
0:30 rolled the gantry (or worse - remounted the scan head) 480 times and the gantry, with its extra trusses and counterweights, weighed 800 kilograms – about a ton 480 individually aimed scans 2 billion polygons 7,000 color images 32 gigabytes 30 nights of scanning 22 people 4:00 through architectural reps Time = 0:45

Hard problem #5: accurate scanning in the field
1:15 in the field i.e. using a field-deployable gantry One of the fundamental design choices we made in the DMP rotating scan head instead of translating (as the primary scanning motion) because it reduced the chances of colliding with the statue during scanning It is fundamentally hard to make a rotating scanner accurate because rotational errors are magnified by the lever arm of the standoff 1/100 degree of rotational error –> 1/4mm error at 1m standoff – equal to our range sample spacing Also, remounting the scan head…not repeatable For this reason, and others, it is essential to have a way to recalibrate a large reconfigurable scanner in the field We didn’t have a way to do this, and the quality of our data suffered as a result rotating scan motion is hard to make accurate field reconfigurations are not repeatable need a way to recalibrate in the field Time =

Digital Michelangelo Project range processing pipeline
1:15 up to (but not including) What really happens?, 3:00 for entire range processing pipeline steps 1. manual initial alignment - should have tracked gantry 2. ICP to one existing scan [Besl92] 3. automatic ICP of all overlapping pairs [Rusinkiewicz01] 4. global relaxation to spread out error [Pulli99] 5. merging using volumetric method [Curless96] Time =

steps 1. manual initial alignment - should have tracked gantry 2. ICP to one existing scan [Besl92] 3. automatic ICP of all overlapping pairs [Rusinkiewicz01] 4. global relaxation to spread out error [Pulli99] 5. merging using volumetric method [Curless96]

steps 1. manual initial alignment 2. ICP to one existing scan 3. automatic ICP of all overlapping pairs 4. global relaxation to spread out error 5. merging using volumetric method

What really happens? steps ICP is unstable on smooth surfaces
1. manual initial alignment 2. ICP to one existing scan 3. automatic ICP of all overlapping pairs 4. global relaxation to spread out error 5. merging using volumetric method ICP is unstable on smooth surfaces

What really happens? steps ICP is unstable on smooth surfaces
1:45 unstable on smooth surfaces Rusinkiewicz01 reduces this problem by distributing point pairs more uniformly around the Gaussian sphere of directions but this doesn’t solve the problem in all situations distributes errors unevenly for example if one part of the statue was overscanned (scanned multiple times), hence sampled more densely a pair of undulating overlapped surfaces will dominate over a smaller but reliable “lock and key” suggesting that perhaps sum-of-squares is not a good error metric for these algorithms perhaps maximum error – the L-infinity norm – is better there are many such sources of bias in global registration algorithms still an open problem steps 1. manual initial alignment 2. ICP to one existing scan 3. automatic ICP of all overlapping pairs 4. global relaxation to spread out error 5. merging using volumetric method ICP is unstable on smooth surfaces relaxation distributes errors unevenly Time =

Digital Michelangelo Project color processing pipeline
0:45, 2:30 for entire color pipeline discard specular pixels we haven’t yet characterized specularity, although we have the data we need to do it correct for irradiance converts color to diffuse reflectance sometimes called de-shading steps 1. compensate for ambient illumination 2. discard shadowed or specular pixels 3. map onto vertices – one color per vertex 4. correct for irradiance  diffuse reflectance Time =

What really happens? steps we treated reflectance as Lambertian
1:00 treated reflectance as Lambertian Oren and Nayar (and others) have shown the error in this approximation used aggregate surface normals caused fine-scale geometry to pollute diffuse reflectance made reflectance less useful for scientific applications ignored interreflections Nayar’s work on Shape from Interreflections suggests that this problem can be addressed, althought it would be expensive steps 1. compensate for ambient illumination 2. discard shadowed or specular pixels 3. map onto vertices – one color per vertex 4. correct for irradiance  diffuse reflectance we treated reflectance as Lambertian we used aggregate surface normals we ignored interreflections we ignored subsurface scattering – derived reflectance is incorrect – renderings are not realistic Time =

The importance of subsurface scattering
0:15 (no time for movie) BRDF BSSRDF [Jensen et al., Siggraph 2000] Time =

artificial surface reflectance
0:30 through David’s heads artificial surface reflectance Time = 0:15 for 2 slides

estimated diffuse reflectance
Time =

with diffuse reflectance
photograph 1.0 mm computer model with diffuse reflectance Time =

Hard problem #6: filling holes in dense polygon models
0:15 (defer to James’s talk) signed distance function volumetric diffusion process Time =

Hard problem #7: handling large datasets
0:30, 2:15 for entire problem large datasets one of the implications of the 20,000:1 dynamic range It’s hard to convey just how much data we acquired of the David I’ll try to do it with a sequence of zooms 3mm mesh 1mm 0.3mm Time =

Some solutions we’ve used
1:00 range images keep our data in the form of regular arrays as long as possible eventually, we need to convert the data to polygons the output of our merging algorithm is a polygon mesh Schroeder’s semi-regular meshes may point toward an alternative approach (skip over list of requirements) range images instead of polygon meshes z(u,v) yields 18:1 lossless compression multiresolution using (range) image pyramid multiresolution viewer for polygon meshes 2 billion polygons immediate launching real-time frame rate when moving progressive refinement when idle compact representation fast pre-processing Time =

The Qsplat viewer [Rusinkiewicz and Levoy, Siggraph 2000]
0:45 including brief demo and streaming Qsplat, but skipping details hierarchy of bounding spheres with position, radius, normal vector, normal cone, color traversed recursively subject to time limit spheres displayed as splats Time =

Streaming Qsplat [Rusinkiewicz and Levoy, I3D 2001]
other aspects of handling large datasets structure of interactive programs that manipulate the data 1 second 10 seconds 60 seconds Time =

Hard problem #8: creating digital libraries of 3D content
2:00 for entire problem metadata since 3D scanning is not (yet) standardized secure viewers so far we’ve been allowed to distribute our data only to scholars we’d like to give it to artists, art schools, public schools but the Italian authorities don’t want the models appearing in video games, and they don’t want unauthorized physical replicas secure viewers would allow downloading and viewing, but not extracting and storing this is an unsolved problem across the computer graphics industry watermarking may be unsolvable legal action may be the only effective solution to this problem longevity very unlikely that our digital data will last longer than the statue of course, this is a problem for all digital archives, not just archives of 3D content metadata – data about data versioning – cvs for data archives secure viewers for 3D models robust 3D digital watermarking viewing, measuring, extracting data indexing and searching 3D content insuring longevity for the archive Time =

How to think about range data
1:30 for this slide, 23:00 from beginning through this slide noisy topology Schroeder’s group has observed that the genus of David’s head is over 300! geometric signal processing as opposed to combinatoral geometry not unorganized points at least in a sweeping scanner the same assumptions that permit us to identify an observation as a range sample allow us to decide if it is connected to its neighbor if one of these assumptions is counterindicated, then both are connectivity can be used in alignment, merging, compression, hole filling, etc. lines of sight like connectivity, it’s another aspect of the structure of a range scan James Davis will show tomorrow how it can be used to advantage during hole filling range data is measured data both the geometry and the topology are noisy “geometric signal processing” range data is not a cloud of unorganized points use connectivity between range samples use lines of sight to scanner and camera range datasets are large representations that are exact only in the limit algorithms that are fast and approximate implementations that don’t need entire model in memory Time =

Games with range data volumetric scan conversion range image synthesis
almost nothing for this slide, 2:00 for all games …I didn’t want to make this entirely a negative talk… volumetric scan conversion range image synthesis multi-modality scanned models Time =

Game #1: volumetric scan conversion
1:00 volumetric scan conversion Cyra time-of-flight scan of a tree (actually a movie set), represented as a cloud of points the best representation for range data like this may not be points or a mesh – it might be a volume? simply by counting the number of range samples that fall in a voxel volume texture synthesis generating more “tree-like vegetation” from acquired data [Wei01] convert point cloud to volume densities facilitates volumetric / statistical analyses new texture synthesis algorithms volume texture synthesis Time =

Game #2: range image synthesis
0:30 in range images or semi-regular meshes, or merged surface meshes (Hertzmann) texture synthesis of range data in range images in merged surface meshes application to hole filling Time =

Game #3: multi-modality scanned models
0:30 to anybody considering doing this, sensor fusion is hard time-of-flight scans for overall shape triangulation scans for range texture photography for reflectance Time =

Three final questions Can range scanning be fully automated?
0:15 for this slide, 3:30 for entire final questions Can range scanning be fully automated? Can we build a 3D fax machine? Will IBR replace 3D scanning? Time =

The Forma Urbis Romae: a marble map of ancient Rome
1:15 for entire automatic 3D scanning Time =

Time =

Can we replace this? uncrating... Time = 0:15 for sequence

Can we replace this? positioning... Time =

Can we replace this? scanning... Time =

Can we replace this? aligning... Time =

Automatic 3D scanning? requires 6 DOF robot
to allow access to any point from the laser plane at any orientation a system like this has never been built a “grand challenge” for our field requires 6 DOF robot automatic view planning [Connolly85, Maver93, Pito96, Reed97,…] hard to guarantee safety use CT scanning instead? Time =

Can we build a 3D fax machine?
1:00 rapid protyping is improving new technologies come on the market clearly catering to graphics community – all over Siggraph color printing some companies can do spatially-varying primary colors, but subtle color mixtures are still elusive how to match BRDF a holy grail for rapid prototyping Is there a technology that can “print” (i.e. match) a specified reflectance characteristic? quality and cost of rapid prototyping is improving need automatic scanning need color printing how to match BRDF? looking for a killer app ? Time = [Curless96]

Will IBR replace 3D scanning?
…assuming the reason for scanning a particular object is to fly around it Yes simple, brute force universal meta-primitive geometry is implicit independent of scene complexity multiresolution is trivial hardware acceleration 1:00 fly around it as opposed to making a physical replica, for example Will IBR replace 3D scanning? a special version of the general question: Will image-based rendering replace model-based rendering? specific versions of this question have been posed for volume rendering, and for point rendering – Will they take over the world? You can see my Yes’s and No’s I suspect the answer is: Maybe, for some applications, but not for all. No discards structure of model rasterized geometry big and slow 0:45 Time = Total conclusions = 1:15

Computer Science Department

Similar presentations

Presentation on theme: "Computer Science Department"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Computer Science Department

Similar presentations

Presentation on theme: "Computer Science Department"— Presentation transcript:

Similar presentations

About project

Feedback