1. Differential Instant Radiosity for Mixed Reality Martin Knecht, Christoph Traxler, Oliver Mattausch, Werner Purgathofer, Michael Wimmer Institute of Computer Graphics and Algorithms Vienna University of Technology

2. Motivation 2 Martin Knecht

3. Motivation 3 Martin Knecht

4. Problem statement Virtual objects should seamlessly blend into the real scene Provide a plausible illusion Simulation of light interaction between real and virtual objects is necessary Solution should run at interactive to real-time frame rates 4 Martin Knecht

5. Related Work Image Based Lighting Ritschel and Grosch used a HDR camera to capture environment Hierarchical Importance Sampling, Clarberg Real-time Global Illumination Instant Radiosity introduced by Keller Imperfect Shadow Maps from Ritschel et. al Merging Real and Virtual Scenes Differential Rendering introduced by Fournier and extended by Debevec Differential Photon Mapping from Grosch 5 Martin Knecht

6. Assumptions Geometry of real scene known BRDFs of real surfaces are known Surrounding illumination is distant 6 Martin Knecht

7. Differential Instant Radiosity for MR Idea: Combine Differential Rendering (DR) with Instant Radiosity (IR) Why?  DR adds virtual objects to real scene  IR fast method to compute GI solution in real-time But:Two solutions are needed for DR!  not really How?Keep track on the light paths 7 Martin Knecht

8. GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 8 Martin Knecht

9. GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 9 Martin Knecht

10. GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 10 Martin Knecht

11. GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 11 Martin Knecht

12. GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 12 Martin Knecht

13. Differential Rendering Method to add virtual objects into real scene images Compute two global illumination solutions and add only difference to real scene image +Minimizes visual error from wrong BRDF approximations -We need to calculate two GI solutions in real-time 13 Martin Knecht

14. Differential Rendering Difference Buffer Grey means no difference GI(Real + Virtual) GI(Real) 14 Martin Knecht

15. Differential Rendering Masked Video Image Difference Buffer Final Image 15 Martin Knecht

16. How to get GI(Real+Virtual) and GI(Real)? Keep track on the light paths L:Lightsource D:Diffuse light bounce S:Specular light bounce E: Eye Heckberts classification Courtesy of Paul Guerrero 16 Martin Knecht

17. How to get GI(Real+Virtual) and GI(Real)? Keep track on the light paths L:Lightsource D:Diffuse light bounce S:Specular light bounce E: Eye Direct illumination: LDE Heckberts classification Courtesy of Paul Guerrero 17 Martin Knecht

18. How to get GI(Real+Virtual) and GI(Real)? Keep track on the light paths L:Lightsource D:Diffuse light bounce S:Specular light bounce E: Eye Direct illumination: LDE 1st bounce diffuse indirect illumination: LDDE Heckberts classification Courtesy of Paul Guerrero 18 Martin Knecht

19. Differential Instant Radiosity for MR Calculate shading result only once Decide on path information to which GI solution the result belongs L [x] D [x] E L [x] D [x] D [x] E [x] can be „real“ or „virtual“ All „real“ then add to GI(Real+Virtual) and GI(Real) One [x] „virtual“, only add to GI(Real+Virtual) 19 Martin Knecht

20. L R D R D R E path Differential Instant Radiosity for MR 20 Martin Knecht GI(Real + Virtual) GI(Real)

21. L R D R D R E path Added to both Buffers Differential Instant Radiosity for MR GI(Real + Virtual) GI(Real) 21 Martin Knecht

22. L R D R D V E path Differential Instant Radiosity for MR 22 Martin Knecht GI(Real + Virtual) GI(Real)

23. L R D R D V E path Add only to the GI(Real+Virtual) Buffer Differential Instant Radiosity for MR GI(Real + Virtual) GI(Real) 23 Martin Knecht

24. Differential Instant Radiosity for MR Take blocking geometry into account 24 Martin Knecht

25. L R D R D R E path Differential Instant Radiosity for MR 25 Martin Knecht GI(Real + Virtual) GI(Real)

26. L R D R D R E path Virtual red wall blocks light! Differential Instant Radiosity for MR GI(Real + Virtual) GI(Real) 26 Martin Knecht

27. Limitations 27 Martin Knecht

28. Limitations Double shadowing Incosistent color bleeding 28 Martin Knecht

29. Results Test device Core2 Quad CPU Q9550 at 2.8GHz 8GB RAM NVIDIA Geforce GTX 285 with 1GB VRAM Windows 7 64-Bit C# with SlimDX to access DirectX 10 29 Martin Knecht

30. Results 30 Martin Knecht

31. Results 31 Martin Knecht

32. Results 32 Martin Knecht

33. Results 33 Martin Knecht

34. Our method uses instant radiosity to simulate mutual influence between objects Direct Illumination Indirect Illumination Combination of Differential Rendering and Instant Radiosity Lightsources, objects, camera can be dynamic Cannot take all paths into account Works reasonable fast (20 – 30 fps) Conclusion 34 Martin Knecht

35. Consider all light paths Add camera artifacts Camera-based, real-time reconstruction of geometry Real-time BRDF estimation Future Work 35 Martin Knecht

36. Thank you for your attention! 36 Martin Knecht