Ali Ercan & Ulrich Barnhoefer Motion Compensated SNR and DR Enhancement With Motion Blur Prevention Using Multicapture Ali Ercan & Ulrich Barnhoefer

Introduction & Motivation Single exposure trade-off High noise if short exposure time Motion blur if long exposure time When presenting the example mention Simulated scene at Matlab Camera moving Short exposure: Shot and read noise Long exposure: Motion blur EE392J Project Ali Ercan & Ulrich Barnhoefer

Introduction & Motivation DR is another problem Short exposure: Dark areas in the scene cannot be seen Long exposure: Bright areas saturate If both high DR scene and motion, with single capture Motion blur free, but noisy and non-visible dark areas image Less noisy, but motion blurred and saturated image EE392J Project Ali Ercan & Ulrich Barnhoefer

Introduction & Motivation Our approach to solve these problems: Use of multicapture combined with motion estimation High speed is definitely needed Normal video mode can be used – poorer results due to noise adding CMOS imagers suitable For a better understanding, let us introduce a simple model of CMOS imagers and describe multicapture EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Sensor Model Charge Integration Light on photodiode generates charges Saturation when well capacity is reached Noise sources (Reset noise) Shot noise UT Read noise VT,Vo (Dark current) (Fixed pattern noise) - Charge integrates on intrinsic capacitor as light shines on photodiode - Maximum charge proportional to reset voltage - saturation CDS Equation: Signal + noise EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Multicapture Nondestructive multiple readout – Single integration Less noise per capture compared to conventional video sensor – readout noise cumulative read out noise in a conventional video sensor EE392J Project Ali Ercan & Ulrich Barnhoefer

Implemented Algorithm SCENE CAMERA SIMULATOR MOTION ESTIMATOR PHOTO-CURRENT FINAL IMAGE EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Camera Simulator Multicapture, noise, ADC implemented – pixel values out Synthetic scene Black background – to see the effect of read noise only, will be clear later why Camera moves EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Motion Estimator Block based motion estimation on difference frames Search range ±1 and block size 3x3 Fast imager (e.g. 10,000 fps available) Search range and block size can be increased in expense of computational load Noise suppression Known noise levels – characterized CMOS sensor Error = SSD + xDistance  is proportional to noise Thanks to Sebe! EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Motion Estimator Estimated and perfect motion vectors EE392J Project Ali Ercan & Ulrich Barnhoefer

Photocurrent Estimator B C EE392J Project Ali Ercan & Ulrich Barnhoefer

Photocurrent Estimator Useful when you have read noise When only shot noise is present, take the last Q_tilda EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Results EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Results EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Results Emphasize we get perfect result even with not perfect motion vectors Ambiguity means lighting did not change Symmetry of the scene EE392J Project Ali Ercan & Ulrich Barnhoefer

Results IMAGE ERRORS (STD OF ERROR IMAGE) CHECKER LENA CAMERAMAN 10 ms image 100.9 69.43 71.31 160 ms image 70.79 33.84 37.41 Const. with est. motion vectors 2.587 21.28 12.05 Const. with perfect motion vectors 2.576 17.22 3.092 EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Conclusion Promising results achieved with this preliminary analysis Motion blur reduced Noise reduced DR increased in dark end and in bright end in special cases EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Conclusion Lots of more things to do Use more sophisticated motion estimation algorithms Separate motion detection from motion estimation and do estimation when detection occurs Include extension of DR with sensor saturation Handle the occlusions EE392J Project Ali Ercan & Ulrich Barnhoefer

Ali Ercan & Ulrich Barnhoefer Questions EE392J Project Ali Ercan & Ulrich Barnhoefer