Deep screen image crop and enhance Week 3 (Aaron Ott, Amir Mazaheri)

Problem We have taken a photo of an image, and we want the original image. This can be broken into 2 parts: Image Detector/Cropper Image Enhancer

Cropper Uses a frozen VGG-19 model to get feature map Applies convolutions, normalizations, and activations Final dense layer creates 6-number affine transformation STN takes input image and applies affine transformation

Enhancer Pretrained EDSR (trained on DIV2K) Modified form of Resnet https://github.com/krasserm/super-resolution Pretrained EDSR (trained on DIV2K) Modified form of Resnet Uses modified residual block, which excludes batch normalization and final ReLU layer 16 Residual blocks Subpixel Conv2D layers for upscaling the image Scales the image 4x Lim, Son, Kim, Nah, Lee. “Enhanced Deep Residual Networks for Single Image Super-Resolution”. 10 July 2017

Combined Cropper and Enhancer Trained with 2 outputs and 2 Loss Functions: - Trained Cropper on VGG + Cosine Proximity - Trained Enhancer on VGG + MSE

Results Cropper & Enhancer Metric\Model Cropper Cropper & Enhancer PSNR 11.1903 16.2060 SSIM 0.4254 0.4909 MSE 0.0796 0.0281 Input Cropper Actual

Shortcomings of PSNR, SSIM, and MSE Metrics: PSNR: 18.3130 SSIM: 0.5358 MSE: 0.0164 Input Output Loss Functions VGG + Cosine Proximity MSE Actual

Building the GAN: Discriminator Used Discriminator from https://github.com/krasserm/super-resolution Skips Batch Normalization in first Discrimination Block Pairs of each level of number of feature maps Final Dense layers, with a single value output Discrimination Block

Only trained on 15 epochs, starting with existing weights Current GAN Output Input Output Actual Metric\Model Cropper Cropper & Enhancer GAN PSNR 11.1903 16.2060 16.2404 SSIM 0.4254 0.4909 0.4899 MSE 0.0796 0.0281 0.0277

What’s Next Short Term (next week) Optimize GAN and get it training properly Try new enhancers Synthetically create dataset Long Term (to the end of the summer) Develop network to work on harder datasets Connect model to solve existing issues: identification/classification