1. Deep screen image crop and enhance
Week 3
(Aaron Ott, Amir Mazaheri)

2. Problem
We have taken a photo of an image, and we want the original image.
This can be broken into 2 parts:
Image Detector/Cropper
Image Enhancer

3. Cropper Uses a frozen VGG-19 model to get feature map
Applies convolutions, normalizations, and activations
Final dense layer creates 6-number affine transformation
STN takes input image and applies affine transformation

4. Enhancer Pretrained EDSR (trained on DIV2K) Modified form of Resnet
Pretrained EDSR (trained on DIV2K)
Modified form of Resnet
Uses modified residual block, which excludes batch normalization and final ReLU layer
16 Residual blocks
Subpixel Conv2D layers for upscaling the image
Scales the image 4x
Lim, Son, Kim, Nah, Lee. “Enhanced Deep Residual Networks for Single Image Super-Resolution”. 10 July 2017

5. Combined Cropper and Enhancer
Trained with 2 outputs and 2 Loss Functions:
- Trained Cropper on VGG + Cosine Proximity
- Trained Enhancer on VGG + MSE

6. Results Cropper & Enhancer Metric\Model Cropper Cropper & Enhancer
PSNR
SSIM
0.4254
0.4909
MSE
0.0796
0.0281
Input
Cropper
Actual

7. Shortcomings of PSNR, SSIM, and MSE
Metrics:
PSNR:
SSIM:
MSE:
Input
Output
Loss Functions
VGG + Cosine Proximity
MSE
Actual

8. Building the GAN: Discriminator
Used Discriminator from
Skips Batch Normalization in first Discrimination Block
Pairs of each level of number of feature maps
Final Dense layers, with a single value output
Discrimination Block

9. Only trained on 15 epochs, starting with existing weights
Current GAN Output
Input
Output
Actual
Metric\Model
Cropper
Cropper & Enhancer
GAN
PSNR
SSIM
0.4254
0.4909
0.4899
MSE
0.0796
0.0281
0.0277

10. What’s Next Short Term (next week)
Optimize GAN and get it training properly
Try new enhancers
Synthetically create dataset
Long Term (to the end of the summer)
Develop network to work on harder datasets
Connect model to solve existing issues: identification/classification