Single Image Super-Resolution Adam Vest

Problem Overview Challenging problem → difficult to infer the values for multiple pixels from a single pixel Significant advances in the last two years due to deep learning methods First deep learning systems used ConvNets, recent state-of-the-art uses GANs Goal is to find some mapping f(ILR) = ISR where ISR is as close as possible to IHR

The First Wave: ConvNets Convolutional networks made great progress in SISR Several variants over the years: SRCNN DRCN ESPCN General take-aways: learning upscaling leads to dramatic increases in performance, deeper networks performed better Suffers from blurry image outputs/cannot capture image textures well due to reliance on L1/L2 losses SRResNet is the current state-of-the-art ConvNet for SISR

SRGAN First model to use adversarial learning for super-resolution 16 block deep ResNet structure + 2 PixelShuffle blocks (Transpose Conv.) Utilized a content loss (MSE or VGG) and an adversarial loss (BCE) to produce realistic images Pretty good PSNR, very good Mean-Opinion-Score

Project Goal To use adversarial learning to improve on the current state-of-the-art in SISR Build on the SRGAN baseline by exploring different choices for network architectures, the merits of different loss functions, etc. Implement an existing/design a new adversarial model that has not yet been used for SISR

Where Are We Now? Downloaded ImageNet and other datasets Setup on the cluster Modular implementations of all SRResNet and SRGAN variants Trained SRResNet on ImageNet, currently training SRGAN Implemented evaluation suite for BSD100, Urban100, Set5, and Set14 datasets Tested SRResNet implementation on evaluation suite

SRResNet Implementation Results LR SR HR

SRResNet Implementation Results Cont. PSNR Results for 4x Upscaling Dataset Ours (RGB) Theirs (Y-Channel) Set5 29.90 32.05 Set14 26.42 28.49 BSD100 26.15 27.58 Urban100 24.31 Not Reported

Where Are We Going? SRGAN + Wasserstein discriminator SRGAN + PatchGAN discriminator? Do super-resolution on Y-channel of the image only Investigate other adversarial models/network structures Investigate different evaluation metrics → SSIM Could lead to different choices for loss functions

Thank you! Questions?