Generalized sampling theorem (GST) interpretation of DSR Δ hi Inter sample spacing needed to guarantee that band-limited image 𝑜 𝑥,𝑦 ⊛ℎ 𝑥,𝑦 is not aliased Δ lo =𝐺 Δ hi Inter sample spacing that we can get away with according to the GST ℳ 1 (𝜉,𝜂) Transfer function induced by sub-pixel shifting of the object 𝒟 𝜉,𝜂 Transfer function induced by low-resolution pixel geometry 𝒟 𝜉,𝜂 = Δ lo 2 sinc( Δ lo 𝜉)sinc( Δ lo 𝜂) Note The following model is incompatible with the matrix formulation of DSR that relies on a blur matrix and a down-sampling matrix. 𝑖 lo [𝑚,𝑛;1] Band-limited image ℳ 1 (𝜉,𝜂)𝒟(𝜉,𝜂) ⊗ 𝒢 1 (𝜉,𝜂) ⋮ ⋮ ⋮ 𝑖 lo [𝑚,𝑛;𝑘] ℋ(𝜉,𝜂) ℳ 𝑘 (𝜉,𝜂)𝒟(𝜉,𝜂) ⊗ 𝒢 𝑘 (𝜉,𝜂) ⊕ 𝑜(𝑥,𝑦) Reconstructed image Geometric image OTF 𝑖 lo [𝑚,𝑛;𝐹] ℳ 𝐹 (𝜉,𝜂)𝒟(𝜉,𝜂) ⊗ 𝒢 𝐹 (𝜉,𝜂) 𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ lo ,𝑦−ℓ Δ lo Observations The MTF’s of the bank of filters ℳ 𝑘 (𝜉,𝜂)𝒟(𝜉,𝜂) 𝑘=1 𝐹 is identical. The only thing that changes is the PTF, on account of shifting of the object by amounts smaller than Δ lo (sub-pixel shifting) Frequencies corresponding to nulls in 𝒟 𝜉,𝜂 cannot be recovered using any amount of sub-pixel shifting. This is because sub-pixel shifting does not affect |𝒟 𝜉,𝜂 |. Consequently frequencies that are not resolved by the filter 𝒟(𝜉,𝜂), can never be reconstructed in the super-resolved image. This fact seems to have been conveniently ignored in DSR literature. Proof: 𝒟 𝜉,𝜂 =0⇒sinc Δ lo 𝜉 =0 or sinc Δ lo 𝜂 =0. This occurs when either 𝜉 or 𝜂 is a harmonic of 1/Δ lo It is obvious that more the DSR gain 𝐺, more the number of zeros due to 𝒟 𝜉,𝜂 in the interval 𝜉 , 𝜂 ≤ 1/Δ hi Papoulis

⊗ Relationship between geometric image of scene & low-resolution image Inter sample spacing of the detector Δ hi = Δ hi ÷𝐺 Desired inter sample spacing ℋ 𝑜𝑝𝑡 (𝜉,𝜂) Optical Transfer Function ( ℋ 𝑜𝑝𝑡 𝜉,𝜂 ≝ℱ ℎ 𝑜𝑝𝑡 𝑥,𝑦 ) Δ lo 2 sinc − Δ lo 𝜉 sinc − Δ lo 𝜂 = Δ lo 2 sinc Δ lo 𝜉 sinc Δ lo 𝜂 Transfer function induced by low-resolution pixel geometry NOTE: sinc is an even function Assumption The detector geometry of the light sensitive element in the low-resolution imager is assumed to be square, i.e. ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo 𝑜(𝑥,𝑦) ⊗ 𝑚,𝑛∈ℤ 𝛿 𝑥−𝑚 Δ lo ,𝑦−𝑛 Δ lo ℋ 𝑜𝑝𝑡 (𝜉,𝜂) Δ lo 2 sinc Δ lo 𝜉 sinc Δ lo 𝜂 𝑖 lo [𝑚,𝑛] Geometric image of scene Aliased image Pixel transfer function of low resolution detector ×OTF 𝑖 lo [𝑚,𝑛]= 𝑜 𝑥,𝑦 ⊛ ℎ 𝑜𝑝𝑡 𝑥,𝑦 ℎ 𝑑𝑒𝑡 𝑥−𝑚 Δ lo ,𝑦−𝑛 Δ lo 𝑑𝑥𝑑𝑦

⊗ Equivalent model that is consistent with existing DSR models 𝑜 [𝑘,ℓ] Δ lo Inter sample spacing of the detector Δ hi = Δ hi ÷𝐺 Desired inter sample spacing ℋ(𝜉,𝜂) Discrete Time Fourier Transform of the sampled optical PSF ℎ 𝑘,𝓁 ≝ ℎ 𝑜𝑝𝑡 (𝑘 Δ hi ,𝓁 Δ hi ) 𝒟 𝜉,𝜂 Discrete Time Fourier Transform of 𝑑 𝑘,𝓁 ≝ ℎ 𝑑𝑒𝑡 𝑥,𝑦 rect 𝑥−𝑘 Δ hi Δ hi rect 𝑦−𝓁 Δ hi Δ hi 𝑑𝑥𝑑𝑦 In the absence of a focal plane mask, ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo . 𝑑 𝑘,𝓁 has exactly 𝐺 2 non-zero entries since 𝐺 is an integer. These entries are 1 in the absence of a focal plane mask, and correspond to a 𝐺×𝐺 code in the presence of a focal plane mask 𝑜(𝑥,𝑦) ⊗ 𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ hi ,𝑦−ℓ Δ hi Δ hi 2 sinc Δ hi 𝜉 sinc Δ hi 𝜂 circ 𝜆 2𝑁𝐴 𝜉 2 + 𝜂 2 𝑜 (𝑥,𝑦) Geometric image of scene Pixel transfer function of high resolution detector ×Ideal low pass filter 𝑜 [𝑘,ℓ] 𝑂𝑇𝐹×Pixel transfer function of low resolution detector ↓ Δ lo Δ hi ℋ(𝜉,𝜂) 𝒟 𝜉,𝜂 𝑖 lo [𝑚,𝑛] Subsampling Aliased image 𝑖 lo 𝑚,𝑛 = 𝑘,𝓁∈ℤ 𝑜 𝑘,𝓁 Γ 𝑑ℎ 𝑚𝐺−𝑘,𝑛𝐺−𝓁 = 𝑜 𝑘,𝓁 ⊛ Γ 𝑑ℎ 𝑘,𝓁 ↓𝐺 Observations The above model is compatible with the matrix formulation of DSR wherein one is interested in recovering an image of the scene as sampled by a detector with smaller pixels (not just smaller inter-sample spacing) Frequencies corresponding to nulls in either ℋ 𝜉,𝜂 or 𝒟 𝜉,𝜂 cannot be recovered. When not using focal plane masks, the detector integration pattern 𝑑[𝑘,ℓ] corresponds to a linear phase FIR filter with even or odd number of taps. The magnitude response of such a filter can be expressed as a trigonometric series. The problem of nulls in ℋ(𝜉,𝜂) 𝒟 𝜉,𝜂 may be remedied by diversity in either the sampled PSF or the detector integration geometry (focal plane mask).

⊗ ⊕ Filterbank interpretation of DSR Δ lo Inter sample spacing of the detector Δ hi = Δ hi ÷𝐺 Desired inter sample spacing ℋ(𝜉,𝜂) Discrete Time Fourier Transform of the sampled optical PSF ℎ 𝑘,𝓁 ≝ℎ(𝑘 Δ hi ,𝓁 Δ hi ) 𝒟 𝜉,𝜂 Discrete Time Fourier Transform of 𝑑 𝑘,𝓁 ≝ ℎ 𝑑𝑒𝑡 𝑥,𝑦 rect 𝑥−𝑘 Δ hi Δ hi rect 𝑦−𝓁 Δ hi Δ hi 𝑑𝑥𝑑𝑦 In the absence of a focal plane mask, ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo . 𝑑 𝑘,𝓁 has exactly 𝐺 2 non-zero entries since 𝐺 is an integer. These entries are 1 in the absence of a focal plane mask, and correspond to a 𝐺×𝐺 code in the presence of a focal plane mask Analysis filter-bank 𝑖 lo [𝑚,𝑛;1] ↓ Δ lo Δ hi Synthesis filter-bank ↑ Δ lo Δ hi ℋ 1 (𝜉,𝜂) 𝒢 1 (𝜉,𝜂) ⋮ ⋮ ⋮ ⋮ 𝑖 lo [𝑚,𝑛;𝑘] ⊗ ↓ Δ lo Δ hi ↑ Δ lo Δ hi 𝑜 (𝑥,𝑦) ℋ 𝑘 (𝜉,𝜂) 𝒢 𝑘 (𝜉,𝜂) ⊕ Reconstructed image Band-limited image 𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ hi ,𝑦−ℓ Δ hi 𝑖 lo [𝑚,𝑛;𝐹] ↓ Δ lo Δ hi ↑ Δ lo Δ hi ℋ 𝐹 (𝜉,𝜂) 𝒢 𝐹 (𝜉,𝜂) Samples from band-limited image Observations The above model is compatible with the matrix formulation of DSR Introduce noise before down-sampling !!!

⊗ Standard DSR workflow Δ lo Inter sample spacing of the detector Δ hi = Δ hi ÷𝐺 Desired inter sample spacing ℋ(𝜉,𝜂) Discrete Time Fourier Transform of the sampled optical PSF ℎ 𝑘,𝓁 ≝ℎ(𝑘 Δ hi ,𝓁 Δ hi ) 𝒟 𝜉,𝜂 Discrete Time Fourier Transform of 𝑑 𝑘,𝓁 ≝ ℎ 𝑑𝑒𝑡 𝑥,𝑦 rect 𝑥−𝑘 Δ hi Δ hi rect 𝑦−𝓁 Δ hi Δ hi 𝑑𝑥𝑑𝑦 In the absence of a focal plane mask, ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo . 𝑑 𝑘,𝓁 has exactly 𝐺 2 non-zero entries since 𝐺 is an integer. These entries are 1 in the absence of a focal plane mask, and correspond to a 𝐺×𝐺 code in the presence of a focal plane mask Image that can be recovered by Digital Super Resolution Pixel transfer function of high resolution detector ×Ideal low pass filter Analysis filter-bank ℋ 1 (𝜉,𝜂) 𝒟 𝜉,𝜂 ↓ Δ lo Δ hi 𝑖 lo [𝑚,𝑛;1] ⋮ ⋮ Δ hi 2 sinc Δ hi 𝜉 sinc Δ hi 𝜂 circ 𝜆 2𝑁𝐴 𝜉 2 + 𝜂 2 𝑜 (𝑥,𝑦) ⊗ ℋ 𝑘 (𝜉,𝜂) 𝒟 𝜉,𝜂 ↓ Δ lo Δ hi 𝑖 lo [𝑚,𝑛;𝑘] 𝑜(𝑥,𝑦) Geometric image of scene 𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ hi ,𝑦−ℓ Δ hi ℋ 𝐹 (𝜉,𝜂) 𝒟 𝜉,𝜂 ↓ Δ lo Δ hi 𝑖 lo [𝑚,𝑛;𝐹] Observations The above model is compatible with the matrix formulation of DSR wherein one is interested in recovering an image of the scene as sampled by a detector with smaller pixels (not just smaller inter-sample spacing) Any frequencies lost to blurring due to detector integration i.e. 𝒟 𝜉,𝜂 =0 cannot be recovered. This occurs when either 𝜉 or 𝜂 is a harmonic of 1/Δ lo . The optical PSF could either be diffraction limited or due to a pseudo-random phase mask as in Amit Ashok’s work Sub-pixel shift induces changes in the phase of the transfer function ℋ(𝜉,𝜂). These transfer function are denoted as ℋ 𝑘 (𝜉,𝜂) in the above block schematic. Introduce noise before down-sampling !!! In this scheme, all we are doing is changing the PTF of each filter in the filter-bank For perfect recovery, the filters ℋ 𝑖 (𝜉,𝜂) 𝒟 𝜉,𝜂 should have a broad response up-to the optical cutoff frequency.

⊗ DSR workflow in Focal Plane Coding Δ lo Inter sample spacing of the detector Δ hi = Δ hi ÷𝐺 Desired inter sample spacing ℋ(𝜉,𝜂) Discrete Time Fourier Transform of the sampled optical PSF ℎ 𝑘,𝓁 ≝ℎ(𝑘 Δ hi ,𝓁 Δ hi ) 𝒟 𝜉,𝜂 Discrete Time Fourier Transform of 𝑑 𝑘,𝓁 ≝ ℎ 𝑑𝑒𝑡 𝑥,𝑦 rect 𝑥−𝑘 Δ hi Δ hi rect 𝑦−𝓁 Δ hi Δ hi 𝑑𝑥𝑑𝑦 In the absence of a focal plane mask, ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo . 𝑑 𝑘,𝓁 has exactly 𝐺 2 non-zero entries since 𝐺 is an integer. These entries are 1 in the absence of a focal plane mask, and correspond to a 𝐺×𝐺 code in the presence of a focal plane mask Image that can be recovered by Digital Super Resolution Pixel transfer function of high resolution detector ×Ideal low pass filter Analysis filter-bank ℋ 𝜉,𝜂 𝒟 1 𝜉,𝜂 ↓ Δ lo Δ hi 𝑖 lo [𝑚,𝑛;1] ⋮ ⋮ Δ hi 2 sinc Δ hi 𝜉 sinc Δ hi 𝜂 circ 𝜆 2𝑁𝐴 𝜉 2 + 𝜂 2 𝑜 (𝑥,𝑦) ⊗ ℋ 𝜉,𝜂 𝒟 𝑘 𝜉,𝜂 ↓ Δ lo Δ hi 𝑖 lo [𝑚,𝑛;𝑘] 𝑜(𝑥,𝑦) Geometric image of scene 𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ hi ,𝑦−ℓ Δ hi ℋ 𝜉,𝜂 𝒟 𝐹 𝜉,𝜂 ↓ Δ lo Δ hi 𝑖 lo [𝑚,𝑛;𝐹] Observations The above model is compatible with the matrix formulation of DSR wherein one is interested in recovering an image of the scene as sampled by a detector with smaller pixels (not just smaller inter-sample spacing) Any frequencies lost to blurring due to detector integration i.e. 𝒟 𝜉,𝜂 =0 cannot be recovered. This occurs when either 𝜉 or 𝜂 is a harmonic of 1/Δ lo . Each low-resolution image utilizes a different focal plane mask. The transfer function due to these masks is denoted as 𝒟 𝑘 (𝜉,𝜂) in the above block schematic. In this scheme, we are changing the shape of the OTF of each filter in the filter-bank.

Papoulis Generalized sampling theorem (GST) 𝑓(𝑡) Band-limited signal 𝒴 𝑘 (𝜈) 𝒴 𝑀 (𝜈) 𝒴 1 (𝜈) ⋮ ℋ 𝑘 (𝜈) ℋ 𝑀 (𝜈) ℋ 1 (𝜈) ⊕ ⊗ 𝑛∈ℤ 𝛿 𝑡−𝑛𝑀 𝑇 0 𝑓 (𝑡) Reconstructed 𝑔 𝑘 (𝑡) 𝑔 𝑀 (𝑡) 𝑔 1 (𝑡) Pre-filtering Post-filtering ℱ 𝜈 =0, for 𝜈 ≥0.5/ 𝑇 0 Papoulis

Generalized sampling theorem (GST) interpretation of Digital Super Resolution 𝒪 𝑏 𝜈 =0, for 𝜈 ≥0.5/ Δ hi 𝑜(𝑥) Geometric image ℋ 𝑜𝑝𝑡 𝜈 Band-limited 𝑜 𝑏 (𝑥) 𝒴 𝑘 (𝜈) 𝒴 𝑀 (𝜈) 𝒴 1 (𝜈) ⋮ ℋ 𝑘 𝜈 ℋ 𝑀 𝜈 ℋ 1 𝜈 ⊕ ⊗ 𝑚∈ℤ 𝛿 𝑥−𝑚 Δ lo 𝑜 (𝑥) Reconstructed 𝑔 𝑘 (𝑥) 𝑔 𝑀 (𝑥) 𝑔 1 (𝑥) Pre-filtering Post-filtering ℋ 𝑑𝑒𝑡 −𝜈 𝑜 𝑏𝑑 (𝑥) Papoulis