Slow moving point target Tracking & Detecting in real and compressed infrared imagery sequences Students Liav Viner Omri Ravid Supervisors Dr. Ofer Hadar Mrs. Revital Huber-Shalem

Motivation Where is the target..? Infrared imagery sequences are used for detection of moving targets in the presence of evolving cloud clutter or background noise This research concentrates on slow moving point targets, which are one pixel in size, such as long-range aircraft

Point target IR movies compression and detection system Motivation Infrared sequences: - Captured by ground sensors - Contain enormous amount of data Transmission and storage are very time and resource consuming =>Compression which retains the point target detection capabilities Point target IR movies compression and detection system

Compression IR compression method containing point targets cannot use a video compression standard MPEG used for video streams watched by human viewers Exploit the human visual system characteristics Different measures of quality than IR movies containing point targets A point target cannot be detected in IR movies by a human viewer, Target size is only one pixel Target intensity is not always noticeable within the background

Compression methods Spatial 2-D DCT Can we use a lossy spatial 2-D DCT compression? DCT Transform ZigZag Vector Compression IDCT Transform Most of the data is in the low-frequency coefficients and the DC itself, and most of the AC coefficients are negligible Low-frequency coefficients represent slow spatial changes, while high-frequency coefficients represent fast spatial changes

Background Temporal profile Temporal profile is the temporal behavior of a pixel along the movie frames Target pixel Clear sky pixel The intensity of the point target is proportional to the height of the peak The velocity of the point target is inversely proportional to the width of the peak Cloud edge pixel

Compression methods Spatial 2-D DCT Main drawback of a standard compression for IR imagery containing point targets: lossy spatial compression (as spatial DCT), reduces high spatial frequencies might diminish the point target Each frame has 78080 DCT coefficients 64,000 protected coefficients 16,000 protected coefficients 1,000 protected coefficients 100 protected coefficients Compression rate: 1.22 Compression rate: 4.88 Compression rate: 78.08 Compression rate: 780.8 Temporal profile of a Target’s pixel

Compression methods Temporal 1-D DCT The 1-D DCT-based lossy temporal compression method is more suitable for IR movies containing a slow moving point target 4th Coefficient 1st frame 2nd frame 3rd frame 4th frame 129 172 121 108 170 162 100 163 124 130 524 382 101 148 201 408 123 667 623 429 362 167 182 140 132 149 141 152 120 138 154 136 3rd Coefficient 129 172 121 108 170 162 100 163 124 130 524 382 101 148 201 408 123 667 623 429 362 167 182 140 132 149 141 152 120 138 154 136 2nd Coefficient 1st Coefficient (DC) 129 172 121 108 170 162 100 163 124 130 524 382 101 148 201 408 123 667 623 429 362 167 182 140 132 149 141 152 120 138 154 136 129 172 121 108 170 162 100 163 124 130 524 382 101 148 201 408 123 667 623 429 362 167 182 140 132 149 141 152 120 138 154 136 DCT coefficients of pixel (x,y) Target Noise Temporal profile This compression is supposed to reduce the temporal noise and compress the temporal profile Temporal profile after Temporal DCT compression

Compression methods Temporal 1-D DCT How many coefficients should we protect? Movie score Compression Rate Number of Protected Coefficients Number of Protected Coefficients

Compression methods Temporal 1-D DCT Number of Protected Coefficients How does the elimination of coefficients affects the temporal profile of a target pixel? Compression rate 1 2 5 9 13 95 47.5 19 10.55 7.3

Compression methods Temporal 1-D DCT First two coefficients (including DC) do not contain target information but more background value Target appears in coefficients range [3,13] Coefficients larger than 13 contain mainly fast changes and noise information 1/95 coefficient 2/95 coefficient 5/95 coefficient 9/95 coefficient 13/95 coefficient

Compression methods Quantization After choosing the number of protected coefficients, we can now use Lloyd-Max quantization for further compression Lloyd Max Quantization 129 172 121 108 170 162 100 163 124 130 524 382 101 148 201 408 123 667 623 429 362 167 182 140 132 149 141 152 120 138 154 136 129 172 121 108 170 162 100 163 124 130 524 382 101 148 201 408 123 667 623 429 362 167 182 140 132 149 141 152 120 138 154 136 129 172 121 108 170 162 100 163 124 130 524 382 101 148 201 408 123 667 623 429 362 167 182 140 132 149 141 152 120 138 154 136 129 172 121 108 170 162 100 163 124 130 524 382 101 148 201 408 123 667 623 429 362 167 182 140 132 149 141 152 120 138 154 136 Code book 3700 11 = 3350 10 = 2000 01 = 750 00 = 250 3000 As the number of levels decrease, the target becomes less noticeable 1000 500 40 Levels 10 Levels 4 Levels

Spatial 2-D DCT Compression summery Temporal 1-D DCT Spatial 2-D DCT compression performance becomes poor above compression rate of 3 Temporal 1-D DCT compression performance becomes poor above compression rate of 12 Further compression on the Temporal 1-D DCT’s output with Max-Lloyd Quantization improves the performance even better Spatial 2-D DCT Temporal 1-D DCT

Point Target Tracking & Detecting VERS - Variance Estimation Ratio Score SNR based Evaluates the final score of each sub-temporal profile Temporal process compares the overall estimated variance of a temporal profile and its highest fluctuation If this fluctuation is high enough, it may be considered as a target

VERS background Scores The ratio between the maximum variance and the average of the K lowest variances is the score of the pixel Short-Term Variance Window The variance is calculated over shorter time sections after subtracting the DC from the partial temporal profile Pixel Score(i)= Block Score(i)= Pixel profile Movie Score(i)= Variance profile

VERS background Parameters The VERS algorithm has another important parameter: Long-term DC window The long-term window for DC estimation should be short enough in order to track DC changes caused by a clutter entrance or departure, but long enough to perform an accurate estimation to suppress noise and not to suppress the target Long-Term DC Window Edge cloud Point target Pixel Profile DC Profile Result

VERS background Parameters The long-term DC window’s size has a great effect on the movie’s score Too small – Might erase target Too big - Might suspect cloud as a target Temporal profiles of a cloud Window Size - 20 Window Size - 30 Pixel Profile DC Profile Result Mistakenly suspect as target

VERS background Movies scores VERS shortcomings: Dependency of the performance and the chosen parameters on the unknown target velocity and scene's type Movie’s score VS Set of Parameters   5/15 5/20 5/25 5/30 7/30 7/35 7/37 7/50 10/30 10/40 10/50 30/35 35/50 j2a 53.189 78.538 115.699 104.669 93.203 98.532 96.868 74.086 84.321 78.420 59.288 37.326 13.755 j13c 3.215 4.655 5.481 5.959 7.298 10.519 11.694 8.991 8.450 11.656 8.252 2.796 1.017 m21f 6.357 7.337 7.553 6.173 5.934 5.307 6.193 6.530 5.945 6.743 6.922 9.707 11.169 na23a 19.738 24.580 23.095 17.658 18.092 13.810 11.421 5.718 16.289 7.940 4.504 1.866 0.289 npa 30.183 36.422 40.568 36.051 38.674 38.848 46.336 59.552 34.442 45.023 58.046 21.304 15.881 But what distinguishes between each movie?

Scene’s Type Determining the scene's type automatically will allow us to select the optimal long-term DC window parameter for the given IR sequence - Rapidly evolving clutter - Shorter DC window - Slow evolving clutter, or Clear Sky - Longer DC window Two different methods for determining the scene's type: Spatial 2-D DCT Tail Ratio Hot hazy night Wispy clouds Bright clouds Fluffy clouds Bright clouds Cloudy

Determining Scene’s Type Spatial 2-D DCT We are interested in knowing the scene's type, particularly knowing if the scene has many spatial changes, implicating it is composed from a large amount of small clouds First approach for determining the scene's type: Spatial 2-D DCT on the first frame of the IR sequence

Determining Scene’s Type Spatial 2-D DCT 1’st Frame DCT Vector Applying spatial 2-D DCT on a frame of cloudy sky should result in many large coefficients for all frequencies Applying spatial 2-D DCT on a frame which is mainly composed of homogenous scene, i.e. clear sky or very big clouds, should result in more negligible coefficients

Determining Scene’s Type Spatial 2-D DCT Summing up all DCT coefficients yields a value which implies on the scene’s type 633.7 816.1 1564.7 1845.5 1894.2 2054.7 Homogenous Scene clutter Scene

Determining Scene’s Type Tail Ratio Second approach for determining the scene's type: The Tail Ratio method 1st Frame Histogram Assumption of work: many small clouds result in a large variety of histogram values, each having a small amount of occurrences, and therefore visualized as a low bin in the histogram graph This phenomenon results in a descending tale of higher intensities than the sky's

Determining Scene’s Type Tail Ratio We have generated a MATLAB function that counts all the histogram bins within a certain range 31 42 159 225 237 415 Homogenous Scene clutter Scene The scene might change for longer movies Thus, both of the techniques should be applied periodically in case of longer movies

Determining Scene’s Type Automatic Method Spatial DCT Tail Ratio Clutter Level 1 2 3 4 5 Automatic Method categorizes the scene into one of five different levels of scene’s type Optimal Long-Term DC Window for the selected scene’s type Higher movie’s score

Path Tracking Algorithm PTA The resulting output of the VERS algorithm on an infrared sequence is an image representing each pixel’s score It is not necessarily true that the pixel with the highest value is the target So, we need to track the pixels through which the target traversed Cloud VERS 129 172 121 108 170 162 100 163 124 130 524 382 101 148 201 408 123 667 623 429 362 167 182 140 132 149 141 152 120 138 154 136 Target PTA

PTA The Algorithm Potential_Targets= all pixels that received high score (according to PS threshold) Declare each pixel from Potential_Targets as a Path Group (PG) For each PG Search for neighboring PGs (according to DBP threshold) If found a neighboring PG Combine both PGs to one PG Else go to 9 Go to 3 If PG has less pixels than PGS threshold Erase PG If there is no PGs left Declare the pixel with the highest score as target If there are neighboring pixels within PG target Combine pixels to the PG target Mark each PG as a different target's track Verify that the resulting path represents a real and logical movement of an aircraft

PTA Algorithm’s Steps Original IR sequence

PTA Algorithm’s Steps All scored pixels (result of VERS) are colored in yellow

PTA Algorithm’s Steps Pixels Score (PS) threshold filtering low-scored pixels

PTA Algorithm’s Steps Initializing each pixel as a group

PTA Algorithm’s Steps Join neighboring groups into same group using the Distance Between Pixels (DBP) threshold

PTA Algorithm’s Steps Delete all groups that contain less pixels than the Paths Group's Size (PGS) threshold

PTA Algorithm’s Steps Verify that the resulting path represents a real and logical movement of an aircraft

PTA Algorithm’s Steps Verify that the resulting path represents a real and logical movement of an aircraft

Conclusions Compression of IR imagery containing point target - Lossy temporal compression yields better point-target detection results than lossy spatial compression - Further compression using Max-Lloyd Quantization improves the performance even better VERS improvement - Finding an optimal set of VERS parameters - Determining the scene’s type  Matching optimal DC window  Yields better results Path Tracking Algorithm (PTA) - The target’s track can be detected even if not all target pixels achieved high pixel scores

Any Questions..?