1. ETISEO François BREMOND ORION Team, INRIA Sophia Antipolis, France

2. 2 Fair Evaluation Unbiased and transparent evaluation protocol Large participation Meaningful evaluation

3. Tasks evaluated GT & Metrics are designed to evaluate tasks all along the video processing chain: Task 1: Detection of physical objects, Task 2: Localisation of physical objects, Task 3: Classification of physical objects, Task 4: Tracking of physical objects, Task 5: Event recognition.

4. Matching Computation To evaluate the matching between a candidate result and a reference data, we may use following distances: D1-The Dice coefficient: Twice the shared, divided by the sum of both intervals: 2*card(RD  C) / (card(RD) + card(C)). D2-The overlapping: card(RD  C) / card(RD). D3-Bertozzi and al. metric: (card(RD  C))^2 / (card(RD) * card(C)). D4-The maximum deviation of the candidate object or target according to the shared frame span: Max { card(C\RD) / card(C), card(RD\C) / card(RD) }. RD C

5. metrics (1) T1- DETECTION OF PHYSICAL OBJECTS OF INTEREST C1.1 Number of physical objects C1.2 Number of physical objects using their bounding box T2- LOCALISATION OF PHYSICAL OBJECTS OF INTEREST C2.1 Physical objects area (pixel comparison based on BB) C2.2 Physical object area fragmentation (splitting) C2.3 Physical object area integration (merge) C2.4 Physical objects localisation 2D and 3D Centroïd or bottom centre point of BB

6. metrics (2) T4- CLASSIFICATION OF PHYSICAL OBJECTS OF INTEREST C4.1 Object Type over the sequence C4.2 Object classification per type C4.3 Time Percentage Good Classification card{ RD  C, Type(C) = Type(RD) } / card(RD  C) T5- EVENT RECOGNITION C5.1 Number of Events recognized over the sequence C5.2 Scenario parameters T3- TRACKING OF PHYSICAL OBJECTS OF INTEREST C3.1 Frame-To-Frame Tracking: Link between two frames C3.2 Number of object being tracked during time C3.3 Detection time evaluation C3.4 Physical object ID fragmentation C3.5 Physical object ID confusion criterion C3.6 Physical object 2D trajectory C3.7 Physical object 3D trajectory

7. 7 Metric Evaluation Distance for matching groundtruth and algorithms results  Similar measures: D1, D2, D3, D4. Few main metrics measure general trends  Discriminant and meaningful  Detection M1.2.1: CNumberObjectsBoundingBox  Localization M2.4.3: CCentroid2DLocalisationPix.  Tracking M3.3.1: CtrackingTime  Object Classification M4.1.3: CobjectTypeOverSequenceBBoxID  Event Recognition M5.1.2: CNumberNamedEvents

8. 8 Metric Evaluation (cont’d) Secondary metrics:  Complementary information  Pixel-based (M2.1.1) versus object-based (M1.2.1) metrics  Potential algorithm errors.  Example: M3.3.1 complemented (eg., about stability) by M3.2.1, M3.4.1 and M3.5.1. Non-informative Metrics:  Add noise to the evaluation or non-discriminative  Example: M1.1.1 CNumberObjects gives the object number per frame without position information.  The same for M4.1.1 and M5.1.1.

9. 9 Evaluation on ETI-VS2-BE-19-C1

10. 10 Global Results: Video Remarks:  For similar scenes, very dissimilar results!  For different scenes, results can spread over a large range or concentrate in a narrow range.

11. 11 Detection of Physical Objects ( ETI-VS2-BE-19-C1.xml ) Group Precision (global) Sensitivity (global)F-Score 120,9377750,8353440,883600841 170,7558750,9439190,839495579 80,955650,7306810,828159238 10,9858190,7026410,820483574 140,9988070,545810,705882233 130,9893740,5464620,704053421 30,5645330,8999020,693816217 190,9216220,5559180,69351254 2810,5288560,691832324 2910,5167920,681427645 320,9993110,4727750,641877252 910,452560,62312056 2010,4388650,610015533 1510,3625690,532184425 230,5489130,0987940,167450131 Group Precision (global) Sensitivity (global)F-Score 10,9634030,6866640,801827026 80,8545840,6534070,74057626 140,9952210,5432020,702805454 130,980310,5357030,692810692 290,9798110,5063580,667669879 120,7064420,6292790,665631693 280,9451290,4998370,653870671 190,8367570,5047280,629652493 90,9711820,4395170,605162404 200,9762260,4284320,595513581 320,90,4254970,577816925 150,9982010,3619170,531227307 170,4292430,5360290,476729245 30,2781750,443430,341879949 230,3540490,0612980,104502961 M1.1.1: NumberObjectsM1.2.1: NumberObjectsBoundingBoxD1

12. 12 Detection of Physical Objects ( ETI-VS2-BE-19-C1.xml ) Group Precision (global) Sensitivity (global)F-Score 10,9634030,6866640,801827026 80,8545840,6534070,74057626 140,9952210,5432020,702805454 130,980310,5357030,692810692 290,9798110,5063580,667669879 120,7064420,6292790,665631693 280,9451290,4998370,653870671 190,8367570,5047280,629652493 90,9711820,4395170,605162404 200,9762260,4284320,595513581 320,90,4254970,577816925 150,9982010,3619170,531227307 170,4292430,5360290,476729245 30,2781750,443430,341879949 230,3540490,0612980,104502961 M1.2.1: NumberObjectsBoundingBoxD1 Group Precision (global) Sensitivity (global) Specificity (global)F-Score 10,8483270,9726450,9928440,90624 80,7746010,9362180,9887890,84778 120,8607010,8152010,9945710,83733 190,8755450,7674140,9955110,81792 280,7280530,9143770,9859450,81065 140,7492450,8625540,9881340,80192 130,7858160,7952390,9911680,7905 30,7683210,7811950,9903070,7747 290,89180,6299870,9968550,73837 200,6325580,7863170,9812040,70111 150,6741770,6964510,9861490,68513 90,8697520,5538470,9965870,67675 170,4958790,8664460,9637530,63076 320,7659140,4252880,9946570,5469 230,2705340,0867960,9910390,13143 M2.1.1: ObjectsArea

13. 13 Detection of Physical Objects ( ETI-VS2-BE-19-C1.xml ) GroupSplitting value 151 130,999734 140,998935 290,998915 90,998647 200,996937 320,996753 280,992038 230,989362 120,972667 10,969471 190,928911 80,919553 170,822301 30,352762 M2.2.1: SplittingD5M2.3.1: MergingD2 Group Merging v alue 190,862026 150,859442 280,841472 140,834221 30,822119 170,820084 290,814503 130,805644 120,799294 80,786615 10,783842 200,776757 230,760934 90,747107 320,700614

14. 14 Summary on Detection of Physical Objects Main metric measures:  Detection M1.2.1: CNumberObjectsBoundingBox  Problems: static objects, contextual objects, background, masks…  Advantages: objects vs pixels, large objects and bounding boxes Secondary metrics:  M2.1.1 (area): indication on the precision and handling shadows Split/Merge measures (M2.2.1, M2.3.1):  Advantage: indicate potential merge  Inconvenients: threshold-dependent, non-detected objects not taken into account

15. 15 Localisation of Physical Objects

16. 16 Localisation of Physical Objects ( ETI-VS2-BE-19-C1.xml ) Groupvalue 200,032379 150,022457 130,021595 230,021043 280,020659 170,020328 320,019422 30,019145 140,018123 290,017538 80,015945 120,012152 90,012141 10,010686 190,009591 M2.3.1: MergingD2 M2.4.3: Centroid2DLocalisationPixD1 GroupValue (pixel) 2019,583632 1514,954423 1313,282747 2313,255976 2813,135057 1713,06773 312,369682 1412,31259 3211,936302 2911,463824 89,851749 97,693683 127,628216 16,658153 195,964608

17. 17 Localisation of Physical Objects ( ETI-VS2-BE-19-C1.xml ) Groupvalue 200,032379 150,022457 130,021595 230,021043 280,020659 170,020328 320,019422 30,019145 140,018123 290,017538 80,015945 120,012152 90,012141 10,010686 190,009591 M2.4.1: Centroid2DLocalisationD1M2.4.3: Centroid2DLocalisationPixD1 GroupValue (pixel) 2019,583632 1514,954423 1313,282747 2313,255976 2813,135057 1713,06773 312,369682 1412,31259 3211,936302 2911,463824 89,851749 97,693683 127,628216 16,658153 195,964608

18. 18 Localisation of Physical Objects ( ETI-VS2-BE-19-C1.xml ) Groupvalue 99,544993 238,881964 18,441473 88,107458 157,347973 146,725589 296,613178 286,344898 196,032267 3210,068101 131,429808 201,279278 171,265509 121,214232 31,100734 M2.4.2.: Centroid3DLocalisationD1

19. 19 Summary on Localisation of Physical Objects M2.4.1, M2.4.2, M2.4.3, main metrics:  Problems: low utilisation of 3D info and calibration  Good performance: good precision on reliable TP (handling shadow and merge)  Advantages: complementary to the Detection; normalised, pixel or meter metrics

20. 20 Tracking of Physical Objects

21. 21 Tracking of Physical Objects ( ETI-VS2-BE-19-C1.xml ) GroupPrecisionSensitivityF-Score 110,8333330,909091 120,7510,857143 320,833333 910,6666670,8 1410,6666670,8 130,80,6666670,727273 170,54545510,705882 280,666667 200,666667 190,50,6666670,571429 290,60,50,545455 80,37510,545455 150,40,6666670,5 30,02409610,047059 M3.2.1: NumberObjectTrackedD1M3.3.1: TrackingTime GroupTracking Time 10,473258 120,365271 140,332216 190,319806 90,312904 320,297917 290,297554 130,296646 170,29289 80,26263 280,246822 200,20839 150,12739 30,110475 230,018877

22. 22 Tracking of Physical Objects ( ETI-VS2-BE-19-C1.xml ) GroupPersistence 90,875 140,833333 200,777778 10,766667 290,75 320,7 190,666667 120,666667 130,625 280,611111 80,59 170,5625 150,520833 230,4375 30,113126 M3.4.1: PhysicalObjectIdFragmentationM3.5.1: PhysicalObjectIdConfusion GroupConfusion 30,985106 150,875 140,875 170,791667 200,766667 190,761905 280,75 320,75 80,733333 130,666667 230,638889 290,633333 90,583333 120,569444 10,5625

23. 23 Tracking of Physical Objects ( ETI-VS2-BE-19-C1.xml ) GroupPrecisionSensitivityF-Score 90,750,50,6 1 0,50,545454545 120,3750,50,428571429 140,50,3333330,39999976 80,250,6666670,363636413 130,40,3333330,363636165 290,40,3333330,363636165 200,333333 280,333333 320,333333 190,250,3333330,285714163 170,1818180,3333330,235293882 150,10,1666670,125000094 30,0080320,3333330,015686029 23000 M3.6.1: PhysicalObject2DTrajectories Groupvalue 200,032379 150,022457 130,021595 230,021043 280,020659 170,020328 320,019422 30,019145 140,018123 290,017538 80,015945 120,012152 90,012141 10,010686 190,009591 M2.4.1: Centroid2DLocalisationD1

24. 24 Summary on Tracking of Physical Objects M3.3.1, main metric:  Problems: propagation of detection errors  Advantages: good global overview M3.2.1, secondary metric:  Good performance: consistent TP over time for few TPs  Problems: not taking into account of complete FN Fragmentation/confusion (M3.4.1, M3.5.1):  Advantage: indicate potential ID switching  Inconvenients: not discriminative; favoring under- detection (few IDs); over-detection (multiple IDs)

25. 25 Object Classification

26. 26 Object Classification ( ETI-VS2-BE-19-C1.xml ) Group Precision (global) Sensitivity (global)F-Score 10,9025620,64330,751190125 2810,5288560,691832324 120,726940,6475380,684945519 2910,5167920,681427645 80,782090,5979780,677753001 190,8708110,5252690,655277668 200,9925710,4356050,605484045 130,8288080,4577760,5897919 90,9438040,4271270,588102787 320,8421780,3984350,540947405 170,397650,4965760,441641031 150,700540,2539940,372816383 30,2159950,344310,265459842 2810,994720,997353012 2910,9736010,986623943 10,8814680,989440,932338413 30,1871470,9102430,310462546 M4.1.1: ObjectTypeOverSequence Group Precision (global) Sensitivity (global)F-Score 10,8316560,5927620,692176136 290,9684540,5004890,659931085 280,9451290,4998370,653870671 80,7351810,5621130,637102765 190,8351350,503750,62843225 200,9762260,4284320,595513581 120,6259150,5575480,589756767 90,9077810,4108250,565656655 130,7845340,4333220,558285778 320,7463820,3531140,479415902 170,3300260,4121290,366536061 150,6348920,2301920,337879464 30,2157910,3439840,265208883 M4.1.1b: ObjectTypeOverSequenceBounding BoxD1 Subtype

27. 27 Object Classification ( ETI-VS2-BE-19-C1.xml ) Group Precision (global) Sensitivity (global)F-Score 10,8316560,5927620,692176136 290,9678230,5001630,659501188 280,943280,4988590,652591349 80,7313430,5591780,633776461 190,8345950,5034240,628025691 200,974740,427780,594607246 120,6251830,5568960,589067079 90,9077810,4108250,565656655 130,7833530,432670,557445612 320,7463820,3531140,479415902 170,3300260,4121290,366536061 150,6339930,2298660,337400976 30,213950,341050,262946477 M4.1.3: ObjectTypeOverSequenceBounding BoxIdD1 Group Precision (global) Sensitivity (global)F-Score 10,8316560,5927620,692176136 290,9684540,5004890,659931085 280,9451290,4998370,653870671 80,7351810,5621130,637102765 190,8351350,503750,62843225 200,9762260,4284320,595513581 120,6259150,5575480,589756767 90,9077810,4108250,565656655 130,7845340,4333220,558285778 320,7463820,3531140,479415902 170,3300260,4121290,366536061 150,6348920,2301920,337879464 30,2157910,3439840,265208883 M4.1.2: ObjectTypeOverSequenceBounding BoxD1

28. 28 Object Classification

29. 29 Object Classification Group13891213171920282932 Precision (mean by frame) 0,670,170,610,500,480,630,240,63 0,770,660,48 Sensitivit y (mean by frame) 0,530,380,520,310,480,430,380,430,360,480,400,30 F-Score0,5920,2350,5610,3840,4820,5130,2930,5110,4620,5890,4980,367

30. 30 Summary on Object Classification M4.1.2, M4.1.3, same main metrics:  Problems: low classification of subtypes (doors, bikes, bags), favoring a few good quality TPs.  Advantage: reliable. M4.1.1 (without BBox):  Inconvenients: wrong evaluation result in case of double errors (classified noise and FN)  Advantage: indicate potential double errors.

31. 31 Event Recognition

32. 32 Event Recognition ( ETI-VS2-BE-19-C1.xml )

33. 33 Event Recognition ( ETI-VS2-BE-19-C1.xml ) BE-19-C1 1812141519202829MeanVarMinMax Precision 0,330,250,290,400,040,13 1,000,000,290,29940,001,00 Sensitivity 0,600,25 0,100,050,15 0,350,050,220,1750,050,60 F-Score 0,430,250,270,160,040,14 0,520,000,250,22090,000,75

34. 34 Event Recognition ( ETI-VS2-BE-19-C1.xml ) Group Precision SensitivityF-Score 120,880,750,8108 80,65 10,50,90,6428 190,560,650,6046 2810,350,5185 200,420,50,4545 1410,250,4 150,040,050,0454 290,010,750,0282 M5.1.1: NumberEvents GroupPrecisionSensitivityF-Score 2810,350,51 10,380,70,50 120,470,40,43 80,25 140,40,10,16 190,130,150,14 200,1250,150,14 150,040,050,045 290,000,050,002 M5.1.2: NumberNamedEventsD1

35. 35 Summary on Event Recognition M5.1.2 (with time), main metrics:  Problems: lack of understanding of ground truth definition  Advantages: good global overview per scenario type. M5.1.1, secondary metric:  Problems: not taking into account of occurrence time

36. 36 Evaluation Results

37. 37 Evaluation on ETI-VS2-BE-19-C4

38. 38 Detection of Physical Objects (ETI-VS2-BE-19-C4.xml) Group Precision (global) Sensitivity (global)F-Score 10,9551850,5440440,6932 90,9226360,2417420,3831 170,2973980,440440,3550 1410,1649150,2831 130,8185540,150150,2537 80,3107610,1676680,2178 120,9885060,1076080,1940 290,4461940,0850850,1429 190,2783020,0590590,0974 230,6147190,0355360,0671 M1.2.1: NumberObjectsBoundingBoxD1 M2.1.1: ObjectsArea Group Precision (global) Sensitivity (global) Specificity (global)F-Score 10,7584930,5573980,9843180,64258 80,7739120,3081280,9920460,44077 170,5677640,3443770,9768340,42872 90,9280780,226650,9984480,36433 120,7437210,219560,9933150,33903 140,9012950,1964420,9981010,32258 130,6810460,1932480,9924450,30107 290,4983420,1009030,9910250,16783 190,5723260,0835590,9944830,14583 230,6095940,0491880,9972170,09103

39. 39 Tracking of Physical Objects (ETI-VS2-BE-19-C4.xml) M3.2.1: NumberObjectTrackedD1 GroupPrecisionSensitivityF-Score 80,510,666667 1410,50,666667 90,750,50,6 1210,3333330,5 130,3750,50,428571 290,40,3333330,363636 10,1904760,6666670,296296 190,250,3333330,285714 170,10169510,184615 M3.3.1.D1: TrackingTime GroupTracking Time 90,203304 80,146138 120,124236 10,118673 140,110668 290,079072 170,064894 130,062049 190,020582 230,007754 200

40. 40 Event Recognition (ETI-VS2-BE-19-C4.xml) M5.1.2: NumberNamedEventsD1 GroupPrecisionSensitivityF-Score 10,2641510,3783780,3111 120,750,1621620,2666 140,4444440,1081080,1739 80,2666670,1081080,1538 190,750,0810810,1463 200,1250,0810810,0983 290,018450,1351350,0324 M5.1.1: NumberEvents Group Precision SensitivityF-Score 80,8666670,3513510,4999 200,6250,4054050,4918 10,3584910,5135140,4222 1410,2432430,3913 1210,2162160,3555 1910,1081080,1951 290,1107010,8108110,1948

41. 41 Evaluation Results

42. 42 Evaluation on ETI-VS2-MO-1-C1

43. 43 Detection of Physical Objects ( ETI-VS2-MO-1-C1.xml ) Group Precision (global) Sensitivity (global)F-Score 1410,7755910,873614475 120,9981730,7169730,834521481 80,6778750,7760280,723638345 10,510550,7515310,608034115 2810,3118990,475492397 190,3401850,7095360,459881253 30,2714050,4978130,351289068 2610,1692910,289561794 170,1481080,539370,232400199 290,1790360,2712160,215690004 200,0412950,5892390,077181007 230,2448980,0314960,055813856 M1.2.1: NumberObjectsBoundingBoxD1 M2.1.1: ObjectsArea Group Precision (global) Sensitivity (global) Specificity (global)F-Score 140,755820,7590660,9825880,75744 120,6533090,8196780,9691150,7271 80,8674170,5708670,9938050,68857 10,3086290,8609680,8710950,45438 190,2608670,8126890,8484560,39496 30,2191720,87020,7956190,35015 200,2075530,7554440,8082180,32564 170,1893480,7256610,7923590,30033 290,3569850,2492360,9703280,29353 260,8594020,0240070,9997210,04671 280,999990,02386810,04662 230,2322020,0238040,9948240,04318

44. 44 Tracking of Physical Objects ( ETI-VS2-MO-1-C1.xml ) M3.2.1: NumberObjectTrackedD1 GroupPrecisionSensitivityF-Score 14111 8111 12111 2810,3333330,5 2610,3333330,5 10,09090910,166667 290,06818210,12766 190,05454510,103448 170,03409110,065934 30,02608710,050847 200,01807210,035503 M3.3.1.D1: TrackingTime GroupTracking Time 80,776399 140,758954 120,647392 190,393109 290,276964 170,254244 280,244262 10,202911 260,13258 30,103148 200,011822 230,002927

45. 45 Event Recognition ( ETI-VS2-MO-1-C1.xml ) M5.1.2: NumberNamedEventsD1 GroupPrecisionSensitivityF-Score 80,4285710,50,461538213 190,2857140,3333330,307692 120,2857140,3333330,307692 280,50,1666670,250000375 140,50,1666670,250000375 10,0769230,1666670,105263152 M5.1.1: NumberEvents Group Precision SensitivityF-Score 80,5714290,6666670,615385006 120,5714290,6666670,615385006 2810,3333330,499999625 1410,3333330,499999625 190,4285710,50,461538213 10,0769230,1666670,105263152

46. 46 Evaluation Results

47. 47 Evaluation on ETI-VS2-RD-6-C7

48. 48 Detection of Physical Objects ( ETI-VS2-RD-6-C7.xml ) Group Precision (global) Sensitivity (global)F-Score 10,9997670,5369120,69863244 140,992960,4235490,593808038 80,9996980,414790,586310712 280,9808940,4175430,585747407 120,9983960,3893890,560266064 190,9137990,355480,511845337 30,6720360,3915170,494782147 320,9344260,3280780,485645373 90,9948150,3120620,475092849 130,9749410,3115620,472217426 200,6544480,3663660,46975746 290,9056760,2715220,41779031 150,9947590,2374870,383433715 1710,160410,276471247 230,900850,039790,076213687 M1.2.1: NumberObjectsBoundingBoxD1M2.1.1: ObjectsArea Group Precision (global) Sensitivity (global) Specificity (global)F-Score 10,9120610,7762230,9935090,83868 80,8046230,4643090,9902220,58883 280,8104960,4595060,9906820,5865 120,77810,4439260,989020,56532 30,8193430,4304110,9917690,56436 140,7392550,4191080,987190,53494 90,8728150,3774560,9952290,527 130,9161270,3642490,9971080,52125 320,7961210,3763270,9916410,51107 200,5516440,4516940,9681590,49669 150,6387760,4045340,9801590,49536 190,802820,3576110,9923820,49481 170,7890870,3556830,9917540,49034 290,8921790,3167290,996680,46749 230,5830140,0378020,9976550,071

49. 49 Detection of Physical Objects: Reference Data Filtering ( ETI-VS2-RD-6-C7.xml: M1.2.1 - NumberObjectsBoundingBoxD1 ) Group Precision (global) Sensitivity (global)F-Score 10,9997670,5369120,69863244 140,992960,4235490,593808038 80,9996980,414790,586310712 280,9808940,4175430,585747407 120,9983960,3893890,560266064 190,9137990,355480,511845337 30,6720360,3915170,494782147 320,9344260,3280780,485645373 90,9948150,3120620,475092849 130,9749410,3115620,472217426 200,6544480,3663660,46975746 290,9056760,2715220,41779031 150,9947590,2374870,383433715 1710,160410,276471247 230,900850,039790,076213687 No filteringWith filtering Group Precision (global) Sensitivity (global)F-Score 80,8897570,7013970,7844 140,992960,4235490,5938 280,9808940,4175430,5857 120,9983960,3893890,5602 10,5214350,5299550,5256 320,9344260,3280780,4856 90,9948150,3120620,4750 130,9749410,3115620,4722 290,9056760,2715220,4177 30,3100490,59910,4086 150,9947590,2374870,3834 190,3860670,3293870,3554 1710,160410,2764 230,900850,039790,0762

50. 50 Detection of Physical Objects: Reference Data Filtering ( ETI-VS2-RD-6-C7.xml: M2.1.1 - ObjectsArea ) Group Precision (global) Sensitivity (global) Specificity (global)F-Score 10,9120610,7762230,9935090,83868 80,8046230,4643090,9902220,58883 280,8104960,4595060,9906820,5865 120,77810,4439260,989020,56532 30,8193430,4304110,9917690,56436 140,7392550,4191080,987190,53494 90,8728150,3774560,9952290,527 130,9161270,3642490,9971080,52125 320,7961210,3763270,9916410,51107 200,5516440,4516940,9681590,49669 150,6387760,4045340,9801590,49536 190,802820,3576110,9923820,49481 170,7890870,3556830,9917540,49034 290,8921790,3167290,996680,46749 230,5830140,0378020,9976550,071 No filteringWith filtering Group Precision (global) Sensitivity (global) Specificity (global)F-Score 280,8104960,4595060,9906820,5865 120,77810,4439260,989020,56532 140,7392550,4191080,987190,53494 90,8728150,3774560,9952290,527 130,9161270,3642490,9971080,52125 320,7961210,3763270,9916410,51107 150,6387760,4045340,9801590,49536 170,7890870,3556830,9917540,49034 290,8921790,3167290,996680,46749 230,5830140,0378020,9976550,071

51. 51 Tracking of Physical Objects ( ETI-VS2-RD-6-C7.xml ) M3.2.1: NumberObjectTrackedD1 GroupPrecisionSensitivityF-Score 2810,7058820,827586 1410,7058820,827586 910,6470590,785714 1210,6470590,785714 150,7857140,6470590,709677 1710,5294120,692308 80,7692310,5882350,666667 130,7692310,5882350,666667 290,7142860,5882350,645161 320,6111110,6470590,628571 190,4583330,6470590,536585 10,50,4117650,451613 30,0597830,6470590,109453 M3.3.1.D1: TrackingTime GroupTracking Time 140,603553 280,584835 120,503574 320,42068 90,419863 290,387679 130,352851 150,322041 170,265919 230,030452

52. 52 Tracking of Physical Objects: Reference Data Filtering ( ETI-VS2-RD-6-C7.xml: M3.2.1 - NumberObjectTrackedD1 ) GroupPrecisionSensitivityF-Score 2810,7058820,827586 1410,7058820,827586 910,6470590,785714 1210,6470590,785714 150,7857140,6470590,709677 1710,5294120,692308 80,7692310,5882350,666667 130,7692310,5882350,666667 290,7142860,5882350,645161 320,6111110,6470590,628571 190,4583330,6470590,536585 10,50,4117650,451613 30,0597830,6470590,109453 No filteringWith filtering GroupPrecisionSensitivityF-Score 10,9285710,7647060,83871 2810,7058820,827586 1410,7058820,827586 1210,6470590,785714 910,6470590,785714 80,8461540,6470590,733333 150,7857140,6470590,709677 200,650,7647060,702703 1710,5294120,692308 130,7692310,5882350,666667 190,647059 290,7142860,5882350,645161 320,6111110,6470590,628571 30,3076920,9411760,463768

53. 53 Tracking of Physical Objects: Reference Data Filtering ( ETI-VS2-RD-6-C7.xml: M3.3.1.D1 - TrackingTime) GroupTracking Time 140,603553 280,584835 120,503574 320,42068 90,419863 290,387679 130,352851 150,322041 170,265919 230,030452 No filteringWith filtering GroupTracking Time 140,603553 280,584835 10,534872 120,503574 80,48195 200,443102 190,42611 320,42068 90,419863 290,387679 30,380634 130,352851 150,322041 170,265919 230,030452

54. 54 Event Recognition ( ETI-VS2-RD-6-C7.xml ) M5.1.2: NumberNamedEventsD1 GroupPrecisionSensitivityF-Score 110,250,4 2810,250,4 1210,250,4 190,50,250,33 80,50,1250,2 150,0170,1250,03 M5.1.1: NumberEvents Group Precision SensitivityF-Score 1910,50,67 110,250,4 810,250,4 2810,250,4 1210,250,4 1410,250,4 150,0701750,50,12

55. 55 Event Recognition ( ETI-VS2-RD-6-C7 )

56. 56 Evaluation Results

57. 57 Evaluation on ETI-VS2-RD-10-C4

58. 58 Detection of Physical Objects ( ETI-VS2-RD-10-C4.xml ) Group Precision (global) Sensitivity (global)F-Score 140,9955650,4085530,579354538 10,9721480,3620560,527613493 130,9725860,3583260,523705325 190,7837050,3903550,521137191 170,5885790,422020,491574026 200,5599490,3620560,439765284 290,8488740,2708830,410706137 120,9318960,2477710,391460987 230,7965620,0252960,049034827 M1.2.1: NumberObjectsBoundingBoxD1 M2.1.1: ObjectsArea Group Precision (global) Sensitivity (global) Specificity (global)F-Score 10,6883310,8387450,9836690,75613 130,7554030,6994740,990260,72636 140,8834340,6051140,9965670,71825 190,6830340,562480,9887750,61692 200,4892390,574190,9742220,52832 120,4732460,5790050,9722860,52081 290,8249360,3749420,9965780,51556 170,3160570,7556320,9296830,44569 230,4849390,0482110,9978970,0877

59. 59 Tracking of Physical Objects (ETI-VS2-RD-10-C4.xml) M3.2.1: NumberObjectTrackedD1 GroupPrecisionSensitivityF-Score 1410,5789470,733333 290,8461540,5789470,6875 200,4571430,8421050,592593 120,50,5263160,512821 190,3589740,7368420,482759 130,3333330,5789470,423077 170,2459020,7894740,375 10,2413790,3684210,291667 M3.3.1.D1: TrackingTime GroupTracking Time 140,481446 170,320325 190,286615 130,27612 290,271027 10,27099 200,270541 120,21312 230,010886

60. 60 Event Recognition (ETI-VS2-RD-10-C4.xml) M5.1.2: NumberNamedEventsD1 GroupPrecisionSensitivityF-Score 120,5333330,20,2909 190,1860470,20,1927 10,40,10,16 140,250,0250,0454 M5.1.1: NumberEvents Group Precision SensitivityF-Score 120,9333330,350,5090 190,3720930,40,38556 10,50,1250,2 1410,10,1818

61. 61 Evaluation Results

62. 62 Evaluation on ETI-VS2-AP-11-C7

63. 63 Detection of Physical Objects ( ETI-VS2-AP-11-C7.xml ) Group Precision (global) Sensitivity (global)F-Score 10,9985560,9750940,9866 80,989510,9309210,9593 280,9842810,6179510,7592 1410,5944550,7456 120,980650,5953950,7409 200,9737860,5935150,7375 90,9864760,5827070,7326 290,9879420,5775380,7289 190,9526760,5770680,7187 150,991590,5540410,7108 170,9725660,5164470,6746 320,8798190,5469920,6745 130,9288970,4788530,6319 30,4963370,3820490,4317 230,8905110,0573310,1077 M1.2.1: NumberObjectsBoundingBoxD1M2.1.1: ObjectsArea Group Precision (global) Sensitivity (global) Specificity (global)F-Score 10,890950,9790690,9957120,93293 80,9208190,8227380,9974690,86902 190,8671150,4375840,9976010,58164 280,79950,4505020,9959580,57628 200,7778260,4467610,9954340,56754 140,8221140,4328930,9966530,56715 120,9040410,4124740,9984330,56649 170,7583720,4410030,9949720,5577 150,8183450,4049780,9967830,54182 90,9476210,377360,9992540,53977 320,8852370,3673860,9982960,51927 290,9197860,3569960,9988860,51436 30,8749960,3423740,998250,49217 130,9804520,3271670,9997670,49062 230,5252550,0446250,9985570,08226

64. 64 Detection of Physical Objects: Reference Data Filtering ( ETI-VS2-AP-11-C7.xml: M1.2.1 - NumberObjectsBoundingBoxD1 ) Group Precision (global) Sensitivity (global)F-Score 10,9985560,9750940,9866 80,989510,9309210,9593 280,9842810,6179510,7592 1410,5944550,7456 120,980650,5953950,7409 90,9864760,5827070,7326 290,9879420,5775380,7289 190,9526760,5770680,7187 150,991590,5540410,7108 170,9725660,5164470,6746 320,8798190,5469920,6745 130,9288970,4788530,6319 30,4963370,3820490,4317 230,8905110,0573310,1077 No filteringWith filtering Group Precision (global) Sensitivity (global)F-Score 10,9985560,9750940,9866 80,989510,9309210,9593 280,9842810,6179510,7592 1410,5944550,7456 120,980650,5953950,7409 200,9737860,5935150,7375 90,9864760,5827070,7326 290,9879420,5775380,7289 190,9526760,5770680,7187 150,991590,5540410,7108 170,9725660,5164470,6746 320,8798190,5469920,6745 130,9288970,4788530,6319 30,4963370,3820490,4317 230,8905110,0573310,1077

65. 65 Detection of Physical Objects: Reference Data Filtering ( ETI-VS2-AP-11-C7.xml: M2.1.1 - ObjectsArea ) Group Precision (global) Sensitivity (global) Specificity (global)F-Score 10,890950,9790690,9957120,93293 80,9208190,8227380,9974690,86902 190,8671150,4375840,9976010,58164 280,79950,4505020,9959580,57628 200,7778260,4467610,9954340,56754 140,8221140,4328930,9966530,56715 120,9040410,4124740,9984330,56649 170,7583720,4410030,9949720,5577 150,8183450,4049780,9967830,54182 90,9476210,377360,9992540,53977 320,8852370,3673860,9982960,51927 290,9197860,3569960,9988860,51436 30,8749960,3423740,998250,49217 130,9804520,3271670,9997670,49062 230,5252550,0446250,9985570,08226 No filteringWith filtering Group Precision (global) Sensitivity (global) Specificity (global)F-Score 10,890950,9790690,9957120,93293 80,9208190,8227380,9974690,86902 190,8671150,4375840,9976010,58164 280,79950,4505020,9959580,57628 140,8221140,4328930,9966530,56715 120,9040410,4124740,9984330,56649 170,7583720,4410030,9949720,5577 150,8183450,4049780,9967830,54182 90,9476210,377360,9992540,53977 320,8852370,3673860,9982960,51927 290,9197860,3569960,9988860,51436 30,8749960,3423740,998250,49217 130,9804520,3271670,9997670,49062 230,5252550,0446250,9985570,08226

66. 66 Tracking of Physical Objects ( ETI-VS2-AP-11-C7.xml ) M3.2.1: NumberObjectTrackedD1 GroupPrecisionSensitivityF-Score 10,83333310,909091 1410,80,888889 80,71428610,833333 280,8 290,8 9 200,8 170,6666670,80,727273 120,6666670,80,727273 320,6666670,80,727273 150,4444440,80,571429 190,3636360,80,5 130,2105260,80,333333 30,0677970,80,125 M3.3.1.D1: TrackingTime GroupTracking Time 10,842267 80,776723 140,764944 200,764574 280,66347 120,646376 90,626901 290,612043 170,536711 150,440843 190,434929 320,432908 30,265107 130,03841 230,037081

67. 67 Tracking of Physical Objects: Reference Data Filtering ( ETI-VS2-AP-11-C7.xml: M3.2.1 - NumberObjectTrackedD1 ) GroupPrecisionSensitivityF-Score 10,83333310,909091 1410,80,888889 80,71428610,833333 280,8 290,8 9 120,6666670,80,727273 170,6666670,80,727273 320,6666670,80,727273 150,4444440,80,571429 190,3636360,80,5 130,2105260,80,333333 30,0677970,80,125 No filteringWith filtering GroupPrecisionSensitivityF-Score 10,83333310,909091 1410,80,888889 80,71428610,833333 280,8 290,8 9 120,6666670,80,727273 170,6666670,80,727273 320,6666670,80,727273 150,4444440,80,571429 190,2727270,60,375 130,2105260,80,333333 30,0508470,60,09375

68. 68 Tracking of Physical Objects: Reference Data Filtering ( ETI-VS2-AP-11-C7.xml: M3.3.1.D1 - TrackingTime) GroupTracking Time 10,842267 80,776723 140,764944 200,764574 280,66347 120,646376 90,626901 290,612043 170,536711 150,440843 190,434929 320,432908 30,265107 130,03841 230,037081 No filtering With filtering GroupTracking Time 10,842267 80,776723 140,764944 280,66347 120,646376 90,626901 290,612043 170,536711 150,440843 190,434929 320,432908 30,265107 130,03841 230,037081

69. 69 Event Recognition ( ETI-VS2-AP-11-C7.xml ) M5.1.2: NumberNamedEventsD1 GroupPrecisionSensitivityF-Score 1111 1210,6250,7692 140,7142860,6250,6666 810,50,6666 280,40,750,52173 200,3636360,50,4210 290,2272730,6250,3333 190,40,250,3076 M5.1.1: NumberEvents Group Precision SensitivityF-Score 1111 1410,8750,9333 1210,6250,7692 280,53333310,6956 810,50,6666 200,5454550,750,6315 190,80,50,6153 290,36363610,5333 150,10,250,1428

70. 70 Event Recognition ( ETI-VS2-AP-11-C7.xml ) M5.1.2: NumberNamedEventsD1 GroupPrecisionSensitivityF-Score 120,5333330,20,2909 190,1860470,20,1927 10,40,10,16 140,250,0250,0454 M5.1.1: NumberEvents Group Precision SensitivityF-Score 120,9333330,350,5090 190,3720930,40,38556 10,50,1250,2 1410,10,1818

71. 71 Evaluation Results

72. 72 Understanding versus Competition ETISEO Goal  Not a competition nor benchmarking  Emphasis on gaining insight into video analysis algorithms  Better understanding of evaluation methodology Why? ETISEO limitations:  Algorithm results depend on time and manpower (parameter tuning),  format understanding (XML), objective definition (ground truth), and algorithm capacities (static, occluded, portable and contextual objects)  previous similar experiences,  number of processed videos, frame rate, start frame  Metrics and parameters (split/merge)  learning stage required or not.

73. 73 Understanding versus Competition (cont’d) Warmest thanks to the 16 teams:  8 teams achieved high quality results  9 teams performed event recognition  10 teams produced results on all priority sequences Special thanks to teams 1, 8, 12, 14 and 28 :  Stable and high-quality results on a large video set More evaluation results…

74. 74 Conclusions Good performance comparison per video: automatic, reliable, consistent metrics. A few insights into video surveillance algorithms. For example,  Shadows  merge A few limitations:  Lack of understanding of the evaluation rules (output XML, time-stamp)  Data subjectivity: video, background, masks  Metrics and evaluation parameters Future improvements: flexible evaluation tool  Filters for reference data  Selection of metrics and parameters  Selection of videos