1. Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group (VIREO), City University of Hong Kong Digital Video and Multimedia Lab (DVMM), Columbia University 1 NIST TRECVID Workshop, Nov. 2009

2. Overview: framework Local Feature Global Feature SVM Classifiers 6 6 5 5 1-4 VIREO-374: 374 LSCOM concept detectors VIREO-374: 374 LSCOM concept detectors Domain Adaptive Semantic Diffusion

3. 3 Overview: performance DASD Local + global features Local feature alone Local feature is still the most powerful component (MAP=0.150) Global features help a little bit (MAP=0.156) DASD further contributes incrementally to the final detection

4. Overview: framework Local Feature Global Feature SVM Classifiers 6 6 5 5 1-4 VIREO-374: 374 LSCOM concept detectors VIREO-374: 374 LSCOM concept detectors Domain Adaptive Semantic Diffusion

5. Local feature representation 5 Chang et al TRECVID 2008; Jiang, Yang, Ngo & Hauptmann, IEEE TMM, to appear

6. Context-based concept detection Local Feature Global Feature SVM Classifiers 6 6 5 5 1-4 VIREO-374: 374 LSCOM concept detectors VIREO-374: 374 LSCOM concept detectors DASD: Domain Adaptive Semantic Diffusion

7. DASD - motivation Most existing methods aim at the assignment of concept labels individually –but concepts do not occur in isolation! military personnel smoke explosion_fire roadoutdoor vehicle building 7

8. 8 Documentary Videos Broadcast News Videos Most existing methods aim at the assignment of concept labels individually –but concepts do not occur in isolation! Domain change between training and testing data was not considered DASD - motivation

9. 9 Jiang, Wang, Chang & Ngo, ICCV 2009 DASD - overview road vehicle 0.05 0.19 0.80 0.46 0.13 0.01 0.12 0.91 0.18 0.05 water 0.11 0.58 0.10 0.13 0.02 sky 0.01 0.36 0.53 0.17 0.23

10. DASD - overview Domain adaptive semantic diffusion (DASD) –Semantic graph Nodes are concepts Edges represent concept correlation – Graph diffusion Smooth concept detection scores w.r.t the concept correlation vehicle road Watersky 0.1 0.2 0.8 0.5 0.1 … 0.4 0.1 0.6 0.1 0.0 … 0.8 0.0 0.4 0.5 0.2 0.8 … 0.7 0.0 0.1 0.9 0.2 0.1 … 0.3 10

11. DASD - formulation Energy function 11 Concept affinity Detection score of concept c i on test samples

12. DASD - formulation (cont.) Gradually smooth the function makes the detection scores in accordance with the concept relationships Detection score smoothing process 12

13. DASD - formulation (cont.) Graph adaptation Graph adaptation process 13

14. Iteration: 8Iteration: 12Iteration: 0Iteration: 4Iteration: 16Iteration: 20 Graph adaptation - example Broadcast news video domain Documentary video domain 14

15. Experiments on TV 05-07 Baseline detectors –VIREO-374 Graph construction: –Ground-truth labels on TRECVID 2005 SPORTSWEATHER OFFICEBUILDING DESERT MOUNTAIN WALKING PEOPLE- MARCHING EXPLOSION-FIRE MAP TRUCK CORP. LEADER SPORTSWEATHEROFFICE DESERT MOUNTAIN WATER POLICE MILITARY ANIMAL TWO PEOPLE NIGHT TIME TELEPHONE STREET CLASSROOMBUS TRECVID 05/06 (Broadcast News Videos) TRECVID 07 (Documentary Videos) 15

16. Results on TV 05-07 Performance gain on TRECVID 05-07 Datasets TRECVID-200520062007 # of evaluated concepts3920 Baseline (MAP)0.1660.1540.099 SD11.8%15.6%12.1% DASD11.9%17.5%16.2% 16 SD: semantic diffusion (without graph adaptation) Consistent improvement over all 3 data sets DASD: domain adaptive semantic diffusion Graph adaptation further improves the performance

17. Results on TV 05-07 (cont.) TRECVID 2006 Test Data 17 TRECVIDJiang et alAytar et alWeng et alDASD 20052.2%4.0%N/A11.9% 2006N/A 16.7%17.5% Comparison with the state-of-the-arts

18. 18 Results on TRECVID 09 30% 10% 5%

19. Results on TRECVID 09 (cont.) 19 Quality of contextual detectors (VIREO-374) Context VIREO-374 TV09 detectors TV06 detectors TV07 detectors 18% 16% 5% DASD performance gain

20. DASD - computational time Complexity is O(mn) –m: # concepts; n: # video shots Only 2 milliseconds per shot/keyframe! 20 TRECVID 05TRECVID 06TRECVID 07 SD59s84s12s DASD89s165s28s

21. Summary 21 A well-designed approach using local features achieves good results for concept detection. Context information is helpful ! –Domain adaptive semantic diffusion effective for enhancing concept detection accuracy can alleviate the effect of data domain changes highly efficient ! – Future directions include: detector reliability: diffusion over directed graph web data annotation: utilize contextual information to improve the quality of tags –Source code available for download from DVMM lab research page

22. 22