1. Lead: Roth (UIUC) Abdelzaher (UIUC) Huang (UIUC) Lei (IBM) Presented by: Tarek Abdelzaher

2. Task Goal and Overview Accounting for Prior Knowledge with Constrained Conditional Models Prior Knowledge Information Network Uncovering Links in Heterogeneous Content ? Communication Network Resource Bottleneck Sensor, text, image, and human sources Latency Latency Analysis Goal: Foundations for utilizing context and prior knowledge in fusion Foundations for analysis of fusion latency.

3. Data Fusion Threads Thread 1: Enable exploitation of prior knowledge and information network links in the design of algorithms for data fusion (Dan Roth: UIUC) Thread 2: Enhance ability to uncover links between heterogeneous content items, such as text and video (Huang, UIUC) Thread 3: Advance latency analysis of distributed data fusion algorithms (Abdelzaher, UIUC) Thread 4: Validate the results on viable platforms and crowd-sourcing applications (Lei, IBM Research)

4. Outline Accounting for Prior Knowledge with Constrained Conditional Models Prior Knowledge Information Network Uncovering Links in Heterogeneous Content ? Communication Network Resource Bottleneck Sensor, text, image, and human sources Latency Latency Analysis

5. Outline Accounting for Prior Knowledge with Constrained Conditional Models Prior Knowledge Information Network Uncovering Links in Heterogeneous Content ? Communication Network Resource Bottleneck Sensor, text, image, and human sources Latency Latency Analysis Thread 1

6. Thread 1: A Framework for Integrating Prior Knowledge: Fundamentals of Context-aware Real-time Data Fusion Advances in Learning & Inference of Constrained Conditional Models CCM: A computational framework for learning and inference with interdependent variables in constrained settings Formulating Information Fusion as CCMs. Preliminary theoretical and experimental work on Information Fusion Key Publications: R. Samdani and D. Roth, Efficient Learning for Constrained Structured Prediction, submitted. M. Chang, M. Connor and D. Roth, The Necessity of Combining Adaptation Methods, EMNLP’10. M. Chang, V. Srikumar, D. Goldwasser and D. Roth, Structured Output Learning with Indirect Supervision, ICML’10. M. Chang, D. Goldwasser, D. Roth and V. Srikumar, Discriminative Learning over Constrained Latent Representations, NAACL’10 G. Kundu, D. Roth and R. Samdani, Constrained Conditional Models for Information Fusion, submitted. 6

7. 7  Predict values of multiple, interdependent labels (in contexts as diverse as information extraction, information trustworthiness, information fusion, etc.)  Modeling complex dependencies leads to intractability of learning & inference (decision making)  Leads to over-simplification & unjustified independence assumptions  Constrained conditional models (CCMs) pair relatively simple learning models with expressive prior knowledge in the form of declarative constraints in supporting global decisions.  Learn models for sub-problems; incorporate models ’ information, along with prior knowledge/constraints, in making globally coherent decisions Fusion as a Decision Problem Learn models; Acquire knowledge/constraints; Make decisions.  Recent Progress: LoCL (Locally Consistent Learning): a scheme which is consistent with Global Learning under certain conditions while being efficient.  Theoretical contribution and experimental confirmation on info extraction tasks.

8. Illustrative Example Learning an optimal path AC A B C AB opt BC opt Sub-problem AB Sub-problem BC

9. Illustrative Example Learning an optimal path A B C AB opt BC opt Constraint: No left turns at B

10. Illustrative Example Learning an optimal path A B C BC opt Constraint: No left turns at B LoCL: Using local models + constraints  find global optima Global Optimum

11. Application: Disaster Scenario Information Sources Resource constraints: Router Command Center Selected Information Predicted States @ various locations: {y 1,y 2,…y n } Feedback Text Messages Images Data from sensors Predict output states of different locations over consecutive time steps Output space is spatially and temporally structured Expressing this structure using constraints can help make coherent predictions and boost accuracy.

12. Outline Accounting for Prior Knowledge with Constrained Conditional Models Prior Knowledge Information Network Uncovering Links in Heterogeneous Content ? Communication Network Resource Bottleneck Sensor, text, image, and human sources Latency Latency Analysis Thread 1

13. Outline Accounting for Prior Knowledge with Constrained Conditional Models Prior Knowledge Information Network Uncovering Links in Heterogeneous Content ? Communication Network Resource Bottleneck Sensor, text, image, and human sources Latency Latency Analysis Thread 2 Thread 1

14. Thread 2: Constructing Cross- Domain Translator (UIUC, IBM) Source instances (text) Target instances (images) Bridge the cross-domain gap?

15. Constructing Cross-Domain Translator Source instances (text) Target instances (images) W (s) W (t) Common Latent Space Inner product in latent space as translator

16. Technical Contributions Cross-Domain Knowledge Propagation Propagating Knowledge in surrounding text to visual data Published in WWW’11, collaboration with Dr. Charu Aggarwal, IBM Cross-Category Knowledge Sharing Exploring the concept correlations to enhance the inference accuracy To appear in CVPR’11, collaboration with Dr. Charu Aggarwal, IBM Modeling Context-Aware Image Similarity Applications into Disaster Assessment (Collaboration with Prof. Tarek Abdelzaher) KDD’11, submitted

17. Outline Accounting for Prior Knowledge with Constrained Conditional Models Prior Knowledge Information Network Uncovering Links in Heterogeneous Content ? Communication Network Resource Bottleneck Sensor, text, image, and human sources Latency Latency Analysis Thread 1 Thread 2

18. Outline Accounting for Prior Knowledge with Constrained Conditional Models Prior Knowledge Information Network Uncovering Links in Heterogeneous Content ? Communication Network Resource Bottleneck Sensor, text, image, and human sources Latency Latency Analysis Thread 3 Thread 1 Thread 2

19. Thread 3: Latency Analysis In Collaboration with Aylin Yener, CNARC Goal: Answer the question: How much work can be done “on time” (given different data fusion workflows and different end-to-end deadlines) Derive the real-time capacity region (load region where deadlines are met) Model: Different data flows share distributed computational and communication resources Each flow is represented by its own workflow graph Different flows have different end-to-end deadlines (worst-case allowable end-to-end latency) Results: An algebra for reducing distributed workflows to equivalent canonical “centralized systems” A real-time capacity region for the canonical system

20. A Reduction Theory for Distributed Systems In collaboration with CNARC (OICC) Based on reduction of distributed systems to an “equivalent uniprocessor” C 1,1 = 2C 1,2 = 1.1 C 2,1 = 1C 2,2 = 1.8 Stage 1Stage 2 C 1,max = 2 C 2,max = 1.8 C max,1 = 2C max,2 = 1.8 F1F1 F2F2 C 1 eq = 2 C 2 eq = 1.8 Equivalent Uniprocessor

21. Reduction of Busy Pipelines C 1,1 = 2C 1,2 = 1.1 C 2,1 = 1C 2,2 = 1.8 Stage 1Stage 2 C 1,max = 2 C 2,max = 1.8 C max,1 = 2C max,2 = 1.8 F1F1 F2F2 10 pipeline jobs of F 1 10 pipeline jobs of F 2 Stage 2 Stage 1 Time (b) Uniprocessor Approximation 10 uniprocessor jobs of C 1,max each10 uniprocessor jobs of C 2,max each (a) Original Pipeline Execution

22. Reduction of Data Fusion Trees 1 2 2 1 3 1 1 F1F1 F2F2 2 1 2 2 2 F3F3 2 1 2 1 1 2 2 1 2 2 3 3 F1F1 F2F2 F3F3 (a) Distributed Data Fusion System of Three Workflows (b) Equivalent Uniprocessor

23. New: Reduction of Data Fusion Trees 1 2 2 1 3 1 1 F1F1 F2F2 2 1 2 2 2 F3F3 2 1 2 1 1 2 2 1 2 2 3 3 F1F1 F2F2 F3F3 (a) Distributed Data Fusion System of Three Workflows (b) Equivalent Uniprocessor

24. The Real-time Capacity Region The real-time capacity theorem: In a system with a set, S, of processing workflows, where each workflow F i in S incurs an effective utilization u i effect on an equivalent uniprocessor and has a job rate R i and a per-job end-to-end maximum latency constraint, D i, all jobs meet their end-to-end deadlines if: where:

25. The Real-time Capacity Region The real-time capacity theorem: In a system with a set, S, of processing workflows, where each workflow F i in S incurs an effective utilization u i effect on an equivalent uniprocessor and has a job rate R i and a per-job end-to-end maximum latency constraint, D i, all jobs meet their end-to-end deadlines if: where: Guaranteed (safe) real-time capacity region

26. Performance Evaluation Theoretically predicted real-time capacity bound is very close to empirical onset of deadline misses

27. Thread 4: Validation IBM, UIUC Develop a general platform reusable for different mobile crowd-sensing applications to experiment with data fusion applications

28. Road Ahead Analysis of trade-offs between timeliness and fusion quality Investigation of the dependency of fusion quality and timeliness on distributed resource allocation. Integration of prior knowledge, constraints, and resource distribution issues into future data fusion algorithms. Improving quality/cost trade-offs via link discovery (between text and video) Information-network-aware real-time capacity of data fusion. Validation, documentation and publications.

29. Collaborations Fusion Task I1.1 New fusion algorithms Accurate, timely QoI Task I1.2 Better storage policies Better fusion from human sources Capacity Task I1.2 Community Modeling S2.2 CNARC QoI Task I1.2 In-network Storage I2.1/C2.1 Decisions under Stress S3.1 Provenance Task T1.3 Characterization of QoI/cost trade-offs Improved diagnostic capabilities in fusion systems Improved network QoI optimization for fusion systems Improved effective operational capacity

30. Papers Thread 1 (Q1): (UIUC): Gourab Kundu, Rajhans Samdani, Dan Roth, “Constrained Conditional Models for Information Fusion,” submitted to Fusion 2011 (UIUC): Dan Roth at al. “Efficient Learning for Constrained Structured Prediction” Submitted to ICML 2011 Thread 2 (Q2): (INARC+CNARC): Forrest Iandola, Fatemeh Saremi, Tarek Abdelzaher, Praveen Jayachandran, Aylin Yener, “Real-time Capacity of Networked Data Fusion,” submitted to Fusion 2011

31. More Papers Thread 3 (I1.1-I1.2 Collaboration/Multi-institution): (UIUC+IBM) G. Qi, C. Aggarwal, T. Huang, “Towards Semantic Knowledge Propagation between text and web images,” WWW Conference, 2011. (UIUC+IBM) Guo-Jun Qi, Charu Aggarwal, Yong Rui, Qi Tian, Shiyu Chang and Thomas Huang, “Towards Cross-Category Knowledge Propagation for Learning Cross-domain Concepts,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011), Colorado Springs, Colorado, June 21-23, 2011 (IBM+UIUC) C. Aggarwal, Y. Zhao, P. Yu. On Wavelet Decomposition of Uncertain Text Streams, CIKM Conference, 2011. (UIUC+IBM) G. Qi, C. Aggarwal, T. Huang, “Transfer learning with distance functions between text and web images,” Submitted to the ACM KDD Conference, 2011. (UIUC+IBM) G. Qi, C. Aggarwal, H. Ji, T. Huang, “Exploring Content and Context-based Links in Social Media: A Latent Space Method,” Submitted to IEEE Transactions on Pattern Mining (TPAMI) Thread 4 (Q3/Q4) Raghu Ganti, Fan Ye, Hui Lei, “Mobile Crowdsensing: Current State and Future Challenges,” in submission to IEEE Comm. Magazine

32. Military Relevance Enhanced warfighter’s ability to interpret reports, sensory data, and soft information sources for making the right decisions Enhanced exploitation of semantic links between information items to improve data fusion accuracy Improved ability to utilize context and background knowledge in interpreting data Significantly improved situation assessment in the presence of heterogeneous content Improved latency analysis algorithms for data fusion systems to ensure timeliness of fusion results