2. InSight: Recognizing Humans without Face Recognition He Wang, Xuan Bao, Romit Roy Choudhury, Srihari Nelakuditi

3. Motivation – Application Scenarios 2 2 share a ride to airport Bob Elle Bret

4. Overview – Self-Fingerprints 3 Cloud Bret Bob John Elle Self-Fingerprints

5. Overview – Recognition 4 Cloud Glass

6. Overview – Recognition 5 Cloud Glass

7. Overview – Recognition 6 Cloud Glass

8. Challenges Perspectives are different Clothes have wrinkles Lighting conditions change 7

9. Extracting Fingerprints 8 Colors Patterns Spatiograms Wavelets

10. Extracting Fingerprints – Colors 9 RGBHSV Spatiograms Color Conversion

11. Extracting Fingerprints – Colors RGBHSV 10 color histogram spatial distribution Spatiograms

12. Extracting Fingerprints – Patterns 11 Wavelets Wavelet sub-bands: vertical, horizontal and diagonal dimensions.

13. Fingerprint Matching 12 Cloud Glass Matching Spatiograms S = {n’, µ’, σ’} S = {n, µ, σ} cloud glass Similarity = color histograms spatial distributions

14. Fingerprint Matching 13 Cloud Glass Matching Wavelets Bagged Decision Tree (BDT) W = {f 1, f 2, f 3,…} W = {f 1 ’, f 2 ’, f 3 ’,…} cloud glass

15. Evaluation Setting PivotHead glass captured users from the front. 15 users was dressed in their regular attires. Users actively used their smartphones. Phone opportunistically took “profile” pictures of the user. 14

16. Evaluation - Matching Color Spatiograms front

17. Evaluation - Matching Wavelets of Patterns front

18. Evaluation – Combining Colors and Patterns 17 Evaluation – front

19. Evaluation – Performance with Self-Fingerprints Matching front view 18 front

20. Self-fingerprints may not be Sufficient Clothes’ difference is not captured when clothes are similar? Clothes have different colors/patterns at the back? 19 Bret Paul Dan Bret

21. Refining the Self-Fingerprint (Similar Clothes) Cloud Dan Paul Glass Cannot recognize!

22. Refining the Self-Fingerprint (Similar Clothes) 21 Cloud Glass Dan Bob John refining self- fingerprint

23. Refining the Self-Fingerprint (Similar Clothes) 22 Dan Paul Cloud Glass Can recognize after refinement!

24. Refining the Self-Fingerprint 23 Dan Dan = {F1} Dan = {F1, F2} Dan = {F1, F2, F3} F1 F2 F2 F3

25. Refining the Self-Fingerprint (Back View) 24 Glass Cloud refining back view fingerprint Bret

26. Evaluation – Performance with Self-Fingerprints Matching back view 25 back

27. Evaluation – Performance with Refined Fingerprints Matching back view after refining fingerprints 26 back

28. Discussion Privacy of opportunistic pictures o User consent before uploading Overlapping users in view o Fingerprint refinement helps Incremental deployment with some non-participants o More time and mobility help Cloud vs p2p o Different trade-offs 27

29. Conclusion Colors and patterns on clothes help fingerprint humans. Preliminary evaluation with 15 people provides promising results. Other type of fingerprints exists such as motion. 28

30. 29 Looking for interns Anyone for beer after this talk? New Primitive for Broadcasting to Visible Vicinity.

31. Questions, Comments? Thank You