1. LoCaF: Detecting Real-World States with Lousy Wireless Cameras Benjamin Meyer, Richard Mietz, Kay Römer 1

2. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras  Introduction –Motivation –Challenges  System Architecture  Evaluation Structure 2

3. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras  Towards the Internet of Things –High-level state of things on the internet –Scalar/specialized sensors are often limited to one scenario –Cameras are more flexible Motivation 3 SFpark project: http://sfpark.org/

4. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras  Sensor nodes –Constrained resources  Low-cost cameras –Low resolution –Poor image quality –Low frame rate  Processing is shifted to the gateway Low-cost hardware 4

5. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Scenarios 5 Occupancy of a room Free seats in a roomIndividual occupancy of parking spots StatesFree/occupiedNumber of personsFree/occupied for each parking spot ChallengesPossibly lots of movement Outdoor  Changing lighting conditions Picture Objects to detectPeople Cars  Flexible Framework to infer and publish states for divers scenarios

6. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras System Architecture: Overview 0 HTM L RDF SQL Tweet Image capture  Compression  Wireless transmission State publication  Text templates  Different media Image processing  Regions of interest  Enhancing filters Object detection  Face detection  Mobile object detection State inference  Rule-based language Customizable workflow 6

7. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras 0 HTM L RDF SQL Tweet State publication  Text templates  Different media Image processing  Regions of interest  Enhancing filters Object detection  Face detection  Mobile object detection State inference  Rule-based language System Architecture: Sensor Node Image capture  Compression  Wireless transmission Customizable workflow 7  Camera equipped sensor node  Two capture modes –Time-triggered –Event-triggered (by PIR)  JPEG-compression in hardware  Fragmented transmission to gateway

8. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras 0 HTM L RDF SQL Tweet State publication  Text templates  Different media Image processing  Regions of interest  Enhancing filters Object detection  Face detection  Mobile object detection State inference  Rule-based language Image capture  Compression  Wireless transmission Customizable workflow 8 Image processing  Regions of interest  Enhancing filters System Architecture: Processing INSTITUTE OF COMPUTER ENGINEERING Parking spot a Parking spot b  Region selection  Lighting compensation  Texture enhancement  Contrast enhancement  Orchestration and parameterization of enhancements

9. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Image processing  Regions of interest  Enhancing filters Object detection  Face detection  Mobile object detection State inference  Rule-based language Object detection  Face detection  Mobile object detection  Face detection  Adaptive background subtraction –Classification into fore- and background –Can adapt to small changes  Blob detection –Each blob is an object  Number of & area covered by objects 9 System Architecture: Processing INSTITUTE OF COMPUTER ENGINEERING

10. Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras 0 HTM L RDF SQL Tweet State publication  Text templates  Different media Image processing  Regions of interest  Enhancing filters Object detection  Face detection  Mobile object detection State inference  Rule-based language Image capture  Compression  Wireless transmission 10 State inference  Rule-based language Customizable workflow  Rule-based state inference count:map:0:1:free count:map:1:-1:occupied State- based Event- based area:switch:free:80:occupied area:switch:occupied:80:free System Architecture: Processing INSTITUTE OF COMPUTER ENGINEERING count:map:0:1:All seats free count:map:10:45:Enough seats count:map:45:70:Almost full count:map:70:-1:No seats left count:map:0:1:free count:map:1:-1:occupied area:map:0:80:free area:map:80:100:occupied free occupied 80% coverag e

11. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras 0 HTM L RDF SQL Tweet State publication  Text templates  Different media Image processing  Regions of interest  Enhancing filters Object detection  Face detection  Mobile object detection State inference  Rule-based language System Architecture: Publishing Image capture  Compression  Wireless transmission Customizable workflow 11  Every text format (HTML, RDF, TXT, …)  Template-based  Publishing via –FTP –Twitter –SQL-Database

12. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras  Camera in front of lecture hall during lecture  Estimate number of students  Also looking at binary state (free/occupied)  One region, background subtraction & no filter  Three phases: –Beginning: Entering persons in dribs and drabs –During: Not many movements –End: Abrupt leaving of students Evaluation Setup 12

13. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Evaluation: Under- and Overestimation  Underestimation –Several persons identified as one –Persons not recognized because of no movement  Overestimation –Legs recognized as individual 13

14. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Evaluation: Entry phase OE: 130% UE: 70%  Avg: 48%  Binary state always correct 14

15. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Evaluation: Lecture phase UE: 105%  Avg: 54%  Binary state always correct 15

16. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Evaluation: Exit phase OE: ∞ UE: 222%  Avg: 95%  Binary state not correct for picture 11-13 16

17. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Evaluation: Entry phase revisited  Image filters can significantly change the estimation 17

18. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Evaluation: Entry phase revisited  Parameters can significantly change the estimation  Improved avg error: 12% 18

19. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Conclusion 19  Flexible framework  Use of cameras to be applicable in divers scenarios  Fully customizable by the user in each step  Accuracy quite high

20. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Questions? Thank you for your attention. Time for questions. 20

21. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras 21 Setup Camera node Gateway Netbook with software

22. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras The Framework: Connection Configuration

23. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras The Framework: Data Exchange

24. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras The Framework: Image Processing

25. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras The Framework: Region Selection / State Inference

26. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras The Framework: Publishing

27. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Filter

28. INSTITUTE OF COMPUTER ENGINEERING Richard Mietz LoCaF: Detecting Real-World States with Lousy Cameras Evaluation: Parking Spot Scenario area:switch:free:80:occupied area:switch:occupied:80:free  Select single spot  State switches from free to occupied when car enters (b) and c))  State will switch back when car leaves 28