1. Thrust IIA: Environmental State Estimation and Mapping Dieter Fox (Lead) Nicholas Roy MURI 8 Kickoff Meeting 2007

2. Task Objective: Human-Centered Maps  Observation: Automatic map-building (SLAM) is solved sufficiently well  Goal: Describe environments by higher-level concepts:  Places (room, hallway, street, walkway, parking lot, …)  Objects (tree, person, building, car, wall, …)  Key challenges:  Estimating concept types is mostly a discrete problem  Complex features and relationships MURI 8 Kickoff Meeting 2007 University of Washington

3. Existing Technology  Human-centered mapping requires  integration of high-dimensional, continuous features from multi-modal sensor data  reasoning about spatial and temporal relationships  Conditional Random Fields provide extremely flexible probabilistic framework for learning and inference MURI 8 Kickoff Meeting 2007 University of Washington

4. Conditional Random Fields  Discriminative, undirected graphical model  Introduced for labeling sequence data to overcome weaknesses of Hidden Markov Models [Lafferty-McCallum-Pereira: ICML-01]  Applied successfully to  Natural language processing [McCallum-Li: CoNLL-03], [Roth-Yih: ICML-05]  Computer vision [Kumar-Hebert: NIPS-04], [Quattoni-Collins-Darrel: NIPS-05]  Robotics [Limketkai-Liao-Fox: IJCAI-05], [Douillard-Fox-Ramos: IROS-07] MURI 8 Kickoff Meeting 2007 University of Washington

5. Conditional Random Fields  Directly models conditional probability p(x|z) (instead of modeling p(z|x) and p(x), and using Bayes rule to infer p(x|z)).  No independence assumption on observations needed! Hidden states x Observations z MURI 8 Kickoff Meeting 2007 University of Washington

6. Online Object Recognition MURI 8 Kickoff Meeting 2007 [Douillard-Fox-Ramos: IROS-07, ISRR-07]

7. From Laser Scans to CRFs MURI 8 Kickoff Meeting 2007 Object type of laser beam 1 Shape and appearance Object type of laser beam 2 Object type of laser beam 3 Object type of laser beam 4 Object type of laser beam n Shape and appearance …

8. Visual Features MURI 8 Kickoff Meeting 2007 steerable pyramid 3-d RGB histogram 3-d HSV histogram

9. Geometric Features MURI 8 Kickoff Meeting 2007

10. Temporal Integration MURI 8 Kickoff Meeting 2007 ………… k-2k-1kk+1  Taking past and future scans into account can improve labeling accuracy.  Match consecutive laser scans using ICP.  Associated laser points are connected in CRF.  Can perform online filtering or offline smoothing via BP.

11. Example Trace: Car vs. Others MURI 8 Kickoff Meeting 2007  Trained on 90 labeled scans  Inference via filtering in CRF

12. 7 Class Example Labeling MURI 8 Kickoff Meeting 2007

13. 7 Class Example Labeling MURI 8 Kickoff Meeting 2007

14. 7 Class Example Labeling MURI 8 Kickoff Meeting 2007

15. 7 Class Example Labeling MURI 8 Kickoff Meeting 2007

16. 7 Class Example Labeling MURI 8 Kickoff Meeting 2007

17. Proposed Technical Advances  Integrate recognition results into maps  Improve results by leveraging web training data and high level object detectorss  Add object types suited for target scenario  Improve CRF training MURI 8 Kickoff Meeting 2007 University of Washington

18. Situation Awareness via Wearable Sensors MicrophoneCamera Light sensors 2 GB SD card Indicator LEDs  Records 4 hours of audio, images (1/sec), GPS, and sensor data (accelerometer, barometric pressure, light intensity, gyroscope, magnetometer)

19. Soldier Activity Recognition  Automatic generation of mission summaries  Motion type (linger, walk, run, drive, …)  Environment (inside, outside building)  Events (conversations, marked via keyword)  Technical challenges  High-dimensional, continuous observations / features  Different data rates: (1 Hz - 256 Hz)  Getting labeled training data  Different persons / environments MURI 8 Kickoff Meeting 2007

20. Activity Model MURI 8 Kickoff Meeting 2007 e t-1 Environment indoor, outdoor, vehicle etet a t-1 atat c t-1 ctct Activity walk, run, stop, up/downstairs, drive, elevator, cover [Subramanya-Raj-Bilmes-Fox: UAI-06, ISRR-06] Sensor data High-dimensional feature vectors

21. Data Visualization / Summarization MURI 8 Kickoff Meeting 2007 GPS traces Image sequence (currently in car) Timeline of soldier activities

22. Milestones  Goals:  Real time wearable interface on cell phone  Data sharing among soldiers and robots  Real time display on remote laptop MURI 8 Kickoff Meeting 2007

23. Milestones  Year 1:  Real time data sharing between wearable sensor platforms  Integration of object recognition into mapping  Year 2:  Real time data sharing between soldiers, robots, and remote laptop  Detection of specific soldier states / activities (moving, incapacitated,...)