Course Projects Topics and Plans Martin Jagersand This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation In Slide Show, click on the right mouse button Select “Meeting Minder” Select the “Action Items” tab Type in action items as they come up Click OK to dismiss this box This will automatically create an Action Item slide at the end of your presentation with your points entered.

Today Some tips on carrying out your projectsSome tips on carrying out your projects –Literature search and readings –Quick prototyping (e.g. Matlab), then final implememntation (e.g. c, c++), or combining Matlab with c, c++ mex files. –Balance between reading and doing –Labs and resources available –Win free trip? Short presentation and discussion of your topic and plansShort presentation and discussion of your topic and plans

Preliminary project topics Individual and group aspect:Individual and group aspect: –Every person has some individual focus and all individual pieces combine to a whole. Main topics: Vision for 3D modeling and roboticsMain topics: Vision for 3D modeling and robotics –Real time tracking –Integrating tracking with 3D modeling –Integration of a-priori knowledge in 3D modeling –Predictive display and visualization –Visual servoing for robots (manipulator or mobile) –Visual specification and planning of robot tasks

Resources SW: I/we will install and try to help support:SW: I/we will install and try to help support: –Real time video input (under linux, video pipeline done) –Basic tracking, XVision –Geometry, Hand-eye, Robotics code HW: Access to machines in Robotics/Vision labHW: Access to machines in Robotics/Vision lab –Cameras: Linux IEEE 1394 in both grad labs and course labs csc235. –Web cams, 200Hz high speed cams, 1600x1200 Hi-Res cam available –VZ motion tracker (3000Hz special device) –Could also use digital still camera, camcorder. –Vision for motion control: Robot arms, hands, mobile robot –WAM, Barrett hand, Segway (one dedicated more coming), old Pumas –Vision and haptics: 3 Phantom omnis. –Visualization: Ok in lab (HW acc graphics), In research lab (new ATI and nvidia, SGI HW, projectors and CAVE) –Anything else? Some resources for buying available

Martin’s tips Plan incremental progress and checkpoints.Plan incremental progress and checkpoints. –Makes it easier to identify promising directions as well as difficulties and redefine plans as needed. Find balance between reading and doingFind balance between reading and doing –It is difficult to fully grasp methods by only reading –Some experiments are incomplete, results wrong Practical trying out can add a lot of insight.Practical trying out can add a lot of insight. –Learn how to quickly prototype in e.g. matlab

Literature search Goal: Find the most relevant and recent papers in a subarea.Goal: Find the most relevant and recent papers in a subarea. Method:Method: 1.Seed with a few relevant papers. 2.Do internet search. e.g. “research index” or google scholar 3.Do citation search backwards and forwards. 4.Find common “buzz words”. Do title and abstract text search. 5.Check most recent proceedings manually. (They won’t be indexed yet)

Literature search 2 Expect to:Expect to: –Read the titles of hundreds of papers (and web pages) –Read the abstract of of papers –Skim through dozens of papers In order to find the or so relevant papers. Read these in detail to understand the topic.In order to find the or so relevant papers. Read these in detail to understand the topic. Of these select a handful of the most closely related to benchmark your project to.Of these select a handful of the most closely related to benchmark your project to.

Report: 1.Review –Summarize the main contributions and comparing the results in the papers. 2.Your contribution and experiments. –Methods –Results 3.Discussion –Where does it fit into the bigger picture –Future work

Schedule Now good time to think about and refine project plansNow good time to think about and refine project plans Late OctLate Oct Written project proposal. –Include reference list and a start at literature review, ie. Read some papers and write a few pages summary Throughout course in class:Throughout course in class: –Keep up to date on your project progress. –In class presentation of project readings and analysis End of semester: Project reports.End of semester: Project reports.

Preliminary project topics Objective: Vision for 3D modeling and robotics Objective: Vision for 3D modeling and robotics Individual partsIndividual parts –Real time tracking –Integrating tracking with 3D modeling –Integration of a-priori knowledge in 3D modeling –Predictive display and visualization –Mapping and navigation for mobile robots –Visual servoing for manipulators –Visual specification and planning of robot tasks –Singularity and obstacle avoidance Subgroups: Vision, RoboticsSubgroups: Vision, Robotics

GPU accelerated visual tracking Tracking readings: Color, Feature, SSD, SIFT.Tracking readings: Color, Feature, SSD, SIFT. Investigate what maps naturally to GPU, CPUInvestigate what maps naturally to GPU, CPU Make incremental plan, e.g.Make incremental plan, e.g. –Video pipeline: Cam->Video RAM or CPU RAM? –Basic image processing on GPU: Lin alg, conv, filt, im deriv –Implement and test various tracking: Color, SSD, SIFT –Integrate with other system parts Design and carry out experiments: Test sequences robustness, accuracy etc.Design and carry out experiments: Test sequences robustness, accuracy etc.

Tracking and 3D modeling Make tracking more robust by restricting 2D image tracker movement to those consistent with a 3D interpretationMake tracking more robust by restricting 2D image tracker movement to those consistent with a 3D interpretation –Rigid constraints: –Loose constraints –Language for partial constraints (Collab with visual spec) Convergence tradeoff:Convergence tradeoff: –Restricted tracker may not reach elliptical point –Unrestricted tracker may track wrong points Experiments: What are good test sequences? How accurate is tracking? Captured 3D? How robust?Experiments: What are good test sequences? How accurate is tracking? Captured 3D? How robust?

Use of a-priori knowledge in modeling Can make 3D capture easier:Can make 3D capture easier: –Tells if recovered scene is “probable” –Orth, plan etc: and gives Euclidean structure w/o cumbersome “self calibration” Types of a-priori knowledge:Types of a-priori knowledge: –Generic: orthogonality, parallelism, planes. –Specific: architectural: houses&features(doors windows) Indoors: Furniture on floor, lamps on wall, scales: room, furn, items… How to mathematically incorporate:How to mathematically incorporate: –Hard constraints –Probabilistic How to practically add:How to practically add: –Image editor collaborate with Visual spec project Experiments:Experiments: –Scenes from photos, –Indoor scenes from video

Predictive Display Visual User Interface Systems oriented projectSystems oriented project How to modify 3D capture to incrementally detect changes in 3D remote scene, send and incorporate in modelHow to modify 3D capture to incrementally detect changes in 3D remote scene, send and incorporate in model How to display 3D model in HMDHow to display 3D model in HMD –Minimize latencies How to track and interpret human motions. Control robot motion based on these.How to track and interpret human motions. Control robot motion based on these. Could also be set up in CAVECould also be set up in CAVE

Visual servoing of manipulators Manipulators: Arm’s hands, high DOF devicesManipulators: Arm’s hands, high DOF devices Investigate properties of measurement:Investigate properties of measurement: –E-functions, properties of robot. How to estimate visual-motor:How to estimate visual-motor: –kinematics? Dynamics? –Local, global? Design controllersDesign controllers –Combine joint and visual feedback Integrate with tracking and visual space specificationsIntegrate with tracking and visual space specifications Experiements.Experiements.

Visual specification What are natural visual task primitives?What are natural visual task primitives? What tasks do the solve? Completeness under particular geometry?What tasks do the solve? Completeness under particular geometry? How “smooth” E-functions do they give? (Collaboration with visual servoing).How “smooth” E-functions do they give? (Collaboration with visual servoing). How to have human enter:How to have human enter: –Pointing 2D image editor –and gesturing in 3D Experiments and test in vision-based manipExperiments and test in vision-based manip

Singularity and Obstacle avoidance Investigate what aspects of calibrated Euclidean approach carry overInvestigate what aspects of calibrated Euclidean approach carry over How to find and characterize “difficult regions”How to find and characterize “difficult regions” –Find: Computer vision, contact, other sensing –Region representation: Points, Regions, Potential function How to combine with visual servoing controllerHow to combine with visual servoing controller –Primary or secondary controller objective –Effects on convergence Experiments with synthetic data, real sequencesExperiments with synthetic data, real sequences

Next steps Firm up project ideas, specificationFirm up project ideas, specification Identify readingsIdentify readings Discuss plansDiscuss plans Write project proposalWrite project proposal Investigate hardware needs and plan useInvestigate hardware needs and plan use Plan interaction with other people/groupsPlan interaction with other people/groups Iterate: Devise method, implement, testIterate: Devise method, implement, test Final integration with other partsFinal integration with other parts Write final reportWrite final report