Interactive Space – An application of OpenCV Sam Siciliano

Overview Who I am and the project background. The System The Software Components Communication The Software APIs Custom Code User Interface Ideas, Expansions and Surprises Demonstration and Videos

Who I am Graduated summer before last from PSU with a degree in computer engineering Embedded Systems Engineer for Logic Product Development OMSI Employee at the time of the project

The Project Portland State University Senior Capstone Project done in conjunction with OMSI Team of 6 Was in OMSI 3-4 months, currently lives on my laptop.

The System – Block Diagram

The System - Components “Technobeam” intelligent light fixture DMX-512 Lighting Protocol 180 degrees pan, 90 degrees tilt Adjustable colors, patterns via “litho wheel” and “gobos” Iris and adjustable focus

The System - Components The DMX-512 transceiver Shifts from RS-232 to RS-485 Increases baud rate, from 1152 to 2500

The System – Components The Server Dell, p4 2GHz Machine. Windows XP Professional Generic Video Capture Card

Software Components Computer Vision Input Video Stream via DirectX 9 and Direct Show Calibration Based Mapping TB/PT Serial Control Classes Xvid, Yvid XTB, YTB Frame

Video Capture Via DirectX Direct Show and Filter Graphs Example of Simple Video Capture: Graph Used for the Project:

Video Capture Via DirectX Install DirectX 9.0 SDK (note – DirectShow is not included with latest version!!) Use the DirectShow API to Create a Filter Graph Insert ProxyTrans Filter that comes with OpenCV Define a callback function and start playback

Computer Vision Initial Conditions Tracking “cvgoodfeaturestotrack” Region and Trigger Selection Tracking Optical Flow Point Removal Rules Centroid Calculation

Throw out invalid points in P[] Algorithim No Object Lost Get Trackable Features in ROI (points P[] – 1) > 0 points in P[]? Yes Calculate Centroid of P[] Save Frame (Frame N – 1) Send centroid to remapping functions Get Next Frame (Frame N) Save Current Frame (now N – 1) Pass in all to OpticalFlow Function To get updated points (P[]) Save Current Points (now P – 1) Throw out invalid points in P[]

Calibration and Mapping UI setup to gather calibration data points a set of 2 inputs, 2 outputs Points are collected once per given setup Accounts for “fish-eye” lens distortion

Calibration and Mapping - 2 Our problem context: First Solution: get to calibration points (lower left, upper right) to obtain thresholds, linearly scale and shift: ??? Pan (from 0 to 216) Xvid(from 0 to 320) Yvid(from 0 to 240) Tilt (from 0 to 216)

Calibration and Mapping - 3 Worked fine with initial camera we were using. When switching though to a camera with a wide angle lens, system wasn’t adequate due to “fisheye” distortion. New Scheme: 9 point calibration and piece-wise linear interpolation.

Calibration and Mapping - 4 x3,y3, pan3 x1,y1, pan1 x2,y2, pan2

Ideas and Surprises Feedback enabled “throwing” of the light Hypersonics and Music Use fuzzy logic with camera on pan/tilt unit

Questions?