Ullman's Visual Routines and Tekkotsu Sketches

Slides:



Advertisements
Similar presentations
Reminder: extra credit experiments
Advertisements

Chapter 2: Marr’s theory of vision. Cognitive Science  José Luis Bermúdez / Cambridge University Press 2010 Overview Introduce Marr’s distinction between.
Ray tracing. New Concepts The recursive ray tracing algorithm Generating eye rays Non Real-time rendering.
3D Graphics Rendering and Terrain Modeling
CLASS 9 ADVANCE RENDERING RAY TRACING RADIOSITY LIGHT FIELD CS770/870.
Carnegie Mellon University School of Computer Science Carnegie Mellon University School of Computer Science Cognitive Primitives for Mobile Robots Development.
Features and Objects in Visual Processing
Upcoming Stuff: Finish attention lectures this week No class Tuesday next week – What should you do instead? Start memory Thursday next week – Read Oliver.
EE 7730 Image Segmentation.
Visual Cognition & Visual Routines by Shimon Ulman.
Computational Vision: Object Recognition Object Recognition Jeremy Wyatt.
Visual Attention More information in visual field than we can process at a given moment Solutions Shifts of Visual Attention related to eye movements Some.
Next Week Memory: Articles by Loftus and Sacks (following week article by Vokey)
Features and Object in Visual Processing. The Waterfall Illusion.
Features and Object in Visual Processing. The Waterfall Illusion.
CS292 Computational Vision and Language Visual Features - Colour and Texture.
1 Angel: Interactive Computer Graphics 4E © Addison-Wesley 2005 Models and Architectures Ed Angel Professor of Computer Science, Electrical and Computer.
A Brief Overview of Computer Vision Jinxiang Chai.
1 Perception and VR MONT 104S, Spring 2008 Lecture 22 Other Graphics Considerations Review.
Studying Visual Attention with the Visual Search Paradigm Marc Pomplun Department of Computer Science University of Massachusetts at Boston
COMP 175: Computer Graphics March 24, 2015
Technology and Historical Overview. Introduction to 3d Computer Graphics  3D computer graphics is the science, study, and method of projecting a mathematical.
Distance fields for high school geometry Mark Sawula Peddie School 17 Apr 10 (work in progress)
1. Introduction Motion Segmentation The Affine Motion Model Contour Extraction & Shape Estimation Recursive Shape Estimation & Motion Estimation Occlusion.
CSCE 5013 Computer Vision Fall 2011 Prof. John Gauch
CSC 461: Lecture 3 1 CSC461 Lecture 3: Models and Architectures  Objectives –Learn the basic design of a graphics system –Introduce pipeline architecture.
1 Introduction to Computer Graphics with WebGL Ed Angel Professor Emeritus of Computer Science Founding Director, Arts, Research, Technology and Science.
Black-box Testing.
1Computer Graphics Lecture 4 - Models and Architectures John Shearer Culture Lab – space 2
Advanced Computer Graphics Advanced Shaders CO2409 Computer Graphics Week 16.
Computer Graphics Chapter 6 Andreas Savva. 2 Interactive Graphics Graphics provides one of the most natural means of communicating with a computer. Interactive.
1 Perception and VR MONT 104S, Fall 2008 Lecture 21 More Graphics for VR.
1 Research Question  Can a vision-based mobile robot  with limited computation and memory,  and rapidly varying camera positions,  operate autonomously.
Review on Graphics Basics. Outline Polygon rendering pipeline Affine transformations Projective transformations Lighting and shading From vertices to.
(c) 2000, 2001 SNU CSE Biointelligence Lab Finding Region Another method for processing image  to find “regions” Finding regions  Finding outlines.
CORNELL UNIVERSITY CS 764 Seminar in Computer Vision Ramin Zabih Fall 1998.
COMPUTER GRAPHICS CS 482 – FALL 2015 SEPTEMBER 29, 2015 RENDERING RASTERIZATION RAY CASTING PROGRAMMABLE SHADERS.
Lecture 15: Raster Graphics and Scan Conversion
9/30/ Cognitive Robotics1 Gestalt Perception Cognitive Robotics David S. Touretzky & Ethan Tira-Thompson Carnegie Mellon Spring 2006.
CS552: Computer Graphics Lecture 16: Polygon Filling.
3D Rendering 2016, Fall.
A Plane-Based Approach to Mondrian Stereo Matching
Rendering Pipeline Fall, 2015.
- Introduction - Graphics Pipeline
The Map Builder Cognitive Robotics David S. Touretzky &
Binary Notation and Intro to Computer Graphics
Computer Graphics.
Gestalt Perception Cognitive Robotics David S. Touretzky &
DIGITAL SIGNAL PROCESSING
Understanding Agent Knowledge through Conversation
Ullman's Visual Routines, and Tekkotsu Sketches
Object Recognition Cognitive Robotics David S. Touretzky &
Object Recognition Cognitive Robotics David S. Touretzky &
The Map Builder Cognitive Robotics David S. Touretzky &
3D Graphics Rendering PPT By Ricardo Veguilla.
3D Object Representations
The Graphics Rendering Pipeline
CS451Real-time Rendering Pipeline
Models and Architectures
© University of Wisconsin, CS559 Fall 2004
Models and Architectures
Models and Architectures
Introduction to Computer Graphics with WebGL
Brain States: Top-Down Influences in Sensory Processing
Dr. Jim Rowan ITEC 2110 Chapter 3
Models and Architectures
Guest Lecture by David Johnston
Models and Architectures
Morphological Operators
IT472 Digital Image Processing
Presentation transcript:

Ullman's Visual Routines and Tekkotsu Sketches 15-494 Cognitive Robotics David S. Touretzky & Ethan Tira-Thompson Carnegie Mellon Spring 2012 5/18/2018 15-494 Cognitive Robotics

Parsing the Visual World How does intermediate level vision work? How do we parse a scene? Is the x inside or outside the closed curve? 5/18/2018 15-494 Cognitive Robotics

Ullman: Visual Routines Fixed set of composable operators. Wired into our brains. Operate on “base representations”, produce “incremental representations”. Can also operate on incremental representations. 5/18/2018 15-494 Cognitive Robotics

Base Representations Derived automatically; no decisions to make. Derivation is fully parallel. Multiple parallel streams in the visual hierarchy. Describe local image properties such as color, orientation, texture, depth, motion. Marr's “primal sketch” and “2 ½-D Sketch” 5/18/2018 15-494 Cognitive Robotics

Primal Sketch 5/18/2018 15-494 Cognitive Robotics

Incremental Representations Constructed by visual routines. Describe relationships between objects in the scene. Construction may be inherently sequential: tracing and scanning take time the output of one visual routine may be input to another pipelining may speed things up Can't compute everything; too many combinations. The choice of which operations to apply will depend on the task being performed. What are these operations? Ullman gives 5 examples. 5/18/2018 15-494 Cognitive Robotics

(1) Shift of Processing Focus Attentional operation Determines where in the image the next operation will be applied, e.g.: A particular point A particular contour There is extensive psychological and neurophysiological data on selective attention. 5/18/2018 15-494 Cognitive Robotics

(2) Indexing “Odd man out” phenomenon Easy to find the one element that differs from all the rest But only if it differs in a basic property Indexable properties include: Color, texture Shape, size, orientation Motion Indexing may provide the target for a shift of processing focus. Example task: report the orientation of the red bar in a field of mostly green bars. 5/18/2018 15-494 Cognitive Robotics

Triesman's Visual Search Expt. Find the green letter: X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X 5/18/2018 15-494 Cognitive Robotics

Triesman's Visual Search Expt. Find the O: X X X X X X X X X X X X X X X X X X X X O X X X X X X X X X X X X X X X X X X X 5/18/2018 15-494 Cognitive Robotics

Triesman's Visual Search Expt. Find the green O: X X X O X X O X X O X X X X X X X X X X O X X X O X X X X X X O X X O X X X X X 5/18/2018 15-494 Cognitive Robotics

(3) Bounded Activation (Coloring) Mark a starting point and spread activation outward. Spread is blocked by “boundaries”. Can use this to determine inside/outside relations. What is the subfigure containing the dot? 5/18/2018 15-494 Cognitive Robotics

Bounded Activation in Tekkotsu Using a Sketch<bool> as a boundary: visops::seedfill(index_t point, Sketch<bool> &boundary) visops::fillInterior(Sketch<bool> &boundary) visops::fillExterior(Sketch<bool> &boundary) Using a line shape as a boundary: leftHalfPlane(Shape<LineData> &line) also rightHalfPlane, topHalfPlane, bottomHalfPlane Using a polygon shape as a boundary: isInside(Point p) 5/18/2018 15-494 Cognitive Robotics

(4) Boundary Tracing Trace along the contour until some condition is met. Example: detect open vs. closed curves. Open curves have termination points. Does any curve contain two x's? Contours may not be trivial to recognize: could be broken, or implicit. 5/18/2018 15-494 Cognitive Robotics

(5) Marking Place a marker at a location. Useful for remembering locations or structures that have already been examined. Are any two x's on a common curve? Can also be used to designate a point of interest for later processing. 5/18/2018 15-494 Cognitive Robotics

Points in Tekkotsu fmat::Column<3> or fmat::Column<4> Used internally for arithmetic calculations Point Contains an fmat::Column<3> Also contains a ReferenceFrameType_t Used by shapes for point arithmetic EndPoint Includes valid and active booleans Shape<PointData> 5/18/2018 15-494 Cognitive Robotics

Marking in Tekkotsu Marking a point: Can use a Sketch<bool> with a single pixel set. Can use a Shape<PointData> Marking an object: Can use a Sketch<bool> to show rendering of the object. Can add a shape to a SHAPEVEC 5/18/2018 15-494 Cognitive Robotics

(6) Ray Tracing Not included in Ullman's list. But mentioned in an earlier section of the paper. Start at a point and trace outward in a straight line until you reach something of interest. Which way should the line go? Trace in a particular direction, e.g., “upward”? Trace toward an object of interest? Used by Agre & Chapman in Pengi. 5/18/2018 15-494 Cognitive Robotics

Agre & Chapman's Pengi An AI program that plays the Pengo video game: See videos of the original Pengo arcade game on YouTube. 5/18/2018 15-494 Cognitive Robotics

Visual Routines in Pengi Finding the-block-that-the-block-I-just-kicked-will-collide-with using ray tracing and dropping a marker. 5/18/2018 15-494 Cognitive Robotics

Visual Routines in Pengi Finding the-block-to-kick-at-the-bee when lurking behind a wall. 5/18/2018 15-494 Cognitive Robotics

Visual Routines in Game AI Forbus et al.: visual routines could be used for qualitative spatial reasoning, such as path finding in AI strategy games. Example: Voronoi diagram of open space on a map can be used for route finding. VDdiag(a) = edge(read(labelcc(a), link(a))) 5/18/2018 15-494 Cognitive Robotics

Application to Tekkotsu? Can create sketch spaces for local or world maps. setTmat(scale,tx,ty) controls the mapping of shape space coordinates to sketch space pixels. getRendering() converts shapes to sketches. Marking and coloring can be implemented using sketches. Might use this to implement Pengi-like logic for robotics applications. But we need more primitives... 5/18/2018 15-494 Cognitive Robotics

Do Tekkotsu's Representations Fit Ullman's Theory? What are the base representations? color segmented image: sketchFromSeg() intensity image: sketchFromRawY() depth image: sketchFromDepth() extracted regions What are the incremental representations? Sketches Shapes What's missing? Attentional focus; boundary completion; lots more. 5/18/2018 15-494 Cognitive Robotics

What Do Human Limitations Tell Us About Cognition? Subjects can't do parallel visual search based on the intersection of two properties (Triesman). This tells us something about the architecture of the visual system, and the capacity limitations of the Visual Routines Processor. Base can't do intersection. VRP can't process whole image at once. There must be a limited channel between base and VRP. But in Tekkotsu, we can easily compute intersections of properties. Is that a problem? 5/18/2018 15-494 Cognitive Robotics

Science vs. Engineering Science: figure out how nature works. Limitations of a model are good if they suggest that the model's structure reflects reality. Limitations should lead to nontrivial predictions about comparable effects in humans or animals. Engineering: figure out how to make useful stuff. Limitations aren't desirable. Making a system “more like the brain” doesn't in itself make it better. What is Tekkotsu trying to do? Find good ways to program robots, drawing inspiration from ideas in cognitive science. 5/18/2018 15-494 Cognitive Robotics