1. Neurophysiology of Grasping
Ashley Kleinhans
Background

2. How do we get robots to grasp as well as humans?
The problem
How do we get robots to grasp as well as humans?
What is the research problem
I think for most in robotics it’s a dream to be able to have a robot pet. For some even girlfriend.

3. This is an image of a production plant body shop
This is an image of a production plant body shop. It takes about 2 years to set up this system, but time to implementation is shorter where there are more available skilled workers.

4. In south Africa this isn't always the case, so the time is longer, meaning the initial capital costs are higher. Reducing this timing is important for a production environment.
Technology push
Other uses

5. Why do we care about this?
What about for our home environment?
Tasks like cleaning your room, finding your keys or mobile phone.

6. Extra hands

7. Or just hands

8. How about the option of letting your house know that you are late for a meeting and you need a shower but you know the geyser needs to be switched on, so you pull up an app that switches on a geyser and your home helper ensures that your bath has been run buy the time you get home.

9. Or you are buggered after a long day and you have a cool glass of wine waiting for you – having just been poured – on the counter.

10. Why is robot grasping hard?
Fournier, et al. 2011
Why is robot grasping hard?
We can begin with the tools we are given
Human hand vs robot hand
Shadow hand comes close – compliancy and DOF
Other options
Option for every kind of task – not very efficient

11. Why is robot grasping hard?
Sensory feedback and noise

12. Understanding task relation
Understanding learning – all types of brooms

13. What are the kind of things people look at?
Boris
Birmingham (Wyatt)
Research
Controlled
Cornell. (Saxena)
Lab setup
Complex shapes
research realm
Lab environment
Controlled
Primitive considered solved
Complex close to being solved – in research terms
state of the art when it comes to dynamic environments

14. Touch
Task
Current research
Touch

15. Task based grasping - CMU

16. Why do none of these things work?
Break it down into step by step
Use the best most accurate sensory input
Out design the shit out of the tools

17. but humans can do amazing things, even when their hand is replaced by a hook

18. We could argue a noisy world
What you see

19. What your brain sees
Noisy sensory input
Fournier, et al. 2011

20. What I am trying to do This is my goal..
I am trying to marry the concept of human and robot grasping
Looking at a system that works ..

21. What works? This is what works This is how we gather information
What is available

22. Pause
I know what I want
I know where to find it
I know what works
We can’t experiment on humans
Now how do we build this?

23. Similar in tasks
Not cognition

24. The network (questions that keep me up)
DORSAL STREAM
VENTRAL STREAM
Take a closer look at the tool that evolution has provided us
Macaque monkey brain
What we know
Dorsal and Ventral

25. The network (questions that keep me up)V3
Depth
V3 sensitive to depth information

26. Curvature and Orientation
The network (questions that keep me up)
The network
CIP
Curvature and Orientation V3
Depth
CIP to curvature and orientation

27. Curvature and Orientation
The network (questions that keep me up)
The network
AIP
Shape & Size
CIP
Curvature and Orientation V3
Depth
AIP to shape and size

28. Curvature and Orientation
The network (questions that keep me up)
The network
AIP
Shape & Size
CIP
Curvature and Orientation V3
Depth F5
Hand
Somewhere between F5 and AIP there is a hand transformation that happens so that the robot hand can be controlled

29. V3 V3 V3
Looking at V3 and how it sees depth by stereopsis

30. Stereopsis example
Place your finger in front of your face
Close one eye then the other
The difference that you see is the method the brain uses to gauge depth

31. Depth
Modelling depth in a simulated environment

32. CIP
CIP
CIP
What we know
Higher up we go things are encoded differently
CIP

33. CIP Receives input from V3 and V3a
Depth, 1st and 2nd spatial derivatives
[S. C. de Vries, A. M. L. Kappers, and J. J. Koenderink,
“Shape from stereo: A systematic approach using quadratic surfaces,”
Perception & Psychophysics, vol. 53, no. 1, pp. 71–80, Jan
Modelling our stimulus
All the maths is are equations for the surfaces we use as stimuli
In the experiments these stimuli are placed in front of a Monkey, on a screen, and neurons are recorded from

34. AIP The network AIP Macaque monkey brain AIP – shape and size
Shape and size of objects can be broken down in a number of ways
We looked at two options for surfaces

35. Modelling our stimulus (Superquadrics)
The first is superquadrics
5 – 11 variables for parameterisation
Superquadrics generalize the basic quadric surfaces
and solids
Map to robotic end-effector shapes
Alan H. Barr. 1981
Kindlmann, 2004

36. Isomap – more biologically plausible
Isomap is more biologically plausible because the structure is inherent in the data, rather than designed.
Extends Multi-dimensional scaling or kernel PCA
Seeks lower-dimensional embedding which maintains geodesic distances between points
Three stages
Nearest neighbour search
Shortest path search
Partial eigenvalue decomposition

37. F5 The network F5 Very interesting area Pre-motor or not to pre-motor?
Understanding of the hand – perceptual understanding of where your hand is in its environment
F5ab (F5c mirror)
This is the area of the mirror Fc – it’s the part of the brain that allows you to see what someone else is like and repeat it exactly.

38. F5
Challenge
Mapping
DOF
So where does this leave me?

39. Designing my network
I have to figure out how to design a network that will provide me the same functionality as the brain does.. This means looking at each area discussed and creating a neural network that takes a depth image input and outputs a grasp configuration..

40. Questions?