1. If you have any questions please feel free to interrupt me

2. The vision system for Marie Curie

3. Main Tasks in our system
Image recognition
Machine Learning
Control of robot’s behavior
Environment

4. Marie Curie will be communicating with Schroedinger’s Cat robot

5. Interaction of Marie Curie and Cat
Marie Curie does not change its main coordinates, she can only move her hands, head and legs, but she remains attached to the desk.
Cat can move freely in the area of the stage.
Cat should not bump into Marie or furniture of the lab.
Marie should know where the cat is located and look to him
Cat should know where Marie is located and talk to her.

6. Ideal view of the ceiling camera
Black curtain
Cat
shelf
equipment
Marie Curie
equipment
equipment

7. There will be a Kinect camera looking from the ceiling to the stage

8. The role of the ceiling camera
The camera will be attached to the ceiling or will be in some position very high, as high as we can.
We have done something similar but the robots were small.
The camera should know x,y coordinates of every robot and its orientation (pose)
Marie Curie does not change its main coordinates, so it is easy
Cat is fast so we have to track the triplet
(x, y, )

9. There is nothing like that in Disneyland
All behaviors of robots in commercial theatres are strictly scripted, robots move on rails, they cannot make an error.
In our case we have interaction, improvisation, and robots are subject to noisy behaviors.
This task is more similar to robot soccer than to existing robot theatres in the world.

10. There will be another camera looking to faces of the audience
We will call it the human-control camera or a front camera

11. The role of the front camera
The camera is attached to the wall near the glass window of the theatre, looking towards humans located in the corridor.
This camera will look at the audience
There are several goals of having this camera
Recognizing (x, y, ) of every person that looks at the performance (perhaps not more than 5).
Recognizing the emotion on the faces of these people.
Recognizing their gestures with hands and legs, full bodies and faces.
Use these data to control the behavior of the robots, songs selected, slides selected, lights and other effects.

12. There is nothing like that in Disneyland
All behaviors of robots in Disneyland are strictly scripted.
Rarely humans can change robots’ behaviors.

13. This is a new task for our team
Marek Perkowski has never done anything like this before
Perhaps nobody in the world has done something like this.
This is good as we are doing something new.
Hopefully we have done something similar and have a good experience from the past.
We were doing ROBOT SOCCER.
We will try now to use our past experience and theory for a new task.

14. Ideal view of the ceiling camera
Black curtain
Cat
shelf
equipment
Marie Curie
equipment
equipment

15. This is ideal, in reality the image will be much distorted with noise and lightning and geometry
Y axis
Black curtain c
Cat Yc Xc
shelf
equipment
Marie Curie
equipment
equipment
X axis

16. The idea of Robot Soccer

17. 3. Robot Soccer and Similar Tasks
• Robot Soccer Competition
– RoboCup
– FIRA
– Remote controlled systems
– Autonomous robots
• Clustering

18. 3.1 Robot Soccer
“RoboCup is an international joint project to promote AI,
robotics, and related fields.
It is an attempt to foster AI and
intelligent robotics research by providing a standard problem
where a wide range of technologies can be integrated and
examined.
RoboCup chose to use the soccer game as a central
topic of research, aiming at innovations to be applied for
socially significant problems and industries.
The ultimate goal
of the RoboCup project is: By 2050, develop a team of fully
autonomous humanoid robots that can win against the human
world champion team in soccer.” [RoboCup 1998]

20. As you see, it is difficult to approximate every robot with a rectangle.
It will be even more difficult in our case.

22. Overhead Vision
Our goal is to start with Overhead Vision (Ceiling Camera) and check how it will work.
We may move to more cameras if necessary.

23. Local Vision

24. Design Criteria for robot soccer
• Controller Hardware: Enable on-board image processing
– Interface to digital camera
– Incorporate graphics LCD
– Incorporate user buttons
– Wireless communication between robots
• Sensors: Allow variety of additional sensors:
– Shaft encoders
– Infra-red distance measurement sensors
– Compass module
• Software: Flexibility to accommodate for different robot equipment
– Operating system RoBIOS
– Hardware description table HDT

25. What is AI? Research in Artificial Intelligence (AI) includes:
design of intelligent machines
formalization of the notions of intelligence and rational behavior
understanding mechanisms of intelligence
interaction of humans and intelligent machines.

26. Objectives of AI Engineering : costruct intelligent machines
Scientific : understand what is intelligence.

27. A team of Robots will beat the FIFA World Cup champions by 2050!
Can a robot do these?
Understand?
Simulate its environment?
Act rationally?
Collaborate and compete?
Display emotions?
A bold claim:
A team of Robots will beat the FIFA World Cup champions by 2050!

28. RoboCup - Aim ”pushing the state-of-the-art”
”By mid-21st century, a team of fully autonomous humanoid robot soccer players shall win the soccer game, comply with the official rule of the FIFA, against the winner of the most recent World Cup.
TO BOLDLY GO WHERE MAN HAS GONE BEFORE (cf. Star Trek)
Formalised Testbed

29. Do you really believe that a team of Robots could beat the FIFA World Cup champions by 2050?
By all accounts this may sound overly ambitious.
In fact, if you compare this goal to other ground breaking achievements it is not ambitious at all.
The Wright brothers' first airplane was launched and 50 years later man landed on the moon.
Even more recently Deep Blue the computer programmed to play chess, played chess grand master Garry Kasparov and won -- roughly 50 years after the deployment of the first computer.
It's a long time.
Think what has happened since 1950.

30. Power of AI Is the following AI?
In 1997 a computer, Deep Blue, won a chess match with world champion Kasparov.
Accident?
IBM paid Kasparov to loose?
Brute force with no intelligence?
So, what is intelligence?

31. Simulation
Turing test (1950)

32. Chess versus soccer robot
Distributed
Central
Control
Non-symbolic
Symbolic
Sensor Readings
Incomplete
Complete
Info. accessibility
Real time
Turn taking
State Change
Dynamic
Static
Environment
RoboCup
Chess
Difference of domain characteristics between computer chess and soccer robots

33. Intelligent Agents Agents are situated
Perception of environment
Execution of actions
Agents can communicate and collaborate
they can differ
than can compete and be more or less egoistic/altruistic
The agents have:
objectives,
communications,
intentions.

34. Professor Kim from KAIST
A New Approach
Professor Kim from KAIST
The founder of Robot Soccer and FIRA president
Two organizations:
1. FIRA (earlier)
2. RoboCup (larger)

35. Four Blocks in two PCBs (Printed Circuit Boards)
Micro-controller (upper PCB)
Communication module (upper PCB)
Motor and driving circuits (lower PCB)
Power (lower PCB)
side view
top view
front view

36. Importance of Robot Soccer
Communication
Cooperation
Coordination
Learning
Competence
Real Time
Robot Soccer Evolution
Computer simulations
Wheeled brainless robots
Wheeled autonomous robots
Legged autonomous robots

37. Robot Soccer Purpose
“The number one goal of [robot soccer] is not winning or losing, but accumulating diverse technology.”
- Mr. Dao (Senior VP of Sony Corporation).

38. MiroSot FIRA category 3 robots on 1 team Size : 7.5cm * 7.5cm * 7.5cm
Ball : orange golf ball
Playground : black wooden rectangular playground
(150cm * 130cm * 5cm)
Vision : global vision system
(more than 2m above playground)

39. Experimental Setup of the Vision System
Control panel

40. NaroSot FIRA category 5 robots on 1 team Size : 4cm * 4cm * 5.5cm
Ball : orange table-tennis ball
Playground , Vision : same as Mirosot

41. KheperaSot FIRA category 3 robots on 1 team Ball : yellow tennis ball
Playground : green playground (105cm * 68cm * 20cm)
Robot : Khepera Robot
Vision : K213 Vision Turret

42. RoboSot FIRA category 3 robots on 1 team Size : 15cm * 15cm * 30cm
Ball : red roller-hockey ball
Playground : black wooden rectangular playground (220cm * 150cm * 30cm)
Vision : on the robot
Under preparation

43. Small-Size League

44. Small-Size League (F-180)
Field: 2.7 m x 1.5 m
Size
Area : 18cm rule (fit inside in 18cm diameter cylinder)
Height : 15cm (global vision), 22.5cm (otherwise)
teams of autonomous small size robot play soccer game on a field equivalent to a ping-pong table.
Each team consists of 5 robots.

45. Small size league
The field is the size and color of a Ping Pong table

46. Robots move at speeds as high as 2 meters/second
orange golf ball
Robots move at speeds as high as 2 meters/second
Global vision is allowed

47. Robot Soccer Initiative
Vision system
Host computer
Communication System
Host computer
Communication System
Robots on the playing field
“Brainless” System
Basic Architecture for Robot Soccer Systems

49. Vision System
Vision : global vision system (more than 3m above ground)
Each team has its own camera and PC

50. Small-Size League
20 minutes, 2 breaks

51. Real Robot Small-Size League Competition

52. Middle-Size League

53. Middle-size Real Robot League (F-2000): Local VISION
The field is the size and color of a 3 x 3 arrangement of Ping Pong tables (9-3 5-meter field)
Each team consists of 5 robots playing with a Futsal-4 ball (4 players, one goal-keeper)
Larger (50 centimeters in diameter) robots
Global vision is not allowed.
Each robot has its own vision system
Goals are colored
Field is surrounded by walls to allow for distributed localization through robot sensing
Rule structure based on the official FIFA rules

54. Medium size league
Teams of autonomous mid size robots

55. Real Robot Middle-Size League Competition
Ball : red small soccer ball (FIFA standard size 4 or 5)
Playground : green playground (10m * 7m * 0.5m)

56. Medium Size League

57. Medium Size League

60. Robots can be heterogenous

61. Middle-Size League

62. Sony Legged Robot League

63. Sony Legged Robot League
3 robots on 1 team (including the goalkeeper).
Robot : AIBO ERS-110 (provided by Sony)

64. No communication, autonomous robots, software only. Legged Robot League. 2.8 m x 1.8 m 2 players and 1 goal-keeper in a team

65. Sony Legged Robot League
Is played on a field, approx 3x2 meter
Sony develops the robots, and provides a interface for the programming of the robots.

66. No Hardware modification is allowed
Playing time is 10 minutes per half, with a 10 minute break at halftime

67. Do different Robots have different personalities?
Some teams have robots with very different capabilities.
But it is hard to think of them as having personalities;
rather the robots have different playing styles.

69. Early Sony prototype

70. Robot movements closely mirror those of animals

71. The winner is the team that scores the most goals.
In the event of a tie, a sudden death penalty kick competition will determine the winner

73. The Legged Robot League

74. The Legged Robot League
If opposing teams' robots are damaged or play is excessively rough (whether intentional or not), penalties may be assessed to the offending robot

75. Humanoid League

76. Starting 2002, the humanoid league

77. Humanoid League
Bi-Ped League (Humanoid)
Australia
Japan

79. Where is the science in these robot competitions?
Global vision
Local vision
Other sensors
Cooperation
Sensor fusion
Strategy
Learning

80. Sensors and Actuators for Robot Soccer
Local and Global VISION

81. Sensors for Robot Soccer
• Shaft Encoders
– PI controller to maintain wheel speed
– PI controller to maintain path curvature
– Dead reckoning for vehicle position + orientation
• Infrared Distance Measurement
– Avoid Collision
– Navigate and map unknown environment
– Update internal position in known environment
• Compass
– Update orientation independent of shaft encoders
– Fault-tolerance in case robot gets pushed or wheels slip

82. Sensors for Robot Soccer
• Digital Camera
– Low resolution, 60x80 pixels, 24bit color (Braunl)
– Color or shape recognition
• Communication
– Sharing information among robots
– Receiving commands from human operator

83. VISION: Color Detection
• In robot soccer, objects are color coded:
ball,
goals,
opponents,
team mates,
walls, etc.
Teach ball and goal color (hue) before starting the game
Match colors in HSI space
→ Better in changing lighting conditions

84. Brain-on-board system
Role of Vision
Brain-on-board system
Robots
The robots have functions such as velocity control, position control, obstacle avoidance, etc.
Host computer
The host computer processes vision data and calculates next behaviors of robots according to strategies and sends commands to the robots using RF modem.

85. 2.2 Robot-based system Distributed system
Intelligent part is implemented in the robots.
Suitable when the large number of agents exist
Complex and expensive
Need communication among robots

86. Role of vision Robot-based system Robots Host computer
The robots decide their own behavior autonomously using the received vision data, own sensor data and strategies.
Host computer
The host computer processes only vision data
can be considered as a kind of sensor.

87. System Comparisons Merits Demerits Research purpose Vision system
Multi-agent theory
Cannot use local sensors
High computing power & fast sampling time
Remote-brainless system
Low cost
Easy to develop
Robot system
Multi-agent system development
Robot -based
system
Complex and expensive robots.
Hard to build the system
Suitable for many agents
Can use local information
Robot-based and vision-based systems
Brain-on-board system
Suitable to modularize
Risk of inconsistent property between host computer and robot system

88. VXD: role of color Initialization Configuration
Click ‘Load VXD’ in the Initialize group box
Click ‘Start Grab’
Configuration
‘Load Conf.’: load a configuration file
‘Save Conf.’: save current configuration
‘Set Robot Size’: set the robot size in number of pixels
‘Set Pixel Size’: set the size of each color (ball, team,
robot, opponent) patch in number of pixels
‘Set Boundary’: set the field boundary on the screen
‘Change Color’: change the color setting of each color patch
‘Set Color’: set the range of tolerance of each color

89. Subsystems and Vision Serial Port Home Goal Find Objects
Select the serial communication port
Home Goal
Select the home side on the screen
Find Objects
Check the box of which you like to find on the field
Initial Position: tell the vision system the initial position
of each object
E.g.) for the ball
i) turn on the radio button of ‘Ball’
ii) place the mouse on the ball and press the left button
Repeat above procedure for another object

90. Commands for Vision Select Situation Command
The situation in which the game is about to start
Command
Click ‘Ready’: the vision system starts finding the objects
on the field
Click ‘Start’ : the vision system starts sending commands
to the robots
Click ‘Stop’ : the vision system stops finding objects
and sending commands

91. Tasks for us How to organize the ceiling camera system?
How to describe (learn?) the shapes of robots?
How to find the (x,y,) for each robot?
How to modify the scripted behavior when the triplets for each robot are known?
How to design the interactive behaviors?

92. Task for Robot Theatre Team
For next week
Write a half-page essay about the vision system that we discussed today.
Use you knowledge from other lecture of today.
Add your imagination and crazy ideas about what the robots should see and know for our particular scene of Marie Curie and Schroedinger’s Cat.