I NTELLIGENT A GENTS 1

D EFINITION OF A GENT Anything that: Perceives its environment Acts upon its environment A.k.a. controller, robot 2

D EFINITION OF “E NVIRONMENT ” The real world, or a virtual world Rules of math/formal logic Rules of a game … Specific to the problem domain 3

4 Environment ? Agent Percepts Actions Actuators Sensors

5 Environment ? Agent Percepts Actions Actuators Sensors Sense – Plan – Act

“G OOD ” B EHAVIOR Performance measure (aka reward, merit, cost, loss, error) Part of the problem domain 6

E XERCISE Formulate the problem domains for: Tic-tac-toe A web server An insect A student in B551 A doctor diagnosing a patient IU’s basketball team The U.S.A. 7 What is/are the: Environment Percepts Actions Performance measure How might a “good- behaving” agent process information?

T YPES OF AGENTS Simple reflex (aka reactive, rule-based) Model-based Goal-based Utility-based (aka decision-theoretic, game- theoretic) Learning (aka adaptive) 8

S IMPLE R EFLEX 9 Percept Action Rules Interpreter State

S IMPLE R EFLEX 10 Percept Action Rules

S IMPLE R EFLEX 11 Percept Action Rules In observable environment, percept = state

R ULE - BASED R EFLEX A GENT 12 AB if DIRTY = TRUE then SUCK else if LOCATION = A then RIGHT else if LOCATION = B then LEFT

B UILDING A S IMPLE R EFLEX A GENT Rules: a map from states to action a =  (s) Can be: Designed by hand Learned from a “teacher” (e.g., human expert) using ML techniques 13

M ODEL -B ASED R EFLEX 14 Percept Action Rules Interpreter State Action

M ODEL -B ASED R EFLEX 15 Percept Action Rules Model State Action

M ODEL -B ASED R EFLEX 16 Percept Action Rules Model State Action State estimation

M ODEL -B ASED A GENT 17 AB Rules: if LOCATION = A then if HAS-SEEN(B) = FALSE then RIGHT else if HOW-DIRTY(A) > HOW-DIRTY(B) then SUCK else RIGHT … State: LOCATION HOW-DIRTY(A) HOW-DIRTY(B) HAS-SEEN(A) HAS-SEEN(B) Model: HOW-DIRTY(LOCATION) = X HAS-SEEN(LOCATION) = TRUE

M ODEL -B ASED R EFLEX A GENTS Controllers in cars, airplanes, factories Robot obstacle avoidance, visual servoing 18

B UILDING A M ODEL - BASED R EFLEX A GENT A model is a map from prior state s, action a, to new state s’ s’ = T(s,a) Can be Constructed through domain knowledge (e.g., rules of a game, state machine of a computer program, a physics simulator for a robot) Learned from watching the system behave (system identification, calibration) Rules can be designed or learned as before 19

G OAL -B ASED, U TILITY -B ASED 20 Percept Action Rules Model State Action

G OAL -B ASED, U TILITY -B ASED 21 Percept Action Decision Mechanism Model State Action

G OAL -B ASED, U TILITY -B ASED 22 State Decision Mechanism Action Model Simulated State Action Generator Performance tester Best Action Percept Model

G OAL -B ASED, U TILITY -B ASED 23 State Decision Mechanism Action Model Simulated State Action Generator Performance tester Best Action Sensor Model “Every good regulator of a system must be a model of that system”

B UILDING A G OAL OR U TILITY - BASED A GENT Requires: Model of percepts (sensor model) Action generation algorithm (planner) Embedded state update model into planner Performance metric Model of percepts can be learned (sensor calibration) or approximated by hand (e.g., simulation) 24

B UILDING A G OAL -B ASED A GENT Requires: Model of percepts (sensor model) Action generation algorithm (planner) Embedded state update model into planner Performance metric Model of percepts can be learned (sensor calibration) or approximated by hand (e.g., simulation) Planning using search Performance metric: does it reach the goal? 25

B UILDING A U TILITY -B ASED A GENT Requires: Model of percepts (sensor model) Action generation algorithm (planner) Embedded state update model into planner Performance metric Model of percepts can be learned (sensor calibration) or approximated by hand (e.g., simulation) Planning using decision theory Performance metric: acquire maximum rewards (or minimum cost) 26

B IG O PEN Q UESTIONS : G OAL -B ASED A GENT = R EFLEX A GENT ? 27 Percept Action DM Rules Model State ActionMental Action Mental State Mental Model Physical Environment “Mental Environment”

B IG O PEN Q UESTIONS : G OAL -B ASED A GENT = R EFLEX A GENT ? 28 Percept Action DM Rules Model State ActionMental Action Mental State Mental Model Physical Environment “Mental Environment”

W ITH L EARNING 29 Percept Action Decision Mechanism Model/Learning Action State/Model/DM specs

B UILDING A L EARNING A GENT Need a mechanism for updating models/rules/planners on-line as it interacts with the environment Need incremental techniques for machine learning More next week… 30

B IG O PEN Q UESTIONS : L EARNING A GENTS The modeling, learning, and decision mechanisms of artificial agents are tailored for specific tasks Are there general mechanisms for learning? If not, what are the limitations of the human brain? 31

T YPES OF E NVIRONMENTS Observable / non-observable Deterministic / nondeterministic Episodic / non-episodic Single-agent / Multi-agent 32

O BSERVABLE E NVIRONMENTS 33 Percept Action Decision Mechanism Model State Action

O BSERVABLE E NVIRONMENTS 34 State Action Decision Mechanism Model State Action

O BSERVABLE E NVIRONMENTS 35 State Action Decision Mechanism Action

N ONDETERMINISTIC E NVIRONMENTS 36 Percept Action Decision Mechanism Model State Action

N ONDETERMINISTIC E NVIRONMENTS 37 Percept Action Decision Mechanism Model Set of States Action

A GENTS IN THE BIGGER PICTURE Binds disparate fields (Econ, Cog Sci, OR, Control theory) Framework for technical components of AI Decision making with search Machine learning Casting problems in the framework sometimes brings insights Search Knowledge rep. Planning Reasoning Learning Agent Robotics Perception Natural language... Expert Systems Constraint satisfaction

U PCOMING T OPICS Utility and decision theory (R&N ) Reinforcement learning Project midterm report due next week (11/10) 39