1. Learning in Agents Material collected, assembled and extended by S. Costantini, Computer Sc. Dept. Univ. of L’Aquila Many thanks to all colleagues that share teaching material on the web.

2. Why Learning Agents? Designers cannot foresee all situations that the agent will encounter. To display full autonomy agents need to learn from and adapt to novel environ-ments. Learning is a crucial part of intelligence.

3. What is Machine Learning? Definition: A computer program is said to learn from experience E with respect to some class of tasks T and perform-ance measure P, if its performance at tasks in T, as measured by P, improves with experience E. [Mitchell 97] Example: T = “play tennis”, E = “playing matches”, P = “score”

4. ML (machine learning): Another View ML can be seen as the task of: taking a set of observations represented in a given object/data language and representing (the information in) that set in another language called concept/hypothesis language. A side effect of this step – the ability to deal with unseen observations.

5. When an agent learns: The range of behaviors is expanded: the agent can do more, or The accuracy on tasks is improved: the agent can do things better, or The speed is improved: the agent can do things faster.

6. Machine Learning Biases The concept/hypothesis language specifies the language bias, which limits the set of all concepts/hypotheses that can be expressed/considered/learned. The preference bias allows us to decide between two hypotheses (even if they both classify the training data equally). The search bias defines the order in which hypotheses will be considered. Important if one does not search the whole hypothesis space.

7. Concept Language and Black- vs. White-Box Learning Black-Box Learning: Interpretation of the learning result is unclear to a user. White-Box Learning: Creates (symbolic) structures that are comprehensible.

8. Machine Learning vs. Learning Agents Classic Machine Learning Machine Learning: Learning as the only goal Active Learning Learning as one of many goals: Learning Agent(s) Closed Loop Machine Learning

9. Integrating Machine Learning into the Agent Architecture Time constraints on learning Synchronisation between agents’ actions Learning and recall Timing analysis of theories learned

10. Time Constraints on Learning Machine Learning alone: predictive accuracy matters, time doesn’t (just a price to pay) ML in Agents Soft deadlines: resources must be shared with other activities (perception, planning, control) Hard deadlines: imposed by environment: Make up your mind now!

11. Learning and Recall Agent must strike a balance between: n Learning, which updates the model of the world n Recall, which applies existing model of the world to other tasks

12. Learning and Recall (2) Update sensory information Recall current model of world to choose and carry out an action Learn new model of the world In theory, the two can run in parallel In practice, must share limited resources

13. Learning and Recall (3) Possible strategies: n Parallel learning and recall at all times n Mutually exclusive learning and recall –After incremental, eager learning, examples are discarded… –…or kept if batch or lazy learning used n Cheap on-the-fly learning (preprocessing), off-line computationally expensive learning –reduce raw information, change object language –analogy with human learning and the role of sleep

14. Types of Learning Task Supervised Learning: there is a “teacher” Unsupervised Learning: autonomous Reinforcement Learning: the agent is given a (usually pre-defined) reward if the knowledge coming from learning proves useful for reaching agent’s goals.

15. Learning to Coordinate Good coordination is crucial for good MAS performance. Example: soccer team. Pre-defined coordination protocols are often difficult to define in advance. Needed: learning of coordination Idea: use reinforcement learning

16. Soccer Formation

17. Soccer Formation Control Formation control is a coordination problem. Good formations and set-plays seem to be a strong factor in winning teams. To date: pre-defined. Can (near-)optimal formations be (reinforcement) learned? New idead are being experimented...

18. Learning as Knowledge Extraction Extracting useful patterns from data: Data Mining

19. Data Mining Taxonomy Predictive Method - …predict the value of a particular attribute… Descriptive Method - …foundation of human-interpretable patterns that describe the data…

20. Definition of Data Mining “…The non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data…” Fayyad, Piatetsky-Shapiro, Smyth [1996]

21. Overview Introduction Data Mining Taxonomy Data Mining Models and Algorithms Quick Wins with Data Mining Privacy-Preserving Data Mining

22. Classification & Regression Classification: …aim to identify the characteristics that indicate the group to which each case belongs… Two Crows Corporation Regression: …uses existing values to forecast what other values will be… Two Crows Corporation

23. Clustering & Association Clustering: …divides a database into different groups… …find groups that are very different from each other, with similar members…. Two Crows Corporation Association: …involve determinations of affinity-how frequently two or more things occur together… Two Crows Corporation

24. Deviation Detection & Pattern Discovery Deviation Detection: …discovering most significant changes in data from previously measured or normative values… V. Kumar, M. Joshi, Tutorial on High Performance Data Mining. Sequential Pattern Discovery: …process of looking for patterns and rules that predict strong sequential dependencies among different events… V. Kumar, M. Joshi, Tutorial on High Performance Data Mining.

25. Overview Introduction Data Mining Taxonomy Data Mining Models and Algorithms Quick Wins with Data Mining Privacy-Preserving Data Mining

26. Data Mining Models & Algorithms Neural Networks Decision Trees Rule Induction K-nearest Neighbor Logistic regression Discriminant Analysis

27. Neural Networks - efficiently model large and complex problems; - may be used in classification problems or for regressions; - Starts with input layer => hidden layer => output layer 1 2 3 4 5 6 Inputs Output Hidden Layer

28. Neural Networks (cont.) - can be easily implemented to run on massively parallel computers; - can not be easily interpret; - require an extensive amount of training time; - require a lot of data preparation (involve very careful data cleansing, selection, preparation, and pre-processing); - require sufficiently large data set and high signal-to noise ratio.

29. Decision Trees (cont.) - handle very well non-numeric data; - work best when the predictor variables are categorical;

30. Decision Trees -a way of representing a series of rules that lead to a class or value; -basic components of a decision tree: decision node, branches and leaves; Income>40,000 Job>5 High Debt Low Risk High Risk High Risk Low Risk NoYes NoYes No

31. Rule Induction - method of deriving a set of rules to classify cases; - generate a set of independent rules which do not necessarily form a tree; - may not cover all possible situations; - may sometimes conflict in their predictions.

32. K-nearest neighbor - decides in which class to place a new case by examining some number of the most similar cases or neighbors; - assigns the new case to the same class to which most of its neighbors belong; X X x X Y x X N X X Y

33. Artificial Neural Networks

34. Introduction What is neural computing/neural networks? The brain is a remarkable computer. It interprets imprecise information from the senses at an incredibly high speed.

35. Introduction A good example is the processing of visual information: a one-year-old baby is much better and faster at recognising objects, faces, and other visual features than even the most advanced AI system running on the fastest super computer. Most impressive of all, the brain learns (without any explicit instructions) to create the internal representations that make these skills possible

36. Biological Neural Systems The brain is composed of approximately 100 billion (10 11 ) neurons Schematic drawing of two biological neurons connected by synapses A typical neuron collects signals from other neurons through a host of fine structures called dendrites. The neuron sends out spikes of electrical activity through a long, thin strand known as an axon, which splits into thousands of branches. At the end of the branch, a structure called a synapse converts the activity from the axon into electrical effects that inhibit or excite activity in the connected neurons. When a neuron receives excitatory input that is sufficiently large compared with its inhibitory input, it sends a spike of electrical activity down its axon. Learning occurs by changing the effectiveness of the synapses so that the influence of one neuron on the other changes

37. What is a Neural Net? A neural net simulates some of the learning functions of the human brain. It can recognize patterns and "learn." You can use it to forecast and make smarter business decisions. It can also serve as an "expert system" that simulates the thinking of an expert and can offer advice. Unlike conventional rule-based artificial-intelligence software, a neural net extracts expertise from data automatically - no rules are required. In other words through the use of a trial and error method the system “learns” to become an “expert” in the field the user gives it to study.

38. Components Needed: In order for a neural network to learn it needs 2 basic components: Inputs Which consists of any information the expert uses to determine his/her final decision or outcome. Outputs Which are the decisions or outcome arrived at by the expert that correspond to the inputs entered.

39. How does a neural network learn? A neural network learns by determining the relation between the inputs and outputs. By calculating the relative importance of the inputs and outputs the system can determine such relationships. Through trial and error the system compares its results with the expert provided results in the data until it has reached an accuracy level defined by the user. With each trial the weight assigned to the inputs is changed until the desired results are reached.

40. Artificial Neural Networks Artificial neurons are analogous to their biological inspirers Here the neuron is actually a processing unit, it calculates the weighted sum of the input signal to the neuron to generate the activation signal a, given by An artificial neuron where w i is the strength of the synapse connected to the neuron, x i is an input feature to the neuron

41. Artificial Neural Networks The activation signal is passed through a transform function to produce the output of the neuron, given by The transform function can be linear, or non-linear, such as a threshold or sigmoid function [more later …]. For a linear function, the output y is proportional to the activation signal a. For a threshold function, the output y is set at one of two levels, depending on whether the activation signal a is greater than or less than some threshold value. For a sigmoid function, the output y varies continuously as the activation signal a changes.

42. Artificial Neural Networks Artificial neural network models (or simply neural networks) are typically composed of interconnected units or artificial neurons. How the neurons are connected depends on some specific task that the neural network performs. Two key features of neural networks distinguish them from any other sort of computing developed to date: Neural networks are adaptive, or trainable Neural networks are naturally massively parallel These features suggest the potential for neural network systems capable of learning, autonomously improving their own performance, adapting automatically to changing environments, being able to make decisions at high speed and being fault tolerant.

43. Neural Network Architectures Feed-forward single layered networks Feed-forward multi-layer networks Recurrent networks

44. Neural Network Applications Speech/Voice recognition Optical character recognition Face detection/Recognition Pronunciation (NETtalk) Stock-market prediction Navigation of a car Signal processing/Communication Imaging/Vision ….