1. Towards a Universal Test of Social Intelligence PhD Thesis Javier Insa Cabrera Valencia, SpainMay 16, 2016 Supervisor: José Hernández Orallo

2. 2 Motivation Test

3. Introduction Background Universal Test to Evaluate General Intelligence Extending a General Intelligence Test to Evaluate Social Intelligence Defining Social Intelligence Universally Experimental Analysis for Several Types of Environments and Agents Properties About Social Intelligence Testbeds Characterising Several Multi-Agent and Social Scenarios Conclusions and Future Work 3 Outline

4. 4 Introduction Context Evaluation tools are crucial in any discipline as a way to assess its progress and creations. AI as a paradigmatic case. Late 1990, evaluation of agent intelligence in a principled and general way. Universal test for the evaluation of intelligence for any kind of agent. Preliminary experiments: The test does not place them on the same scale. AI is now more focussed towards social intelligence. Influenced from the vision of human intelligence as highly social.

5. 5 Introduction Goal Social intelligence as the ability to perform well in the context of other agents. We hypothesise social intelligence is different from general intelligence. Evaluation of social behaviours. We understand it as performance interacting with several agents populating a selection of environments. Evaluation of social intelligence is not ensured.  Ignore the rest of agents.  The environment more focussed on solving a certain problem.

6. Analyse the possibility of properly evaluating social intelligence universally. Interaction with other (social) agents in a (social) environment. Evaluating its both cooperative and competitive social intelligence. Attempt to distinguish between general and social intelligence. Scenarios where interaction with other agents has an impact in performance. Assess the suitability of environments and agents populating them. Provide social property models they should meet. 6 Introduction Objectives of this thesis

7. Interaction 7 Background Evaluation

8. Reinforcement Learning Perform actions in an environment to maximise the rewards it obtains. Q-learning  Tries to learn an action-value function: expected utility of performing an action in a state.  Update:  Behaviour: 8 Background Agents

9. 9 Matching Pennies Pac-Man Prisoners’ Dilemma RoboCup Soccer Predator-Prey Background Environments (Tasks)

10. Kolmogorov Complexity Universal Distribution Gives a probability to all elements in an infinite set. 10 Background Difficulty of Tasks Length of the shortest program that describes the object

11. Analyse how a general intelligence test performs when evaluating social intelligence. Universal Intelligence Universal distribution to include such infinite set. 11 Universal Test to Evaluate General Intelligence Definition Performance interacting in the set of all environments

12. Construct a universal and anytime intelligence test. Any kind of system or species. Any moment in its development. Any degree of intelligence. Any speed. Evaluation can be stopped at any time. Tries to solve problems about the Universal Intelligence definition. 12 Universal Test to Evaluate General Intelligence Test Framework

13. Lambda Environment Class Spaces are defined as strongly connected graphs.  Observations are the ‘contents’ of each vertex/cell in the graph.  Example:  Actions are the arrows in the graphs.  Agents can perform actions inside the space.  Rewards: Two special agents, Good ( ⊕ ) and Evil ( ⊖ ), responsible for the rewards.  Rewards are fading, leaving a trail. 13 Universal Test to Evaluate General Intelligence Environments

14. With the test definitions and this environment class, the ‘general intelligence’ of different systems were evaluated. Experiments concluded that the test prototype is not universal.  Different systems (humans and AI agents) obtained similar results. Seems that some abilities do not have enough importance:  Incremental knowledge acquisition.  Environments rarely contain social behaviour. 14 Universal Test to Evaluate General Intelligence Experiments

15. We extend the test to consider several agents Analyse the behaviour of intelligence tests when environments are populated with agents, and how their results are affected. 15 Extending a General Intelligence Test to Evaluate Social Intelligence Goal

16. Simple algorithm Random Reinforcement Learning algorithms: Q-learning SARSA QV-learning Evaluated in isolation 16 Extending a General Intelligence Test to Evaluate Social Intelligence Agents

17. All the agents compete for rewards. 17 Extending a General Intelligence Test to Evaluate Social Intelligence Competition

18. The agents receive the average of obtained rewards. 18 Extending a General Intelligence Test to Evaluate Social Intelligence Cooperation

19. Competition and cooperation. Two teams (Q-learning vs SARSA) compete for rewards. 19 Extending a General Intelligence Test to Evaluate Social Intelligence Teams

20. Including other agents (even random) make other agents perform worse. RL algorithms have to deal with too much information. Algorithms should learn to deal with ‘noise’. Difficulty increases with the inclusion of other agents. The complexity is also related to the complexity of the other agents. We need to calculate first the complexity (or intelligence) of the other agents. First idea about when evaluating an agent surrounded with other agents. First steps for a social intelligence test making the evaluated agent interact with other agents, using for this a multi-agent system. 20 Extending a General Intelligence Test to Evaluate Social Intelligence Conclusions

21. Our desired requirements for a social intelligence test: Evaluates social intelligence for all kinds of interactive system or species. Its measurement can be directly obtained from the definition. The evaluee has to interact in the environment with other agents. Performance affected when compete and cooperate with socially intelligent agents. Role of other agents with respect to the evaluee. Other requirements that every test should have: Finite procedure. Applicable in a reasonable period of time. Discriminate with respect to the overall result. Etc. 21 Defining Social Intelligence Universally Requirements

22. Universal definition of social intelligence. Generalise Thorndike’s definition with the variety of species and machines. Social intelligence is a relative property. Specify the other agents and the environments. We understand an agent is socially intelligent when performing better in social environments. We do not consider other traits, such as being sociable. 22 Defining Social Intelligence Universally Universality The ability to understand and manage men and women, boys and girls, to act wisely in human relations Performance of an agent while interacting with other agents in a wide range of environments

23. Role of the other agents. Their goals are in favour or against the evaluee’s goal. Cooperative social intelligence is the capability to obtain the best performance in a multi-agent environment where other agents share the same rewards. Competitive social intelligence is the capability to obtain the best performance in a multi-agent environment where other agents compete for the same rewards. 23 Defining Social Intelligence Universally Relations of Agents

24. 24 Defining Social Intelligence Universally Teams and Agents’ Setup

25. 25 Defining Social Intelligence Universally Teams and Agents’ Setup

26. Fixing a line-up and varying the environment: Fixing an environment and varying the line-up: Varying the environment and the line-up: This is a skeleton for the definition. 26 Defining Social Intelligence Universally Social Intelligence Definition

27. 27 Exercise Defining Social Intelligence Universally Test Definition

28. 28 Experimental Analysis for Several Types of Environments and Agents Goal

29. To evaluate and populate RandomSelects its actions using a uniform distribution QLSelfishSimilar to Q-learning’s behaviour without teams QLCommunalFocuses on improving the average reward of its team QLMercifulFocuses on improving its teammates’ rewards QLHarmfulFocuses on worsening rewards of agents in other teams We consider agents encouraging its team rewards to be cooperative, while agents concerned on their own or bothering others to be competitive. We train them in the scenario they are going to interact before evaluating them. 29 Coop. Comp. Experimental Analysis for Several Types of Environments and Agents Agents

30. Prisoner’s Dilemma (3 players version) Rewards: [0, 9] Lambda Environment Rewards: [-1, 1] Predator-Prey (predator slots) Rewards: [-1, 1] Distribute rewards: Agents in the same team, share rewards. Agents in different teams, compete for the same rewards. 30 Experimental Analysis for Several Types of Environments and Agents Environments

31. 31 Experimental Analysis for Several Types of Environments and Agents Prisoner’s Dilemma (3 players)

32. 32 Experimental Analysis for Several Types of Environments and Agents Lambda Environment

33. Two Random as teammates Random as the preyQ-learning agent as the prey 33 Experimental Analysis for Several Types of Environments and Agents Predator-Prey (predator slots)

34. Two Q-learning agents as teammates Random as the preyQ-learning agent as the prey 34 Experimental Analysis for Several Types of Environments and Agents Predator-Prey (predator slots)

35. 35 Experimental Analysis for Several Types of Environments and Agents Aggregation of Results

36. Agents’ expected results Prisoner’s Dilemma (3 players) Results: [0, 9] Lambda Environment Results: [-1, 1] Predator-Prey Results: [-1, 1] Random3,27759 QLSelfish3,50843 QLCommunal4,0716 QLMerciful4,06953 QLHarmful3,47243 Random0,03002 QLSelfish0,43221 QLCommunal0,42156 QLMerciful0,39438 QLHarmful0,41125 Random0,06413 QLSelfish0,08025 QLCommunal0,07856 QLMerciful0,07996 QLHarmful0,08293 36 Experimental Analysis for Several Types of Environments and Agents Grouped by Environments

37. Agents’ grading PDLE Random354,0 QLSelfish4,51,53,0 QLCommunal232,5 QLMerciful142,5 QLHarmful4,51,53,0 PDLE Random555,0 QLSelfish222,0 QLCommunal222,0 QLMerciful222,0 QLHarmful444,0 PDLEPP Random4,5554,8 QLSelfish312,52,2 QLCommunal222,52,2 QLMerciful13,52,52,3 QLHarmful4,53,52,53,5 37 Experimental Analysis for Several Types of Environments and Agents Grouped by Partitions

38. 38 Experimental Analysis for Several Types of Environments and Agents Conclusions Partition of agents into teams can have an impact on rewards. Cooperative team partitions can foster cooperation. Team partition is not powerful enough. Not always team partitions can foster cooperation/competition. Social intelligence is not only dependent on the environment used. Different populating agents can provide us with different results. We provided the result for each agent and, more importantly, also for a variety of environments. Our definition of social intelligence is not enough. Environments and populating agents must be suitable.

39. We have introduced a series of formal properties to analyse the suitability of a multi-agent environment to evaluate social intelligence: 39 Properties About Social Intelligence Testbeds Summary

40. We have applied the properties to several MAS: Five MAS environments/games have been analysed (any agent slot):  Matching pennies  Prisoner’s dilemma  Predator-prey  Pac-man  RoboCup Soccer Using all theoretically possible agents. 40 Characterising Several Multi-Agent and Social Scenarios Evaluate Environments

41. The ranges are wide if all possible agents are considered. 41 Characterising Several Multi-Agent and Social Scenarios Evaluate Environments

42. The ranges are (generally) poor if bad populating agents are selected. 42 Characterising Several Multi-Agent and Social Scenarios Evaluate Environments

43. We find several differences if good populating agents are selected. Ranges change radically when using families of agents instead of all. 43 Characterising Several Multi-Agent and Social Scenarios Evaluate Environments

44. For the instrumental properties: Validity problems originate because either: Cooperation is lacking. High levels cannot be measured. Other abilities are relevant. Reliability and efficiency problems, as environments are stochastic: Even with same line-up pattern and agent slot, results can be very different. With several repetitions, the average result slowly converges. 44 Characterising Several Multi-Agent and Social Scenarios Evaluate Environments

45. The lack of tests to evaluate social intelligence may be due to the lack of a proper definition. We provided a formal and parametrised definition of social intelligence. The skeleton for the definition is not enough. We provided a set of properties in order to better analyse the appropriateness of a testbed or multi- agent environment to evaluate social intelligence. 45 Conclusions and Future Work Conclusions

46. 46 Conclusions and Future Work Future Work Open features to be solved: How do we obtain an appropriate set of agents, set of environments (or environment class), weights, distributions, etc.? Which utility function should we use? Discount factor, average, last rewards… How to calculate the difficulty of exercises? Language and communication. Possible extensions: Evaluation of a group or collective of agents. Properties about the evaluation of cooperation through alliances or coalitions (individual teams).

47. Thank you! Towards a Universal Test of Social Intelligence Javier Insa Cabrera