1. Intelligent Systems Colloquium 3 Turing test and problem of testing of Intelligence

2. Acting humanly: Turing Test Turing (1950) "Computing machinery and intelligence": "Can machines think?"  "Can machines behave intelligently?" Operational test for intelligent behavior: the Imitation Game

3. Two versions of Turing test Susan Sterrett’s careful reading of Turing’s 1950 paper reveals a significant distinction between two different versions of what has come to be known as the Turing Test (Sterrett 2000). The first version, dubbed the Original Imitation Game (OIG), appears on the very first page of Computing Machinery and Intelligence (Turing 1950). The OIG has three players: a man (A), a woman (B), and a third person (C) of either sex. The third player (C) is called the interrogator, and his function is to communicate with the other two, through what would nowadays be called a text-only instant messaging chat interface, using two terminals (or today perhaps, two windows) labeled (X) and (Y). The interrogator must decide whether (X) is (A) and (Y) is (B), or (X) is (B) and (Y) is (A), in other words which is the man and which is the woman. The interrogator’s task is complicated by the man (A), who Turing says should reply to the interrogator with lies and deceptions. For example, if the man is asked, “are you a man or a woman?,” he might reply, “I am a woman.” Putting aside the gender and social issues raised by the OIG, consider the OIG as an actual scientific experiment. Turing’s point is that if we were to actually conduct the OIG with a sufficiently large sample of subjects playing the parts of (A), (B), and (C), then we could measure a specific percentage M of the time that, on average, the interrogator misidentifies the woman, so that 100-M% of the time she is identified correctly. Given enough trials of the OIG, at least in a given historical and cultural context, the number M ought to be a fairly repeatable measurement.

4. Two versions of Turing test (2) Turing describes a second game more like the concept of a “Turing Test” as most engineering schools teach it. The setup is similar to the OIG, but now gender plays no role. The player (B) is called “a man” and the player (A) is always a computer. The interrogator must still decide whether (X) is (A) and (Y) is (B), or (X) is (B) and (Y) is (A), in other words which is the man and which is the machine. Sterrett calls this second game the Standard Turing Test (STT). Unlike the OIG, the STT is not a good scientific experiment. What does it mean to “pass” the STT? Must the interrogator identify the machine correctly 50% of the time, or 100%? For how long must the machine deceive the interrogator? Finally, does the interrogator know in advance that he is trying to “out”(Zdenek 2000) the robot, or that one of the players is a machine at all? Unfortunately the STT, though flawed as an experiment, has come to be popularized as the modern “Turing Test.” The STT is the basis of real-world Turing Tests including the Loebner Prize

5. Loebner Prize The Loebner Prize Medal and cash award is presented annually to the designer of the computer system that best succeeds in passing a variant of the Turing Test. The Loebner Prize Competition in Artificial Intelligence was established in 1990 by Hugh Loebner and was first held at the Boston Computer Museum, Boston, Massachusetts, USA in 1991. From 2001, the Competition is to be run by the Science Museum, London. In accordance with the requirements of the Donor, as published in the June 1994 Communications of the ACM, the winner of the US$100,000 Gold Prize must be prepared to deal with audio visual input, and appropriate competitions will be held once Competitors have reached Turing's 50:50 likelihood level of being mistaken for a human. An intermediate Silver Prize of US$25,000 will be offered for reaching this level in a text-only test. There is also an Annual Bronze Prize which is awarded to the designer of the "most human computer" as rated by a panel of Judges. In 2001, this prize will be worth $2,000 Applications must be accompanied by e-mail protocols recording interactions between the computer system to be entered and one or more human subjects. Protocols must not exceed 2500 words. Applications must be submitted by e-mail to the Science Museum. The Science Museum may request an opportunity to interact with Computer Entries.

6. Searle’s Chinese room Searle’s Chinese room provides a good metaphor for thinking about A.L.I.C.E.-like systems. Indeed the AIML contents of the A.L.I.C.E. brain is a kind of “Chinese Room Operator’s Manual.” English language Chinese language Operator’s Manual (dictionary) She is understanding Chinese language?

7. Learning and Turing test Some have argued that Turing, when he predicted that a machine could play his game in “50 years” after his 1950 paper, envisioned something more like a general purpose learning machine, which does not yet exist. The concept is simple enough: build a robot to grow like a child, able to be taught language the way we are. In our terms, the role of the botmaster would be fully automated. But even a child does not, or at least should not, go forth into the world, unprotected, to learn language “on the street,” without supervision.

8. Problems of testing of AI Definition of Intelligence is absent. What is Intelligence? It is possible three approaches to define it: –Intelligence – knowledge (data or facts about world) –Intelligence – capability to solve tasks (logical, mathematical, searching, games and so on) –Intelligence – capability to adapt and learn Human Intelligence is different too –Different intelligence of different people in above 3 sense Different intelligence (in above 3 sense) of tested human and testing human. So lack of understanding is possible and relatively usual. –Different knowledge (facts in memory), in particular, knowledge about language –Different knowledge about solving of tasks –Different capabilities to adapt

9. Kinds of testing of AI So testing of AI is task of estimation of similarity between behavior of Human and Computer (language-based test is more simple and standartiziable) May be other tests, for example, learning to play in game with previously unknown rules (http://www.a-i.com, 2001)http://www.a-i.com For AI in mobile robots and animats may be other tests with any movements, for example: –Navigation in labyrinth –Avoiding of any moving object (Gavrilov A.V., 1990) –Pursuit of any object –Searching of any objects in unknown or changing environment –Navigation in human-like complex and partly unknown environment (DARPA, 2003) –Different games, for example, soccer (from 1995, http://www.robocup.org) http://www.robocup.org

10. Learning to play in game with previously unknown rules Challenge: Create a program that can win at a game without knowing the game's rules Program will compete against the programs of other contestants in simple turn-based games. The rules of the games will not be explained beforehand. Your program will be notified of its opponent's previous move, and after your program makes a move, it will be notified of a win, loss, or draw. Games will consist of many thousands of moves Turns of games are words from determined set of words Program must to learn to get win as result of turn

11. Avoiding of any moving object In closed space any object is moving by any rules (law) Task for robot is to avoid collisions or nearness with this object Control system must to learn any behavior corresponding to law of movement of object Criterion of quality of control system may be frequency of collisions or average distance between object and system