1 Business Proprietary © 2009 Oculus Info Inc. Everyone’s a Critic: Memory Models and Uses for an Artificial Turing Judge W. Joseph MacInnes, Blair C. Armstrong, Dwayne Pare, George S. Cree and Steve Joordens

2 Business Proprietary © 2009 Oculus Info Inc. Turing Test 3 parts to play –Computer Program (AI) –Human Confederate –Judge Is it a test for intelligence? –Likely necessary but not sufficient Unclaimed Loebner prize is evidence that this is still a hard problem for a conversational agent. C?H?

3 Business Proprietary © 2009 Oculus Info Inc. Here Comes the Judge Focus on third party as first step –Can a computer agent function as a Turing Judge? –Language recognition not generation –Can an automated judge tell the difference between human and computer generated conversations? –Human judges may use Grammar Meaningfulness of reply Relatedness of answer ‘Commonsense’ knowledge Task: determine which is human/computer

4 Business Proprietary © 2009 Oculus Info Inc. Rationale and Applications 1.Use of judge as internal critic for language creation/generation for AGI 2.Models of human language comprehension and generation likely include critic Garden path sentences suggest internal critic evaluates a sentence continuously while reading. the cotton clothing is made of… is grown in the South First reading interprets ‘cotton’ as adjective, but later adjust to noun 3.Applications in spam detection and detecting computer generated forum posts for narrow AI.

5 Business Proprietary © 2009 Oculus Info Inc. LSA A1A2A3A4B1B2B3B4 built111 cylons11111 humans111 a111 galactica111 survivors1111 viper111 A1. Humans built the Cylons to make their lives easier. A2. The Cylons did not like doing work for the humans. A3. In a surprise attack, the Cylons destroyed the humans that built them. A4. The Cylons were built by humans to do arduous work. B1. Some survivors escaped and fled on the Galactica. B2. The Galactica protected the survivors using its Viper attack ships. B3. The Cylons were no match for a Viper flown by one of the survivors. B4. A Viper flown by one of the survivors found Earth and led the Galactica there. 1.LSA is a corpus-based statistical method for generating representations that capture aspects of word meaning based on the contexts in which words co-occur. 2.The text corpus is converted into a word x passage matrix, where the passages can be any unit of text (e.g., sentence, paragraph, essay). 3.The elements of the matrix are the frequencies of each target word in each passage The entire matrix is then submitted to singular value decomposition (SVD), the purpose of which is to abstract a lower dimensional (e.g., 300 dimensions) meaning space in which each word is represented as a vector in this compressed space. 4.In addition to computational efficiency, this smaller matrix tends to better emphasize the similarities amongst words. 5.Following the generation of this compressed matrix, new passages can be compared by comparing the cosine of the word vectors the passage contains.

6 Business Proprietary © 2009 Oculus Info Inc. LSA: Example A1A2A3A4B1B2B3B4 built111 cylons11111 humans111 a111 galactica111 survivors1111 viper111 A1. Humans built the Cylons to make their lives easier. A2. The Cylons did not like doing work for the humans. A3. In a surprise attack, the Cylons destroyed the humans that built them. A4. The Cylons were built by humans to do arduous work. B1. Some survivors escaped and fled on the Galactica. B2. The Galactica protected the survivors using its Viper attack ships. B3. The Cylons were no match for a Viper flown by one of the survivors. B4. A Viper flown by one of the survivors found Earth and led the Galactica there. Given the question “The humans built what?” And the possible replies The humans built the Cylons” and “They built the Galactica and the vipers The question overlaps more in space with the first question and would have a higher rating by our judge The effect of SVD is to reduce the dimensionality of the matrix, and discover word similarities which weren’t explicit in the original text (hence the ‘Latent’ part)

7 Business Proprietary © 2009 Oculus Info Inc. Results Artificial Judge Human Judge 1 Human Judge 2 Human Judge 3 Human Judge 4 Hum (.20)Hum (77)Hum (75)Hum (79)Hum (88) Ali (.17)Jab (40)Eug (40)Eug (45)Eug (30) Eug (.12)Eug (35)Ton (30)Jab (20)Ali (10) Jab (.10)Ton (10)Ali (20)Ton (15)Jab (8) Ton (.07)Ali (9)Eug (10)Ali (5)Ton (2) “Humanness” was operationalized as the vector cosine similarity for all of the words in the question relative to all of the words in the answer. Error bars are the standard error of the mean. With the exception of ALICE, humans scored significantly higher than the artificial agents. Rank orderings and performance scores for all judges. Substantial variability in the rank orderings of the different agents by the different human judges There was a higher correlation of agent rankings within human judges (0.83) than with our artificial judge, though the artificial judge did reach a respectable (0.63).

8 Business Proprietary © 2009 Oculus Info Inc. Future work Fom Forbidden Planet content/uploads/2007/12/Adam%20 LevermoreRich%20how%20to%20s pot%20a%20cylon.jpg  Better training data  Live journal ‘conversations’  Better Judge  Topic based, LDA, to try extract the topic or ‘gist’ of a sentence  Pure semantic spaces (LSA) are unlikely sufficient  Better ambitions  Full conversational agent

9 Business Proprietary © 2009 Oculus Info Inc. Potential Fit Response Generation Comprehension Semantics Grammar Context Critic ??