1. Fabio Massimo Zanzotto and Danilo Croce University of Rome “Tor Vergata” Roma, Italy Reading what Machines ‘Think’

2. F.M.Zanzotto University of Rome “Tor Vergata” Prelude Brain Activation Pattern Recognizer chair Tom Mitchell, Invited Talk at the Association for Computational Linguistics Conference 2007 Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A.: Predicting human brain activity associated with the meanings of nouns. Science 320(5880) (May 2008) 1191–1195 Question This is a fascinating research problem. Can we find a more controlled setting where we can test if this is possible? Question This is a fascinating research problem. Can we find a more controlled setting where we can test if this is possible?

3. F.M.Zanzotto University of Rome “Tor Vergata” Idea Cognitive physical object Cognitive task Observed image Observing a chair Sorting a vector Brain Computational Machine

4. F.M.Zanzotto University of Rome “Tor Vergata” Why investigating the computer side is relevant? Foundational perspective –Computers are becoming extremely complex. They are fastly approaching the complexity of human brain –Computers are controlled machines: their behavior and thier internal organization is known –Then, computers offer a way to estimate if the claim on the brain side is reachable: if we can read what machines think, we can hope to read what brains think. Motivation

5. F.M.Zanzotto University of Rome “Tor Vergata” Why investigating the computer side is relevant? Applicative perspective Can we develop technologies that “read the computer mind”? This predictive model can have a wide variety of applications, e.g., detecting malicious software, detecting the intentions of hostile computers by looking at their activation patterns. Motivation

6. F.M.Zanzotto University of Rome “Tor Vergata” Investigating the computer side: Long term research program Sketching the overall observation activity Virtual Observation of Processes Experimental Investigation In the rest of the talk

7. F.M.Zanzotto University of Rome “Tor Vergata” Long-term research program 1010101001011101111000010010 1010100101110111010110100101 0101100000111010101010101111 1010101011101011011111100011 0101010010111011110000100101 0101001011101110101101001010 1011000001110101010101011111 0101010111010110111111000010 1010111010110111111000010101 0111010110111111000010101011 10101101111110001… 011101 111000 010101 010101 01110111 10000101 01010101 11110101 01011101 01101111 110001... Physical Memory Chip Physical Memory Dump Virtual Memory Dump (organized in processes) Process Memory Dump Physical device activation image capturer Virtual activation image capturer

8. F.M.Zanzotto University of Rome “Tor Vergata” Sorting a vector Sketching the overall observation activity Brain Activation Pattern Recognizer chair Sorting a vector Process Activation Pattern Recognizer Building images from processes Defining feature spaces for images Observed Phenomena: Processes

9. F.M.Zanzotto University of Rome “Tor Vergata” Observed Phenomena: Processes

10. F.M.Zanzotto University of Rome “Tor Vergata” Given a cognitive activity, the procedure for extracting images from this activity is then the following: –running the process p representing the cognitive activity c –stopping the process at given states or at given time intervals –dumping the memory associated with the process M(p) –given a fixed height image and the memory dump, read incrementally bytes of the memory dump and fill the associated RGB pixel with the read values I(M(p)) Building activation images from processes

11. F.M.Zanzotto University of Rome “Tor Vergata” Process memory in a given time interval Activation image of the process in a given time interval where is the RGB pixel definition of the image Building activation images from processes

12. F.M.Zanzotto University of Rome “Tor Vergata” Process: Vector Sorter Initial State Process: Vector Sorter Final State Building activation images from processes Smoothing (more similar to real chip observation)

13. F.M.Zanzotto University of Rome “Tor Vergata” We used three major classes of features Chromatic feaures –Capture the color properties of the image determining, an n-dimensional vector representation of the 2D chromaticity histograms texture (OP - OGD) features –emphasize the background properties and their composition. transformation features (OGD) Defining feature spaces for images

14. F.M.Zanzotto University of Rome “Tor Vergata” Experimental Set-up –Collection of activation images –Used Machine Learning algorithms Experimental Results Experimental Evaluation

15. F.M.Zanzotto University of Rome “Tor Vergata” 3 different “cognitive tasks” (algorithms) –sorting, comparing two strings, visiting a binary tree 3 different programming languages –c, java, php for each pair algorithm-programming language –20 different randomly generated input data –3 snapshots (beginning, middle, end) Experimental Set-up

16. F.M.Zanzotto University of Rome “Tor Vergata” We randomly splited the final set (540 images) in: Training: 270 images Testing: 270 images Two classification tasks: Determining the programming language (3 classes) (lang) Determining the cognitive task (3 classes) (algo) Used Machine learning Models: Decision Tree Learners (DecTree) Naive Bayes Experimental Set-up

17. F.M.Zanzotto University of Rome “Tor Vergata” Results Classification accuracy

18. F.M.Zanzotto University of Rome “Tor Vergata” The parallelism between computer and brain/mind is not new in general –Cognitive psychology –Cognitive sciences We looked this parallelism from an other perspective Conclusion

19. F.M.Zanzotto University of Rome “Tor Vergata” Future Work 1010101001011101111000010010 1010100101110111010110100101 0101100000111010101010101111 1010101011101011011111100011 0101010010111011110000100101 0101001011101110101101001010 1011000001110101010101011111 0101010111010110111111000010 1010111010110111111000010101 0111010110111111000010101011 10101101111110001… 011101 111000 010101 010101 01110111 10000101 01010101 11110101 01011101 01101111 110001... Physical Memory Chip Physical Memory Dump Virtual Memory Dump (organized in processes) Process Memory Dump Physical device activation image capturer Virtual activation image capturer

20. F.M.Zanzotto University of Rome “Tor Vergata” Thank you for the attention!