1. Language and Vision: Useful Tools Presenter: Vicente Ordonez

2. Text Analysis Tokenization, Tagging, Parsing, Word Embeddings

3. Python NLTK “My cat likes eating bananas” http://www.nltk.org/

4. Python NLTK: Tokenization http://www.nltk.org/ import nltk nltk.word_tokenize(“My cat likes eating bananas”) >>['My', 'cat', 'likes', 'eating', 'bananas']

5. Python NLTK: POS Tagging http://www.nltk.org/ import nltk words = nltk.word_tokenize(“My cat likes eating bananas”) nltk.pos_tag(words) >>[(‘My', ‘PRP$'), ('cat', 'NN'), ('likes', 'VBZ'), ('eating', 'VBG'), ('bananas', 'NNS')] Penn Treebank Postagging http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html

6. Python NLTK: Named Entities http://www.nltk.org/ import nltk words = nltk.word_tokenize(“My uncle Fred’s cat likes eating bananas”) tags = nltk.pos_tag(words) nlkt.ne_chunk(tag) >>Tree('S', [('My', 'PRP$'), ('uncle', 'NN'), Tree('PERSON', [(Fred', 'NNP')]), ("'s", 'POS'), ('cat', 'NN'), ('likes', 'VBZ'), ('eating', 'VBG'), ('bananas', 'NNS')])

7. Python NLTK: Wordnet http://www.nltk.org/ from nltk.corpus import wordnet wordnet.synsets('dog') // works even if you use ‘dogs’ instead >> [Synset('dog.n.01'), Synset('frump.n.01'), Synset('dog.n.03'), Synset('cad.n.01'), Synset('frank.n.02'), Synset('pawl.n.01'), Synset('andiron.n.01'), Synset('chase.v.01')] synset = wn.synset('dog.n.01') // You can get definition, lemmas, examples, hypernyms (“parent words”), hyponyms (“children words”), etc

8. Python NLTK: Wordnet Similarity http://www.nltk.org/ from nltk.corpus import wordnet dog = wn.synset('dog.n.01') cat = wn.synset('cat.n.01') similarity_score = dog.path_similarity(cat) similarity_score = dog.wup_similarity(cat)

9. Python Final Advice Python Spyder NLTK Scipy Numpy Matplotlib VTK etc

10. Stanford Core NLP: Parsing Properties props = new Properties(); props.setProperty("annotators", "tokenize, ssplit, pos, lemma, ner, parse"); StanfordCoreNLP pipeline = new StanfordCoreNLP(props); String text = “My cat likes eating bananas"; Annotation document = new Annotation(text); pipeline.annotate(document); List sentences = document.get(SentencesAnnotation.class); for(CoreMap sentence: sentences) { Tree tree = sentence.get(TreeAnnotation.class); // Do something here with the tree } http://nlp.stanford.edu:8080/parser/index.jsp http://nlp.stanford.edu/software/corenlp.shtml DEMO:

11. Stanford Core NLP: Dependencies Properties props = new Properties(); props.setProperty("annotators", "tokenize, ssplit, pos, lemma, ner, parse"); StanfordCoreNLP pipeline = new StanfordCoreNLP(props); String text = “My cat likes eating bananas"; Annotation document = new Annotation(text); pipeline.annotate(document); List sentences = document.get(SentencesAnnotation.class); for(CoreMap sentence: sentences) { SemanticGraph graph = sentence.get( CollapsedCCProcessedDependenciesAnnotation.class); // Do something here with the graph } http://nlp.stanford.edu:8080/parser/index.jsp http://nlp.stanford.edu/software/corenlp.shtml DEMO:

12. Stanford Core NLP: Sentiment Properties props = new Properties(); props.setProperty("annotators", "tokenize, ssplit, pos, lemma, ner, parse, sentiment"); StanfordCoreNLP pipeline = new StanfordCoreNLP(props); String text = “My cat likes eating bananas"; Annotation document = new Annotation(text); pipeline.annotate(document); List sentences = document.get(SentencesAnnotation.class); for(CoreMap sentence: sentences) { Tree tree = sentence.get(SentimentCoreAnnotations.AnnotatedTree.class); int sentiment = RNNCoreAnnotations.getPredictedClass(tree); // Do something here with the sentiment tree } http://nlp.stanford.edu:8080/sentiment/rntnDemo.html http://nlp.stanford.edu/software/corenlp.shtml DEMO:

13. Python Scikit-learn: Bag of Words from sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer(analyzer = "word", \ tokenizer = tokenize, \ stop_words = “english", \ max_features = 5000) // Compute vocabulary of top 5000 most frequent words corpus_features = vectorizer.fit_transform(text_corpus) // On test data features = vectorizer.transform(‘My cat likes eating bananas’)

14. Word2Vec My cat likes eating bananas [0.1 0.1 0.1 … 0.1 0.1 0.1] [0.3 0.1 0.4 … 0.1 0.2 0.1] [0.1 0.1 0.1 … 0.5 0.6 0.8] [0.1 0.2 0.4 … 0.6 0.7 0.8] [0.2 0.2 0.4 … 0.1 0.2 0.2] [0.1 0.1 0.1 … 0.1 0.1 0.1] Average https://code.google.com/p/word2vec/

15. Word2Vec https://code.google.com/p/word2vec/ Word Cosine distance ------------------------------------------- los_angeles 0.666175 golden_gate 0.571522 oakland 0.557521 california 0.554623 san_diego 0.534939 pasadena 0.519115 seattle 0.512098 taiko 0.507570 houston 0.499762 chicago_illinois 0.491598 Closest words to “san francisco” You can also try: http://nlp.stanford.edu/projects/glove/

16. Text Analysis Summary Basic Text Analysis using NTLK Splitting a sentence into words – tokenization. Extracting nouns, verbs, adjectives, etc – POS-tagging Computing word similarities – Wordnet Sentence Parsing using StanfordNLP Breaking sentences into its subject, predicate, etc. Resolving word dependencies. Sentiment Analysis Word representations Bag of Words - Scikit-learn Neural Networks – Word2vec

17. Image Analysis Image representations, shape, color + shape, recognition

18. VLFeat: HOG Features - Matlab http://www.vlfeat.org/ image = imread(‘house.jpg’) hog = vl_hog(image, 8) http://www.vlfeat.org/overview/hog.html

19. VLFeat: Dense SIFT - Matlab binSize = 8; magnif = 3; image_smooth = vl_imsmooth(I, sqrt((binSize/magnif)^2 -.25)) ; [frames, descriptors] = vl_dsift(image_smooth, 'size', binSize) ; http://www.vlfeat.org/ http://www.vlfeat.org/overview/dsift.html http://www.vlfeat.org/applications/caltech-101-code.html

20. Convolutional Networks Krizhevsky, Sutskever, Hinton (2012)

21. Caffe import caffe caffe_root = 'CAFFE_INSTALLATION_DIRECTORY' caffe.set_mode_cpu() net = caffe.Classifier( caffe_root + 'models/bvlc_reference_caffenet/deploy.prototxt', caffe_root + 'models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel') net.transformer.set_mean('data', np.load(caffe_root + 'python/caffe/imagenet/ilsvrc_2012_mean.npy').mean(1).mean(1)) net.transformer.set_raw_scale('data', 255) net.transformer.set_channel_swap('data', (2,1,0)) scores = net.predict([caffe.io.load_image(caffe_root + 'examples/images/cat.jpg')]) http://caffe.berkeleyvision.org/

22. Caffe >> scores [('data', (10, 3, 227, 227)), ('conv1', (10, 96, 55, 55)), ('pool1', (10, 96, 27, 27)), ('norm1', (10, 96, 27, 27)), ('conv2', (10, 256, 27, 27)), ('pool2', (10, 256, 13, 13)), ('norm2', (10, 256, 13, 13)), ('conv3', (10, 384, 13, 13)), ('conv4', (10, 384, 13, 13)), ('conv5', (10, 256, 13, 13)), ('pool5', (10, 256, 6, 6)), ('fc6', (10, 4096, 1, 1)), ('fc7', (10, 4096, 1, 1)), ('fc8', (10, 1000, 1, 1)), ('prob', (10, 1000, 1, 1))] http://caffe.berkeleyvision.org/

23. RCNN – Detection – Matlab / C https://github.com/rbgirshick/rcnn http://nbviewer.ipython.org/github/BVLC/caffe/blob/master/examples/detection.ipynb

24. DPM – Detection – Matlab - C http://www.cs.berkeley.edu/~rbg/latent/

25. Image Analysis Summary Overview of the VLFeat Library Basic image operators Distance functions, clustering, etc. Computing Features using Caffe / MatConvNet Obtaining category predictions Obtaining intermediate image representations Object Detection RCNN code – C/C++ - Matlab DPM code – C/C++ - Matlab