Markov Logic and Deep Networks Pedro Domingos Dept. of Computer Science & Eng. University of Washington

Markov Logic Networks Basic idea: Use first-order logic to compactly specify large non-i.i.d. models MLN = Set of formulas with weights Formula = Feature template (Vars→Objects) E.g., HMM: Weight of formula iNo. of true instances of formula i in x State(+s,t) ^ State(+s',t+1) Obs(+o,t) ^ State(+s,t)

State of the Art in MLNs Many algorithms for learning and inference Inference: Millions of variables, billions of features Learning: Generative, discriminative, max-margin, etc. Best-performing solutions in many application areas Natural language, robot mapping, social networks, computational biology, activity recognition, etc. Open-source software/Web site: Alchemy Book: Domingos & Lowd, Markov Logic, Morgan & Claypool, alchemy.cs.washington.edu

Deep Uses of MLNs Very large scale inference Defining architecture of deep networks Adding knowledge to deep networks Transition to natural language input Learning with many levels of hidden variables

MLNs for Deep Learning Basic idea: Use small amounts of knowledge and large amounts of joint inference to make up for lack of supervision Relational clustering: Cluster objects with similar relations to similar objects Cluster relations that hold between similar sets of objects Coreference resolution: Outperforms supervised approaches on MUC and ACE benchmarks Semantic network extraction: Learns thousands of concepts and relations from millions of Web triples

Example: Unsupervised Semantic Parsing Goal: Read text and answer questions No supervision or annotation Input: Dependency parses of sentences (Nodes→Unary predicates / Edges→Binary preds.) Outputs: Semantic parser and knowledge base Basic idea: Cluster expressions with similar subexpressions Maps syntactic variants to common meaning Discovers its own meaning representation “Part of” lattice of clusters Applied to corpus of biomedical abstracts Three times more correct answers than next best