CSC 594 Topics in AI – Natural Language Processing Spring 2018 3. Regular Expressions (Some slides adapted from Jurafsky & Martin)

Document Search ‘Information Retrieval (IR)’ implies a query (e.g. search terms) For a given query, relevant or similar documents are returned. But most basic document retrieval technique is keyword/search term matching. Retrieve all (or selected) documents which contain the search terms -- by string matching Python example: >>> s1 = 'public' >>> s2 = 'public' >>> s2 == s1 True myword = “month python” with open("textfile.txt") as openfile: for line in openfile: if myword in line: print line

Regular Expressions and Text Searching Regular expressions are a compact textual representation of a set of strings that constitute a language In the simplest case, regular expressions describe regular languages Here, a language means a set of strings given some alphabet. Extremely versatile and widely used technology Emacs, vi, perl, grep, etc.

Example Find all the instances of the word “the” in a text. /the/ /\b[tT]he\b/

String Matching Using Patterns Often, we wish to find a substring which matches a pattern e.g. E-mail addresses: Any number of alphanumeric characters and/or dots (not a dot at beginning or end) @ Any number of alphanumeric characters and/or dots (not a dot at beginning or end); must be at least one dot Examples: valid: tomuro@cs.depaul.edu, noriko.tomuro@gmail.com Invalid: .tomuro@cs.depaul.edu, tomuro@depaul But if you want to specify search words by patterns, regular expressions are commonly used.

Regular Expressions (1) Regular expression is an algebra for defining patterns. For example, a regular expression “a*b” matches with a string “aaaab”. But without going through the formal definitions, here is a (partial) summary. Simple Patterns Characters match themselves. Note the chars are case-sensitive. Metacharacters – not to be used literally _as is_ . ^ $ * + ? { } [ ] \ | ( ) To use a metacharacter, a back-slash has to be given before it \. \^ \+ etc. Other special characters \t, \n, \r, \f etc.

Regular Expressions (2) Character classes [abc] – a, b, or c [^abc] – any character except a, b, or c. [a-zA-Z] – a throughx, or A through Z inclusive (range) Predefined character classes . (dot) – any character \d – a digit ([0-9]) \D – a non-digit ([^0-9]) \s – a whitespace character (e.g. space, \t, \n, \r) \S – a non-whitespace character \w – a word character ([a-zA-Z_0-9]) \W – a non-word character ([^\w]) Boundary matchers ^ -- the beginning of a line $ -- the end of a line

Regular Expressions (3) Greedy quantifiers X? – X, once or not at all Z* -- X, zero or more times X+ -- X, one or more times X{n} – X, exactly n times X{n,m} – X, at least n but no more than m times Logical operators XY – X followed by Y X|Y – either X or Y (X) – X, as a capturing group

Regular Expression in Python (1) Regular expressions are in the ‘re’ package. Notation for patterns is slightly different from other languages – using raw string as an alternative to Regular string. First compile an expression (into an re object). Then match it against a string. >>> import re >>> p = re.compile('ab*') Regular String Raw string "ab*" r"ab*" "\\\\section" r"\\section" "\\w+\\s+\\1" r"\w+\s+\1"

Regular Expression in Python (2) Matching a re object against a string is done in several ways. Method/Attribute Purpose match() Determine if the RE matches at the beginning of the string. search() Scan through a string, looking for any location where this RE matches. findall() Find all substrings where the RE matches, and returns them as a list. finditer() Find all substrings where the RE matches, and returns them as aniterator.

>>> import re >>> sent = "This book on tennis cost $3.99 at Walmart." >>> p1 = re.compile("ten") >>> m1 = p1.match(sent) >>> m1 >>> p2 = re.compile(".*ten.*") >>> m2 = p2.match(sent) >>> m2 <_sre.SRE_Match object; span=(0, 42), match='This book on tennis cost $3.99 at Walmart.'> >>> m3 = re.search(p1,sent) >>> m3 <_sre.SRE_Match object; span=(13, 16), match='ten'> >>> m4 = re.search(p2,sent) >>> m4 >>> pp1 = re.compile("is") >>> m5 = re.findall(pp1, sent) >>> m5 ['is', 'is'] >>> pp2 = re.compile("\\d") >>> m6 = re.search(pp2, sent) >>> m6 <_sre.SRE_Match object; span=(26, 27), match='3'> >>> pp3 = re.compile("\\d+") >>> m7 = re.search(pp3, sent) >>> m7

>>> pp3 = re.compile("\\$\\d+\\.\\d\\d") >>> m8 = re.search(pp3, sent) >>> m8 <_sre.SRE_Match object; span=(25, 30), match='$3.99'> >>> pp4 = re.compile(r"\$\d+\.\d\d") >>> m9 = re.search(pp4, sent) >>> m9

Regular Expression in Python (3) Grouping – You can retrieve the matched substrings using parentheses. Capturing groups are numbered by counting their opening parentheses from left to right. In the expression ((A)(B(C))), for example, there are four such groups: ((A)(B(C))) (A) (B(C)) (C) Group zero always stands for the entire expression.

>>> ppp1 = re.compile("(\\w+) cost (\\$\\d+\\.\\d\\d)") >>> mm1 = re.search(ppp1, sent) >>> mm1 <_sre.SRE_Match object; span=(13, 30), match='tennis cost $3.99'> >>> mm1.group(0) 'tennis cost $3.99' >>> mm1.group(1) 'tennis' >>> mm1.group(2) '$3.99'

TutorialsPoint, http://www. tutorialspoint

Regular Expression Modifiers: Option Flags Regular expression literals may include an optional modifier to control various aspects of matching. The modifiers are specified as an optional flag. You can provide multiple modifiers using exclusive OR (|), as shown previously and may be represented by one of these − Modifier Description re.I Performs case-insensitive matching. re.L Interprets words according to the current locale. This interpretation affects the alphabetic group (\w and \W), as well as word boundary behavior (\b and \B). re.M Makes $ match the end of a line (not just the end of the string) and makes ^ match the start of any line (not just the start of the string). re.S Makes a period (dot) match any character, including a newline. re.U Interprets letters according to the Unicode character set. This flag affects the behavior of \w, \W, \b, \B. re.X Permits "cuter" regular expression syntax. It ignores whitespace (except inside a set [] or when escaped by a backslash) and treats unescaped # as a comment marker. TutorialsPoint, http://www.tutorialspoint.com/python/python_reg_expressions.htm