Presentation is loading. Please wait.

Presentation is loading. Please wait.

Introduction Copyright © Software Carpentry 2010 This work is licensed under the Creative Commons Attribution License See

Published byAnissa Morton Modified over 9 years ago

Similar presentations

Presentation on theme: "Introduction Copyright © Software Carpentry 2010 This work is licensed under the Creative Commons Attribution License See"— Presentation transcript:

1 Introduction Copyright © Software Carpentry 2010 This work is licensed under the Creative Commons Attribution License See http://software-carpentry.org/license.html for more information. Regular Expressions

2 Introduction

3 Regular ExpressionsIntroduction

4 Regular ExpressionsIntroduction

5 Regular ExpressionsIntroduction

6 Regular ExpressionsIntroduction

7 Regular ExpressionsIntroduction

8 Regular ExpressionsIntroduction

9 Regular ExpressionsIntroduction Your mission: read 20-30 files containing several hundred measurements each of background evil levels (in millivaders) and convert them to a uniform format for further processing

10 Regular ExpressionsIntroduction Your mission: read 20-30 files containing several hundred measurements each of residual evil levels and convert them to a uniform format for further processing Each reading has a site name, the date, and the level of background evil (in millivaders)

11 Regular ExpressionsIntroduction Your mission: read 20-30 files containing several hundred measurements each of residual evil levels and convert them to a uniform format for further processing Each reading has a site name, the date, and the level of background evil (in millivaders) Some use tabs to separate fields, others use commas

12 Regular ExpressionsIntroduction Your mission: read 20-30 files containing several hundred measurements each of residual evil levels and convert them to a uniform format for further processing Each reading has a site name, the date, and the level of background evil (in millivaders) Some use tabs to separate fields, others use commas Dates are written in several different styles

13 Regular ExpressionsIntroduction Notebook #1 Site Date Evil (millivaders) ---- ---- ------------------ Baker 1 2009-11-17 1223.0 Baker 1 2010-06-24 1122.7 Baker 2 2009-07-24 2819.0 Baker 2 2010-08-25 2971.6 Baker 1 2011-01-05 1410.0 Baker 2 2010-09-04 4671.6 ⋮ ⋮ ⋮

14 Regular ExpressionsIntroduction Notebook #1 single tab as separator Site Date Evil (millivaders) ---- ---- ------------------ Baker 1 2009-11-17 1223.0 Baker 1 2010-06-24 1122.7 Baker 2 2009-07-24 2819.0 Baker 2 2010-08-25 2971.6 Baker 1 2011-01-05 1410.0 Baker 2 2010-09-04 4671.6 ⋮ ⋮ ⋮

15 Regular ExpressionsIntroduction Notebook #1 spaces in site names Site Date Evil (millivaders) ---- ---- ------------------ Baker 1 2009-11-17 1223.0 Baker 1 2010-06-24 1122.7 Baker 2 2009-07-24 2819.0 Baker 2 2010-08-25 2971.6 Baker 1 2011-01-05 1410.0 Baker 2 2010-09-04 4671.6 ⋮ ⋮ ⋮

16 Regular ExpressionsIntroduction Notebook #1 dates in international standard format (YYYY-MM-DD) Site Date Evil (millivaders) ---- ---- ------------------ Baker 1 2009-11-17 1223.0 Baker 1 2010-06-24 1122.7 Baker 2 2009-07-24 2819.0 Baker 2 2010-08-25 2971.6 Baker 1 2011-01-05 1410.0 Baker 2 2010-09-04 4671.6 ⋮ ⋮ ⋮

17 Regular ExpressionsIntroduction Notebook #2 Site/Date/Evil Davison/May 22, 2010/1721.3 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0 ⋮ ⋮ ⋮

18 Regular ExpressionsIntroduction Notebook #2 slashes as separators Site/Date/Evil Davison/May 22, 2010/1721.3 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0 ⋮ ⋮ ⋮

19 Regular ExpressionsIntroduction Notebook #2 Site/Date/Evil Davison/May 22, 2010/1721.3 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0 ⋮ ⋮ ⋮ site names don't appear to have spaces

20 Regular ExpressionsIntroduction Notebook #2 Site/Date/Evil Davison/May 22, 2010/1721.3 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0 ⋮ ⋮ ⋮ month names and day numbers of varying length

21 Regular ExpressionsIntroduction Solution: regular expressions

22 Regular ExpressionsIntroduction Solution: regular expressions A pattern that strings can match

23 Regular ExpressionsIntroduction Solution: regular expressions A pattern that strings can match Like '*.txt' matches filenames ending in '.txt'

24 Regular ExpressionsIntroduction Solution: regular expressions A pattern that strings can match Like '*.txt' matches filenames ending in '.txt' Warning: notation is ugly

25 Regular ExpressionsIntroduction Solution: regular expressions A pattern that strings can match Like '*.txt' matches filenames ending in '.txt' Warning: notation is ugly Writing patterns for strings as strings...

26 Regular ExpressionsIntroduction Solution: regular expressions A pattern that strings can match Like '*.txt' matches filenames ending in '.txt' Warning: notation is ugly Writing patterns for strings as strings......using only the symbols on the keyboard (instead of inventing new symbols like mathematicians do)

27 Regular ExpressionsIntroduction # read the first half-dozen data records from two files readings = [] for filename in ('data-1.txt', 'data-2.txt'): lines = open(filename, 'r').read().strip().split('\n') readings += lines[2:8] for r in readings: print r

28 Regular ExpressionsIntroduction Baker 12009-11-171223.0 Baker 12010-06-241122.7 Baker 22009-07-242819.0 Baker 22010-08-252971.6 Baker 12011-01-051410.0 Baker 22010-09-044671.6 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0

29 Regular ExpressionsIntroduction # select readings in month '06' for r in readings: if '06' in r: print r Baker 12010-06-241122.7

30 Regular ExpressionsIntroduction # select readings in month '06' or month '07' for r in readings: if ('06' in r) or ('07' in r): print r Baker 12010-06-241122.7 Baker 22009-07-242819.0

31 Regular ExpressionsIntroduction # what about readings in month '05'? (shouldn't be any) for r in readings: if ('05' in r): print r Baker 12011-01-051410.0

32 Regular ExpressionsIntroduction # what about readings in month '05'? (shouldn't be any) for r in readings: if ('05' in r): print r Baker 12011-01-051410.0 "in string" is a dangerously blunt tool

33 Regular ExpressionsIntroduction # try using regular expressions instead import re for r in readings: if re.search('06', r): print r Baker 12010-06-241122.7

34 Regular ExpressionsIntroduction # try using regular expressions instead import re for r in readings: if re.search('06', r): print r Baker 12010-06-241122.7 not much of an improvement so far...

35 Regular ExpressionsIntroduction # find records with '06' or '07' in one search import re for r in readings: if re.search('06|07', r): print r Baker 12010-06-241122.7 Baker 22009-07-242819.0

36 Regular ExpressionsIntroduction # find records with '06' or '07' in one search import re for r in readings: if re.search('06|07', r): print r Baker 12010-06-241122.7 Baker 22009-07-242819.0 first argument is the pattern to search for

37 Regular ExpressionsIntroduction # find records with '06' or '07' in one search import re for r in readings: if re.search('06|07', r): print r Baker 12010-06-241122.7 Baker 22009-07-242819.0 first argument is the pattern to search for written as a string

38 Regular ExpressionsIntroduction # find records with '06' or '07' in one search import re for r in readings: if re.search('06|07', r): print r Baker 12010-06-241122.7 Baker 22009-07-242819.0 second argument is the data to search in

39 Regular ExpressionsIntroduction # find records with '06' or '07' in one search import re for r in readings: if re.search('06|07', r): print r Baker 12010-06-241122.7 Baker 22009-07-242819.0 second argument is the data to search in reversing these is a common mistake (and hard to track down)

40 Regular ExpressionsIntroduction # find records with '06' or '07' in one search import re for r in readings: if re.search('06|07', r): print r Baker 12010-06-241122.7 Baker 22009-07-242819.0 vertical bar '|' means OR

41 Regular ExpressionsIntroduction # find records with '06' or '07' in one search import re for r in readings: if re.search('06|07', r): print r Baker 12010-06-241122.7 Baker 22009-07-242819.0 vertical bar '|' means OR match either what's on the left, or what's on the right, in a single search

42 Regular ExpressionsIntroduction # we're going to be trying out a lot of patterns, # so let's write a function def show_matches(pattern, strings): for s in strings: if re.search(pattern, s): print '**', s else: print ' ', s

43 Regular ExpressionsIntroduction # test our function right away show_matches('06|07', readings) Baker 1 2009-11-17 1223.0 ** Baker 1 2010-06-24 1122.7 ** Baker 2 2009-07-24 2819.0 Baker 2 2010-08-25 2971.6 Baker 1 2011-01-05 1410.0 Baker 2 2010-09-04 4671.6 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0

44 Regular ExpressionsIntroduction # why doesn't this work? show_matches('06|7', readings) ** Baker 1 2009-11-17 1223.0 ** Baker 1 2010-06-24 1122.7 ** Baker 2 2009-07-24 2819.0 ** Baker 2 2010-08-25 2971.6 Baker 1 2011-01-05 1410.0 ** Baker 2 2010-09-04 4671.6 ** Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 ** Davison/June 19, 2010/1731.9 ** Davison/July 6, 2010/2010.7 ** Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0

45 Regular ExpressionsIntroduction In mathematics, "ab+c" means (a×b) + c

46 Regular ExpressionsIntroduction In mathematics, "ab+c" means (a×b) + c Multiplication is implied, and has higher precedence

47 Regular ExpressionsIntroduction In mathematics, "ab+c" means (a×b) + c Multiplication is implied, and has higher precedence To force the other meaning, write "a(b+c)"

48 Regular ExpressionsIntroduction In mathematics, "ab+c" means (a×b) + c Multiplication is implied, and has higher precedence To force the other meaning, write "a(b+c)" In regular expressions, "06|7" means '06' or '7'

49 Regular ExpressionsIntroduction In mathematics, "ab+c" means (a×b) + c Multiplication is implied, and has higher precedence To force the other meaning, write "a(b+c)" In regular expressions, "06|7" means '06' or '7'...and there are a lot of 7's in our file

50 Regular ExpressionsIntroduction In mathematics, "ab+c" means (a×b) + c Multiplication is implied, and has higher precedence To force the other meaning, write "a(b+c)" In regular expressions, "06|7" means '06' or '7'...and there are a lot of 7's in our file To force the other meaning, parenthesize "0(6|7)"

51 Regular ExpressionsIntroduction In mathematics, "ab+c" means (a×b) + c Multiplication is implied, and has higher precedence To force the other meaning, write "a(b+c)" In regular expressions, "06|7" means '06' or '7'...and there are a lot of 7's in our file To force the other meaning, parenthesize "0(6|7)" But "06|07" is more readable anyway

52 Regular ExpressionsIntroduction # still matching days when we want to match months show_matches('05', readings) Baker 1 2009-11-17 1223.0 Baker 1 2010-06-24 1122.7 Baker 2 2009-07-24 2819.0 Baker 2 2010-08-25 2971.6 ** Baker 1 2011-01-05 1410.0 Baker 2 2010-09-04 4671.6 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0

53 Regular ExpressionsIntroduction # could rely on context show_matches('-05-', readings) Baker 1 2009-11-17 1223.0 Baker 1 2010-06-24 1122.7 Baker 2 2009-07-24 2819.0 Baker 2 2010-08-25 2971.6 Baker 1 2011-01-05 1410.0 Baker 2 2010-09-04 4671.6 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0

54 Regular ExpressionsIntroduction # could rely on context show_matches('-05-', readings) Baker 1 2009-11-17 1223.0 Baker 1 2010-06-24 1122.7 Baker 2 2009-07-24 2819.0 Baker 2 2010-08-25 2971.6 Baker 1 2011-01-05 1410.0 Baker 2 2010-09-04 4671.6 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0 month has '-' before and after

55 Regular ExpressionsIntroduction # could rely on context show_matches('-05-', readings) Baker 1 2009-11-17 1223.0 Baker 1 2010-06-24 1122.7 Baker 2 2009-07-24 2819.0 Baker 2 2010-08-25 2971.6 Baker 1 2011-01-05 1410.0 Baker 2 2010-09-04 4671.6 Davison/May 23, 2010/1724.7 Pertwee/May 24, 2010/2103.8 Davison/June 19, 2010/1731.9 Davison/July 6, 2010/2010.7 Pertwee/Aug 4, 2010/1731.3 Pertwee/Sept 3, 2010/4981.0 so no matches

56 Regular ExpressionsIntroduction Matching is enough: we need to extract data

57 Regular ExpressionsIntroduction Matching is enough: we need to extract data When a regular expression matches, the library remembers what matched against every parenthesized sub-expression

58 Regular ExpressionsIntroduction # extract the year from the match match = re.search('(2009|2010|2011)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1) 2009

59 Regular ExpressionsIntroduction # extract the year from the match match = re.search('(2009|2010|2011)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1) 2009 pattern to match years is in parentheses

60 Regular ExpressionsIntroduction # extract the year from the match match = re.search('(2009|2010|2011)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1) 2009 first record from data

61 Regular ExpressionsIntroduction # extract the year from the match match = re.search('(2009|2010|2011)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1) 2009 first record from data (remember '\t' is a tab)

62 Regular ExpressionsIntroduction # extract the year from the match match = re.search('(2009|2010|2011)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1) 2009 re.search returns a match object if a match is found

63 Regular ExpressionsIntroduction # extract the year from the match match = re.search('(2009|2010|2011)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1) 2009 re.search returns a match object if a match is found returns None if there is no match

64 Regular ExpressionsIntroduction # extract the year from the match match = re.search('(2009|2010|2011)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1) 2009 match.group(k) returns the text that matched the k th sub-expression in the regular expression

65 Regular ExpressionsIntroduction # extract the year from the match match = re.search('(2009|2010|2011)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1) 2009 match.group(k) returns the text that matched the k th sub-expression in the regular expression k goes from 1 to N (number of matches), not 0 to N-1

66 Regular ExpressionsIntroduction # extract the year from the match match = re.search('(2009|2010|2011)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1) 2009 match.group(k) returns the text that matched the k th sub-expression in the regular expression k goes from 1 to N (number of matches), not 0 to N-1 because match.group(0) is all the text that was matched

67 Regular ExpressionsIntroduction Regular expression to match month would be "(01|02|03|04|05|06|07|08|09|10|11|12)"

68 Regular ExpressionsIntroduction Regular expression to match month would be "(01|02|03|04|05|06|07|08|09|10|11|12)" Expression to match day would be three times longer

69 Regular ExpressionsIntroduction Regular expression to match month would be "(01|02|03|04|05|06|07|08|09|10|11|12)" Expression to match day would be three times longer Use '.' (period) to match any single character

70 Regular ExpressionsIntroduction Regular expression to match month would be "(01|02|03|04|05|06|07|08|09|10|11|12)" Expression to match day would be three times longer Use '.' (period) to match any single character So '....-..-..' matches four characters, a dash, two more characters, another dash, and two more characters

71 Regular ExpressionsIntroduction Regular expression to match month would be "(01|02|03|04|05|06|07|08|09|10|11|12)" Expression to match day would be three times longer Use '.' (period) to match any single character So '....-..-..' matches four characters, a dash, two more characters, another dash, and two more characters And '(....)-(..)-(..)' remembers those matches individually

72 Regular ExpressionsIntroduction # match and extract YYYY-MM-DD date formats match = re.search('(....)-(..)-(..)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1), match.group(2), match.group(3) 2009 11 17

73 Regular ExpressionsIntroduction # match and extract YYYY-MM-DD date formats match = re.search('(....)-(..)-(..)', 'Baker 1\t2009-11-17\t1223.0') print match.group(1), match.group(2), match.group(3) 2009 11 17 Try doing that with substring searches...

74 Regular ExpressionsIntroduction # let's write another function to show match groups def show_groups(pattern, text): m = re.search(pattern, text) if m is None: print 'NO MATCH' return print 'all:', m.group(0) for i in range(1, 1 + len(m.groups())): print '%2d: %s' % (i, m.groups(i)) show_groups('(....)-(..)-(..)', 'Baker 1\t2009-11-17\t1223.0') 01: 2009 02: 11 03: 17

75 Regular ExpressionsIntroduction 1. Letters and digits match themselves.

76 Regular ExpressionsIntroduction 1. Letters and digits match themselves. 2. '|' means OR.

77 Regular ExpressionsIntroduction 1. Letters and digits match themselves. 2. '|' means OR. 3. '.' matches any single character.

78 Regular ExpressionsIntroduction 1. Letters and digits match themselves. 2. '|' means OR. 3. '.' matches any single character. 4. Use '()' to enforce grouping.

79 Regular ExpressionsIntroduction 1. Letters and digits match themselves. 2. '|' means OR. 3. '.' matches any single character. 4. Use '()' to enforce grouping. 5. re.search returns a match object or None.

80 Regular ExpressionsIntroduction 1. Letters and digits match themselves. 2. '|' means OR. 3. '.' matches any single character. 4. Use '()' to enforce grouping. 5. re.search returns a match object or None. 6. match.group(k) is the text that matched group k.

81 June 2010 created by Greg Wilson Copyright © Software Carpentry 2010 This work is licensed under the Creative Commons Attribution License See http://software-carpentry.org/license.html for more information.

Download ppt "Introduction Copyright © Software Carpentry 2010 This work is licensed under the Creative Commons Attribution License See"

Similar presentations

Regular Expressions Pattern and Match objects Genome 559: Introduction to Statistical and Computational Genomics Elhanan Borenstein.

Regular Expressions Pattern and Match objects Genome 559: Introduction to Statistical and Computational Genomics Elhanan Borenstein.
Python: Regular Expressions

Python: Regular Expressions
ISBN 0-321-33025-0 Regular expressions Mastering Regular Expressions by Jeffrey E. F. Friedl –(on reserve.

ISBN Regular expressions Mastering Regular Expressions by Jeffrey E. F. Friedl –(on reserve.
More Regular Expressions. List/Scalar Context for m// Last week, we said that m// returns ‘true’ or ‘false’ in scalar context. (really, 1 or 0). In list.

More Regular Expressions. List/Scalar Context for m// Last week, we said that m// returns ‘true’ or ‘false’ in scalar context. (really, 1 or 0). In list.
Regular Expressions Comp 2400: Fall 2008 Prof. Chris GauthierDickey.

Regular Expressions Comp 2400: Fall 2008 Prof. Chris GauthierDickey.
UNIX Filters.

UNIX Filters.
Regular Expressions. String Matching The problem of finding a string that “looks kind of like …” is common  e.g. finding useful delimiters in a file,

Regular Expressions. String Matching The problem of finding a string that “looks kind of like …” is common  e.g. finding useful delimiters in a file,
More on Regular Expressions 1. 107 - Regular Expressions More character classes \s matches any whitespace character (space, tab, newline etc) \w matches.

More on Regular Expressions Regular Expressions More character classes \s matches any whitespace character (space, tab, newline etc) \w matches.
Regular Expression A regular expression is a template that either matches or doesn’t match a given string.

Regular Expression A regular expression is a template that either matches or doesn’t match a given string.
Mechanics Copyright © Software Carpentry 2010 This work is licensed under the Creative Commons Attribution License See

Mechanics Copyright © Software Carpentry 2010 This work is licensed under the Creative Commons Attribution License See
Language Recognizer Connecting Type 3 languages and Finite State Automata Copyright © 2008-2014 – Curt Hill.

Language Recognizer Connecting Type 3 languages and Finite State Automata Copyright © – Curt Hill.
Pattern matching with regular expressions A common file processing requirement is to match strings within the file to a standard form, e.g. email address.

Pattern matching with regular expressions A common file processing requirement is to match strings within the file to a standard form, e.g. address.
Introduction to Computing Using Python Regular expressions Suppose we need to find all email addresses in a web page How do we recognize email addresses?

Introduction to Computing Using Python Regular expressions Suppose we need to find all addresses in a web page How do we recognize addresses?
Copyright © 2012 Pearson Education, Inc. Publishing as Pearson Addison-Wesley C H A P T E R 2 Input, Processing, and Output.

Copyright © 2012 Pearson Education, Inc. Publishing as Pearson Addison-Wesley C H A P T E R 2 Input, Processing, and Output.
Chapter 6 Functions -- QuickStart. The Practice of Computing Using Python, Punch & Enbody, Copyright © 2013 Pearson Education, Inc. What is a function?

Chapter 6 Functions -- QuickStart. "The Practice of Computing Using Python", Punch & Enbody, Copyright © 2013 Pearson Education, Inc. What is a function?
Numeric Types, Expressions, and Output ROBERT REAVES.

Numeric Types, Expressions, and Output ROBERT REAVES.
CIS 218 Advanced UNIX1 CIS 218 – Advanced UNIX (g)awk.

CIS 218 Advanced UNIX1 CIS 218 – Advanced UNIX (g)awk.
Input, Output, and Processing

Input, Output, and Processing
Strings The Basics. Strings can refer to a string variable as one variable or as many different components (characters) string values are delimited by.

Strings The Basics. Strings can refer to a string variable as one variable or as many different components (characters) string values are delimited by.
CIS 451: Regular Expressions Dr. Ralph D. Westfall January, 2009.

CIS 451: Regular Expressions Dr. Ralph D. Westfall January, 2009.

Similar presentations

Ads by Google