Corpus Statistics Example: Collecting Data from Potato Chip Bags

Defining Corpus Corpus to compare language between types of snack chips * –To choose representative sample, visited large local Wegman’s –Randomly choose a large selection of brands, choose the one most “normal” flavor of that brand Only two brands from Frito-lay * Thanks to a student of Dan Jurafsky’s for this idea. 2

Defining comparison types Define the two corpora to compare as gourmet/upscale or regular snack chips –Method: gourmet/upscale chips are sold in Nature’s Marketplace section, regular chips are sold in the regular snack aisle 3

Collecting Text Wrote all text occurring on the front of the package and all text on the back of the package –Omitted weights from front –Omitted nutrition label from back Typed words by hand into document Limitations: –Extremely small amount of text to analyze, statistical results could be dubious –Imbalance between the two comparison types Gourmet packages have lots of text compared to regular packages 4

Processing Text Chose to lower-case text Removed stop words and punctuation from word frequency lists by hand Removed stop words (Smart.English.stop) and punctuation in bigram analysis 5

Word Frequencies 6 chips 26 natural 21 potato 14 fat 11 more 10 ingredients 9 baked 7 great 7 make 7 organic 7 sticks 7 booty 6 delicious 6 kettle 6 made 6 pirate 6 snack 6 trans 6 Gourmet Top Words chips 19 potato 10 brand 8 fat 8 oil 7 fresh 6 made 6 santitas 6 canola 5 date 5 guaranteed 5 printed 5 tortilla 5 com 4 corn 4 flavor 4 natural 4 original 4 snacks 4 Regular Top Words

7 exotic vegetable real food eat healthier french fry lightly sea american rivers boulder canyon lesser evil terra exotic fake colors gluten free green energy lightly salted sea salted small company taste naturally trans fats wholesome potatoes krinkle sticks canyon products big flavor company family crispy chip food ingredients kettle cooked perfect crunch visit www grams trans cape cod good fun authentic mexican classic chip printed date cooked cape great multigrain multigrain snacks canola oil guaranteed fresh authentic flavor brand tortilla delicious natural reduced fat trans fat multigrain taste santitas brand regular potato enjoy santitas santitas authentic natural oil cod potato fat chip natural flavor original corn original flavor GR

Question What kind of people are the companies appealing to for gourmet and regular snack chips? Sketch of how to approach analysis: –Appeal to health issues for gourmet chips Words: natural, baked, organic –Fresh on regular list AR: real food, eat healthier, gluten free, taste naturally, wholesome potatoes, … high on gourmet list –Appeal to environmental issues for gourmet chips AR: green energy –Chatty, family approach to gourmet chips Small company, company family –Regular chips issue words: good fun 8

Other Observations Regular chips just don’t use a lot of text to appeal to customers Text more standardized on regular chips –Guaranteed fresh by printed date on almost all packages Measures limited by small amounts of text 9

Brainstorm ideas for Corpora? Gutenberg, NLTK book (Presidents’ Inaugural Addresses), positive and negative opinions from annotated text or from debate blogs, BNC, ANC, collect documents on a two topics, compare text on same topic from different eras, Compare song lyrics from different authors or genres ?? 10