Business Identification: Local Neighborhood Alexander Darino

Outline Where Am I? project obtains geolocation of camera from image Objective: Obtain the geolocation and address of Businesses in image – Assume Business is nearby, eg. < 100m from camera – Compare methods of obtaining this information

Outline Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses ImageOCR Detected Text Business Name Matching Business Identification

Outline This Week: – Finding Local Businesses via Geocode Search – Finding Local Addresses via Reverse Geocoding – Extracting Identifying Text (ie. store names) via Optical Character Recognition (OCR) – Matching OCR text to Business Names Next Steps/Weekend Objectives Acknowledgements

Obtaining Business Names ✓

Local Businesses: Geocode Search Used Three Place-Search APIs: – Yelp API - detailed yellow page-type results – Google Places API - "Skinny" + Reference to more information – CityGrid API - minimal yellow page-type results Used by Yellow Pages, Super Pages At present, only interested in business names Aggregated names from all three APIs Example (next slide)

Local Businesses: Geocode Search Denham & Company Salon Ullrich's Shoe Repairing Nicholas Coffee Co Bella Sera On the Square A & J Ribs Starbucks Coffee Jenny Lee Bakery Galardi's 30 Minute Cleaners Jimmy John's Gourmet Sandwiches Charley's Grilled Subs Fresh Corner Lagondola Pizzeria & Restaurant Camera Repair Service Inc Pittsburgh Cigar Bar Original Oyster House MixStirs 1902 Tavern Costanzo's Pittsburgh Silver Llc Graeme St Galardi's 30 Minute Cleaners Denham & Co Salon Bruegger's Bagel Bakery Nicholas Coffee Co Market Square Fat Tommy's Pizzeria Mixstirs Cafe Giggles Rycon Construction Inc Garbera, Dennis C, Dds - Emmert Dental Assoc Bella Sera on the Square Mancini's Bread Co Las Velas Ciao Baby Washington Reprographics Inc Highmark Life Insurance Co Fischer, Donald R, Md - Highmark Life Insurance Co Jimmy John's Lynx Energy Partners Inc Emmert Dental Assoc

Local Businesses: Geocode Search Results: 12 Success, 3 Partial Q9: First Presbyterian Church [turns out it wasn't a cathedral] (SUCCESS) Q28: Moe's (SUCCESS) Q34: Bruegger's Bagels (SUCCESS) Q35: Breuggers, Tavern, Nicholas (SUCCESS) Q42: Tavern, Nicholas, Constanzo's [in distance] (SUCCESS) Q57: Tambellini (SUCCESS) Q63: Benedum Center (SUCCESS) Q141: Roberts/7-Eleven (PARTIAL - misses Roberts) Q200: Goodyear (SUCCESS) Q238: Far from Bruegger's, Tavern, Nicholas (PARTIAL - misses Tavern) Q246: Some theater (can't read it) (SUCCESS) Q249: George Aikens (SUCCESS) Q260: Dogs Dun Wright, Cherrie's diner (SUCCESS) Q300: Giggles, Bruegger's, Tavern (in distance) (SUCCESS) Q318: Fifth Avenue Place, Wines & Spirits (PARTIAL - misses Wine & Spirits)

Local Businesses: Geocode Search Strengths – Aggregated results almost always found Business of interest Weaknesses – Each API limits query result set size - this is why we aggregate – Contacted Yelp, Google, CityGrid for extended API Access. Heard back from CityGrid; conference call next week. – Only businesses listed – Not all businesses listed All but one "Partial" result were for unlisted businesses Limitations – Have only tested for 15 Pittsburgh images - unknown result quality for rural areas.

Local Businesses: Geocode Search ✓ ✓*✓* ✓ * Implicitly verified: APIs can search by latitude/longitude OR address

Local Addresses: Reverse Geocoding Used Two Reverse-Geocoding APIs – Google: provides a range of addresses on the same road Usually the road is correct, but sometime's it's slightly off Sometimes the road is correct, but the actual address number is not in the range Bing: provides one or two proximate addresses – Rates it's own confidence. Even 'Medium' confidences are very accurate – Address is never exact, but is almost always adjacent to correct address – Results returned never consistent: always returns one or the other or both of the two addresses regardless of confidence level

Local Addresses: Reverse Geocoding Intent: Get up to ~500 nearby addresses No Address Search API Available ✓ ✓*✓* ✓ ✗

Extracting Identifying Text: OCR Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses ImageOCR Detected Text Business Name Matching Business Identification

Extracting Identifying Text: OCR Given: – List of nearby businesses (names, addresses, etc) – Image containing businesses with visible names Objective: – Extract name of businesses from image – Identify businesses located in image Match names extracted from image to names in business list

Extracting Identifying Text: OCR Used Two OCR APIs: – GNU OCR (Ocrad) – GOCR OCR APIs highly sensitive to: – Font (only works well with roman font) – Perspective – Scale – Binarization Threshold – Dark on Light vs. Light on Dark (inversion)

Extracting Identifying Text: OCR OCR API evaluations – Ocrad - could not yield any meaningful data across over 200 scale/threshold/inversion combinations – GOCR - produced good results across 10 scales with and without inversion using threshold automatically determined by Otsu's method Examples of GOCR output (next slides)

Extracting Identifying Text: OCR

n. c o.a...u oU..D.oa..e......_RuEGGE..KERy..J...w L M. II.....c.....i l..J.t...llt...l SHA.P. It..tllt _. l...J y. _.c _.... _tt.._....t.._.r t.t_t.._.._.l.. J.r.r.I.

Extracting Identifying Text: OCR

u _nq......eoR.E.l.e...í....e...n....n....n.e.R.E...e....o. _....E.R.E.IKE I.ltlO rE..o......E.....I.K.E.o..... J.n....c...E.R.E.I.E M..E.R.E...E...a J...Gu. ge..ge E.F.._.....E..gE.D... fUlI..lll.lll.IIi.l..Xl..

Extracting Identifying Text: OCR

..e_..w.. _......D u J.....J n......n n _..r.l_d..J.ec.m._..n J.n.._...tn.. ct..._ D.u.v... e. n.... u.. Y.._ w. n. n....Jn G..o..r..._ J...m l.t..l.tt.l.._w _....l....t j..i lI.i..

Extracting Identifying Text: OCR

__. ncu_.l..._..._J...ne _n._..v.....ra......d_..._ i..n..U ll REsT.unAN...r. c.....r...T t.rJll......m...c.....n J n. I..c...r.r ESTAU.ANT.r.O....c.cc. Note: Even though "Tambellini" is a roman font, it is too stretched to be picked up by GOCR

Extracting Identifying Text: OCR Strengths – Applicable to expected input of orthogonal images – Output can be run through word similarity matching algorithms Weaknesses – Only works well(-ish) for strictly roman font Limitations – Will perform poorly for artistic fonts and business signs Conclusion – By itself, OCR is not the best approach towards Business identification (poor recognition, franchises, perspective, etc) – OCR could be used as part of Business identification voting scheme

Matching OCR Text to Business Names Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses ImageOCR Detected Text Business Name Matching Business Identification

Matching OCR Text to Business Names Fuzzy String Matching: TRE Package – Approximate Regular Expression Matching – Returns edit-distance of matched text Filter OCR text – Trimming – Chunking – Uselessness (ie. Less than two letters) Developing algorithm to rate confidence of business name appearing in image

Matching OCR Text to Business Names

Next Steps/Weekend Objectives Implement ‘chunking’ to OCR output Evaluate and refine algorithm against multiple inputs Detect location of text in image

Acknowledgements Subh – Directed us to the Ocrad and GOCR OCR packages – Provided feedback on how to calibrate OCR packages to extract meaningful text (eg. scaling, inversion, etc)

Thank You