A Fully Automated Object Extraction System for the World Wide Web a paper by David Buttler, Ling Liu and Calton Pu, Georgia Tech

Why’d they do it? Identifying object regions and boundaries has been done manually and with some automation mostly relying on syntactic knowledge (ie HTML). Embley, Jiang & Ng (Hmmm… must be some famous scientists in Germany) developed a pretty sweet heuristics-based automatic object extraction system, which we want to copy but throw out the ontology heuristic – and maybe throw in a few ideas of our own. Embley, Jiang & Ng (Hmmm… must be some famous scientists in Germany) developed a pretty sweet heuristics-based automatic object extraction system, which we want to copy but throw out the ontology heuristic – and maybe throw in a few ideas of our own.

Omini (not the book after Jarom) Fully-automated extraction Parses a page into a tree structure Parses a page into a tree structure Locates smallest subtree with all objects Locates smallest subtree with all objects Reduces possibilities for next step Reduces possibilities for next step Finds correct object separator tags Finds correct object separator tags Contributions to IE A few algorithms for subtree extraction and object extraction A few algorithms for subtree extraction and object extraction Most the other stuff is already known Most the other stuff is already known

Some Terms & Definitions Well-Formed Web Document No brackets besides tags No brackets besides tags ALL tags are paired (even,, etc.) ALL tags are paired (even,, etc.) Attribute values in a tag are in quotes Attribute values in a tag are in quotes Nested tags do not overlap Nested tags do not overlap Well-Formed Doc  Tag Tree

System Architecture

Phase 2, Part A: Subtree Extraction 3 Heuristics used to find the minimal subtree containing all objects of interest Fanout Fanout Content Size Content Size Tag Count Tag Count

Phase 2, Part B: Object Separator Extraction Combination of 5 Heuristics SD (Standard Deviation) & RP (Repeating Pattern) are taken from BYU. SD (Standard Deviation) & RP (Repeating Pattern) are taken from BYU. SB (Sibling tag), PP (Partial Path) are new. SB (Sibling tag), PP (Partial Path) are new. IPS (Identifiable Path Separator) is an extension of BYU’s IT (Identifiable Tag). IPS (Identifiable Path Separator) is an extension of BYU’s IT (Identifiable Tag).

Phase 2, Part B Continued: Object Separator Heuristics SD – Distance between consecutive occurrences of a candidate tag. (Objects usually the same size.) RP – Absolute value of difference between pairs of tags together and alone. (Pattern of tags usually means just one thing.) IPS – Ranks tags according to a table of common object separators.

Phase 2, Part B Continued: Object Separator Heuristics SB – Pairs of tags that are immediate siblings of minimal subtree. (ie … … … (# object separators should = # objects) PP – Counts occurrences of same path of tags from a node. (Multiple instances of object should have same object structure.)

Phase 2, Part B Continued: Object Separator Heuristics Combining Heuristics Probability that tag is an object separator if 3 heuristics say 78%, 63% and 85%: 99% Probability that tag is an object separator if 3 heuristics say 78%, 63% and 85%: 99% *63-78*85-63*85+78*63*85 = 99% *63-78*85-63*85+78*63*85 = 99% Combination of all 5 heuristics is best. Combination of all 5 heuristics is best.

Phase 3: Object Extraction Candidate Object Construction Uses Object Separator Tag from Phase 2 Uses Object Separator Tag from Phase 2 Object Extraction Refinement Removes objects that may not be of the same structure, too big or too small Removes objects that may not be of the same structure, too big or too small

Results Ran Omini on 1,500 pages across 25 sites Using the combination of all 5 heuristics: 94% of Object Separators picked correctly 94% of Object Separators picked correctly 100% Precision and 98% Recall 100% Precision and 98% Recall vs BYU Omini as good if not better in all tests Omini as good if not better in all tests Over 5 websites in March 2000: Over 5 websites in March 2000: BYU: 59% success rate Omini: 93% success rate

Criticism of BYU System IT (Identifiable Tag) vs IPS (Identifiable Path Separator): IPS changes tag table based on the node at which the minimal subtree is anchored. IPS changes tag table based on the node at which the minimal subtree is anchored. PP (Partial Path) vs HC (Highest Count): By itself, HC not very successful By itself, HC not very successful In combination with other heuristics, HC can actually make the total accuracy worse! In combination with other heuristics, HC can actually make the total accuracy worse! PP just like HC on some websites PP just like HC on some websites Ontology approach uses human intervention – if goal is fully automated, this won’t do.