Measuring Complexity of Web Pages Using Gate Prepared by: The Who

Subject1: Can more meaningful indicators be extracted from the resources (webpages), e.g. a more interesting complexity, diversity or even other like sentiment.

Complexity Definition: How to learn the features associated to the difficulty to understand the resources.

Our Vision To employ entities liked to diverse contexts as a base to determine the complexity of a Webpage by: Gathering sets of Webpages from different domains Annotating the complexity of the pages (Crowdsourcing) Obtaining the set of named entities on each page (Gate) Determining a complexity score for each entity based on which pages it appears (Centrality / text ranking / Entity authority metrics: how many times it appears in the page vs how many entities are in that page and what is the page complexity score) Employing the set of weighted entities to predict a score for new pages Correlate the outputs with the commonly employed sentence metrics

Proposed approach

Run Entity and Terms Recognition on a sample from the data set . 1. Create Datastore for the sample 1 3 2

1 2 2. Populate the sample on to the corpus & save it to the datastore.

3. Run the TermRaider (it is already contain the annieGazetteer for entity recognition ) 1 2

4. Search for specific Annotation Type

5. Export the Terms and annotation set

Scoring Score the complexity of the entities This score is based on the average complexity score of documents that the entity appears on. 2

Calculate the page based on the scores of the entities that appear in it Score the complexity of the entities This score is based on the average complexity score of documents that the entity appears on.

Compare scores by the two methods Site Vanilla Score Proposed Score https://www.ijcai-18.org/cfp/ 0.6 .475 https://research.fb.com/programs/research-awards/proposals/computational-social-science-methodology-request-for-proposals/ 0.796 https://en.wikipedia.org/wiki/Cosine_similarity 0.568 .75 https://en.wikipedia.org/wiki/Dorian_Gray_(character) 0.536 .45 https://en.wikipedia.org/wiki/Goalball 0.52 .55 iswc2013_demo_36.html .375 https://en.wikipedia.org/wiki/Malawi 0.504 .775 https://en.wikipedia.org/wiki/Oscar_Wilde 0.464 .6 https://en.wikipedia.org/wiki/Scala_(programming_language) 0.48 .725 https://en.wikipedia.org/wiki/The_Picture_of_Dorian_Gray 0.528 https://en.wikipedia.org/wiki/Underwater_rugby .5

Thank You! Gracias! Ευχαριστώ! Prepared by: The Who