Automatic Blog Monitoring and Summarization Ka Cheung “Richard” Sia PhD Prospectus

With/without organized access

Inaccessible? By AskJeeves

Introduction Organized access to blogs  Full coverage  Reflect changes quickly  Filtered and organized presentation Intended Contributions  Efficient techniques to harvest blogs  Algorithms to monitor frequently changing data sources  Algorithms to reconstruct implicit networks and compose topic summaries

Modules Monitoring Collection (future work) Topic detection and tracking (future work) Conclusion

Monitoring Preliminary results

Framework A central server monitors data source changes and provides succinct summaries to users

Overview New challenges  Content change more rapidly with recurring pattern  More time-sensitive requirements Modeling of posting update Definition of delay Strategies for allocation and scheduling

Characteristics Homogeneous Poisson model λ(t) = λ at any t Periodic inhomogeneous Poisson model λ(t) = λ(t-nT), n=1,2,…

Definition of metrics Delay of a data source sum of elapsed time for every post Delay experienced by the aggregator

Definition of metrics τ j – retrieval time λ(t) – posting rate Expected delay  Homogeneous Poisson model  Inhomogeneous Poisson model

Problem formulation Minimization of expected delay experienced by the aggregator under constraint of limited resources. Schedule τ j ’s such that is minimized.

Approach Resource allocation  How often to contact data sources?  O 1 is more active than O 2, how much more often should we contact O 1 than O 2 ? Retrieval scheduling  When to contact a data source?  3 retrievals are allocated for O 1, when should these 3 retrievals be located?

Resource allocation Consider n data source O 1, …, O n  λ i – posting rate of O i  w i – weight of O i  N – total number of retrievals per day  m i – number of retrievals per day allocated to O i Optimal allocation

Retrieval scheduling m retrieval(s) per day are allocated to a data source O, how should we schedule these m retrievals? m=1 m>1

Single retrieval per period λ(t) = 1, t [0,1], λ(t)=0, t [1,2] Periodicity T=2  τ = 0.5, expected delay = 0.75  τ = 1, expected delay = 0.5  τ = 2, expected delay = 1.5

Single retrieval per period For a data source with posting rate λ(t) and period T, the expected delay when retrieved at time τ is given by:

Multiple retrievals per period m retrievals per period are allocated, when scheduled at time τ 1, …, τ m, the expected delay is given by:

Example 6 retrievals for λ(t)=2+2sin(2πt)

Experiment Data – 10k RSS feeds over Oct – Dec 2004

Performance CGM03 – optimize for “age” Ours – both resource allocation and retrieval scheduling

Size of estimation window Resource constraint: 4 retrievals per day per feeds on average 2 weeks is an appropriate choice

Predictability of posting rate 90% of the RSS feeds post consistently

Summaries and extensions Resource allocation is more aggressive Retrieval scheduling optimizes within individual data source Include user access pattern Variable retrieval cost

Collection Future work

Collection Blog hosting website Central repository ~5.3M URLs from weblogs.com limited and contaminated Crawling Retrieve maximum number of blog while reducing number of irrelevant pages downloaded DomainCountCategory spaces.msn.com839,663Blog blogspot.com362,957Blog wretch.cc116,161Blog search-net101.com89,750Spam/ads abalty.com86,329Spam/ads search-now854.com80,109Spam/ads bigebiz.org79,059Spam/ads

Collection Blogs are inter-connected (blogrolls) Selectively following links, discovering hubs for blogs blog [1] Chakrabarti et.al. “Focused Crawling: A New Approach to Topic-specific Web Resource Discovery”, The International WWW conference 1999

Relinquishment of blogs Detection of abandoned blog to save resource [2] D.R. Cox “Regression models and life-tables (with discussion)” Journal of the Royal Statistical Society, B(34), 1972 [3] Gina Venolia “A Matter of Life or Death: Modeling Blog Mortality” Technical report, Microsoft Research

Topic detection and tracking Future work

Overview Characteristics  Document stream  Traces of information propagation among blogs Challenges  Modeling growth and death of a topic  Ranking of blog articles  Malicious content

Influence network in blogs Information are “diffused” among blogs Indicator of popularity Social relationship among bloggers

Influence network in blogs Four major patterns of propagation Reconstruction of implicit network  Ranking (source authority)  Advertising campaign

Data characteristics ~ % daily content are new

Data characteristics Same content last for ~8 days

Topics Topics with different lifespan  Bursty  Mid-range  Sustaining Evolving of topic [4] J. Kleinberg, “Bursty and Hierarchical Structure in Streams” in SIGKDD 2002 [5] J. Kleinberg, “Temploral Dynamics of On-Line Information Streams” Data Stream Management: Processing High-Speed Data Stream, Springer 2005

Document similarity Sparse and diverse ~400 articles clustered into 21 clusters out of 10,000 daily articles (by DBSCAN)

Framework Document stream approach  Filtering  Aggregation

Problems Selecting a representative subset of documents from a topic cluster  Coverage  Distinctiveness among subset Ranking of documents  Time  Source authority

Conclusion 1. Efficient collection of blogs and modeling the relinquishment 2. Monitoring and retrieval scheduling of rapidly changing data sources 3. Composing topic summary 1. Reconstruction of an implicit influence network 2. Representative document selection problem

End Questions?

More examples

Major posting patterns K – means clustering