Emerging Trend Detection Shenzhi Li

Introduction What is an Emerging Trend? –An Emerging Trend is a topic area for which one can trace the growth of interest and utility over time. Example: “XML”, a technology that emerged in the mid 1990’s. Goals –Teach the students how to do literature search –offer students ways to go beyond the knowledge presented in the course work, by exploring current research trends.

semi-automatic methodology Step 1 –decide a domain area to search in Step 2 –Find the famous journals, conference or workshops among this domain area –Read papers to find several candidate emerging trends Step 3 –Find more evidence for the candidate trends Step 4 –Verify the candidate trends using INSPEC database

semi-automatic methodology While (there are more links from the search engine retrieved pages the desired number of candidate trends has not been found) { # Define searchTerm = "Candidate Emerging Trend" "constraining term“ # Define m = frequency of "searchTerm" in page # Define n = sum of all the frequencies of helper terms in page # Define L2 = an empty list used to store the candidate emerging trends Click on link = 1 // first link of interest in the search results if( "Year of page" in range of [current year-4, current year]) { if( m >= 2 n >= 2 ){ Accept the page; Add "searchTerm" to L2 if it is a candidate emerging trend; Look for the phrases with the highest frequency of occurrence. Add them to L2 if they qualify as candidate emerging trends (use domain knowledge); Give special attention to the line (or paragraph) containing the pattern “constraining term helper term”. Add phrases appearing in that paragraph (or sentence) that are judged to be candidate emerging trends to L2 (use domain knowledge) }else{ Reject the page; } } else{ reject the page; } Click on link++ or exit // click on next link of interest or exit }

Improvements Literature search –how to form queries –how to judge the quality of sources –how to tell the authority of the documents –Introduce several search engines

Improvements - tools

Noun Phrases Extraction Fuzzy Phrase Matching –can search using a combination of part of speech tags and exact words to find candidate trends appear in certain patterns –Example: “JJ+programming” will return phrases like “object-oriented programming”

Experiments Methodology –randomly split the students into two groups of roughly equal numbers –Students from both groups A and B were expected to have attended the lectures of the class. They were also expected to have introductory knowledge in the main topic area before participating in the experiment. – All the students had access to their textbooks, reference books and handouts given in the class. –Only Group B has the access to the multimedia tutorial which introduces the methodology for the algorithmic identification of an emerging trend and the tools designed to help searching.

Experiment Metrics –Precision –Did not use recall, as we do not have the resources to obtain a complete list of emerging trends at the time of this experimental evaluation, nor was it our pedagogical goal to have students retrieve all trends. Results –Conducted three experiments –with a confidence level of 95%, Group B in both classes performs significantly better than Group A.

Experiment Methodology RO1XO2 RO1O2 Results –no difference on the pretest between the two randomly assigned groups –the multimedia group showed a significant improvement in their scores from the pretest to the posttest –the scores on the posttest of multimedia group was higher than the control group –there was a greater increase in learning for the group who used the multimedia tool compared to those who only attended the lecture.

Lattice Definition –Let P be a set. An Order (or partial order) on P is a binary relation  on P such that, for all x,y,z  P, x  x x  y and y  x imply x=y x  y and y  z imply x  z –Let P be a non-empty ordered set. If x  y and x  y exist for all x,y  P, then P is called a lattice. If  S and  S exist for all S  P, then P is called a complete lattice.

Fuzzy Lattice Definition: A fuzzy lattice is a pair (L,  (x,y)), where L is a conventional lattice and  : S  [0,1] is a fuzzy membership function on the universe of discourse S =  (x,y): x,y  L . It is  (x,y) = 1 if and only if x  y in L.  is called inclusion measure. Definition: A function h: L  R on a complete lattice L, satisfies the following three properties: –h(O)=0, where O is the least element in L –u  w  h(u)  h(w), u,w  L –u  w  h(x  w)-h(x  u)  h(w)-h(u) x,u,w  L The inclusion measure of a lattice is defined as k(x,y) = h(y)/h(x  y) V. Petridis and v.G. Kaburlasos. "Clustering and Classification in Structured Data Domains Using Fuzzy Lattice Neurocomputing". IEEE Trans on Knowledge and Data Engineering, Vol 12, No 2, March, 2001.

Formal Concept Analysis A context is a triple (G,M,I) where G and M are sets and I  G  M. The elements of G and M are called objects and attributes respectively. For A  G and B  M, define –A’ =  m  M | (  g  A) gIm , –B’ =  g  G | (  m  B) gIm  ; A concept of the context (G,M,I) is defined to be a pair (A,B) where A  G, B  M, A’=B and B’=A. The set of all concepts of the context (G,M,I) is denoted by B(G,M,I). For concepts (A1,B1) and (A2,B2) in B(G,M,I) we write (A1,B1)  (A2,B2), and say that (A1,B1) is a subconcept of (A2,B2), or that (A2,B2) is a superconcept of (A1,B1), if A1  A2 (which is equivalent to B2  B1). (B(G,M,I);  ) is a complete lattice. it is known as concept lattice of the context (G,M,I).

Lattice vs. Association Rules Transact ion ABCDEF 1*** 2**** 3** 4*** 5*** C, (1235) B, (124)D, (245) ABC, (12) CE,(35) BDF,(4) BD, (24) CD, (25) ABCD,(2) CDE, (5) , (12345) ABCDEF, 

Lattice vs. Graph The power set of graph G is lattice- ordered, and corresponding lattice- ordering, lattice-meet, and lattice-joint are conventional set-inclusion, set-intersection and set-union. So G is a complete lattice. The least element and the greatest element are empty set and the master graph.

Conclusions Apply fuzzy concept lattice to association rules building Embed time element and latent semantic into concept lattice to model term-doc matrix Develop algorithms to mine fuzzy concept lattice to detect emerging trends