New Event Detection at UMass Amherst Giridhar Kumaran and James Allan

CIIR, UMass Amherst2 Preprocessing  Lemur Toolkit for tokenization, stopping, k-stemming  BBN Identifinder™ for extracting named entities

CIIR, UMass Amherst3 Systems fielded  Submitted four systems  Didn’t include last year’s system Classification according to LDC categories and term – pruning Didn’t work on exclusively NW story corpus

CIIR, UMass Amherst4 Primary system – UMass1  Utility of named entities acknowledged  Failure analysis indicates Large number of old stories have low confidence score (false alarms) Conflict with new story scores Reasons  Stories on multiple topics  Diffuse topics  Varying document lengths

CIIR, UMass Amherst5 Primary system – UMass1  Focus Identify old stories better – affects cost  Clue Most old stories get low confidence scores as topics linked by  only named entities (large number)  only non-named entities (few)

CIIR, UMass Amherst6 Primary system – UMass1  Approach Look at the set of closest matching stories If consistently high named entity or non-named entity match modify confidence score

CIIR, UMass Amherst7 Primary system – UMass1  Procedure Double original confidence score if less than a threshold Gradually reduce score towards original score if set of closest stories match neither named entities nor non-named entities

CIIR, UMass Amherst8 UMass1 – Examples from TDT3  Russian Financial Crisis - Old Story APW AllSimNESimnoNESim APW APW APW APW APW APW

CIIR, UMass Amherst9 UMass1 – Examples from TDT3  Russian Financial Crisis - Old Story APW AllSimNESimnoNESim APW APW APW APW APW APW

CIIR, UMass Amherst10 UMass1 – Examples from TDT3  Russian Financial Crisis - Old Story APW AllSimNESimnoNESim APW APW APW APW APW APW Threshold = 0.1

CIIR, UMass Amherst11 UMass1 – Examples from TDT3  Russian Financial Crisis - Old Story APW AllSimNESimnoNESim APW APW APW APW APW APW Threshold = 0.1

CIIR, UMass Amherst12 UMass1 – Examples from TDT3  Russian Financial Crisis - Old Story APW AllSimNESimnoNESim APW * APW APW APW APW APW Threshold = 0.1

CIIR, UMass Amherst13 UMass1 – Examples from TDT3  Thai Airbus Crash - New Story APW AllSimNESimnoNESim APW * APW APW APW APW APW

CIIR, UMass Amherst14 UMass1 on TDT3

CIIR, UMass Amherst15 UMass1 on TDT3

CIIR, UMass Amherst16 UMass2  Basic vector space model system  Compare with all preceding stories  Return highest cosine match

CIIR, UMass Amherst17 UMass3  Same model as UMass2  TDT5 – Very large collection  Practical system  Compare with a maximum of stories with highest coordination match Faster

CIIR, UMass Amherst18 UMass4  Similar to UMass1  Rationale is the same  Consider top five matches  Use different formula for modifying confidence score

CIIR, UMass Amherst19 Performance Summary System Topic weighted min. cost (TDT5) Topic weighted min. cost (TDT4) UMass1 – Modify confidence score based on evidence UMass2 – Basic vector space model UMass3 – UMass2 + restriction on number of documents compared with UMass4 – UMass1 with different formula

CIIR, UMass Amherst20 Summary  Basic vector space model did the best  Restricting number of stories to be compared with Improved system speed Didn’t improve performance  Primary system did extremely well on training data, but failed on TDT5