1. Interleaved Evaluation for Retrospective Summarization and
Prospective Notification on Document Streams
Bicheng Fang

2. Background Primary task: Retrospective summarization.
Prospective notification.
The primary task of this paper is to design an evaluation methodology, given two information seeking scenarios.
Lets see this pic.
Suppose there’s something big happened, and you didn’t catch up with the news. So you want to have a clear view of what happened so far. So that is R S, which summarize all key points so far.
Then, you want to keep alerted when there’s follow ups in the future.
So given these two secnarios, the author want to design an evaluation method.

3. 2. Challenges Why not existing interleaving strategies? Temporality
Results of different lengths
Redundancy
Temporality is the most important part in this task. Cuz they want to organize the results by chronological order. While The existing strategies have nothing to do with chronological order
In typical web search, most interleaving strategies assume ranked lists are of equal length, while it is not true in this case. We will see an example soon. Besides, Most existing interleaving strategies don't tell us what to do when we run out of results. For example, when there’s no related tweets in that day.
And the 3rd one is straight forward. In this task, they want to organize the result by chronological order, and give back non-redundant results.
But I should point out here, that the core conception of this paper is not to develop a system that to accomplish the tasks, but to evaluate the results.

4. 3. Interleaving methodology
Interleaving strategy:
Temporal interleaving.
User interactions and Credit assignment:
Assumption: Go through the output, earliest -> latest
3 judgements: Relevant, Not relevant, Redundant
Then lets move on to their interleavfing methodology:
The inter leaving strategy is called temopral interleaving, which is quite simple. Just combine the results by chronological order.
So suppose you have two serach engine, s A and s B. They all found some related documents, then the interleaving strategy will combine them directly. Pay attention to here, both system a and s B has t28, and the interleaved result will pick up only one t28, to maintatin non-redundancy.
Then lets talk about the user interactions and credit assignments:

5. 3. Interleaving methodology
Credit assignment:
Relevant -> +1
Non-Relevant -> 0
Duplication -> depends on previous contents.
Thats is the whole idea of their evaluation strategy. Quite simple and strghait forward/
And then they carried out several experiments.

6. 4. Simulation Results Retrospective summarization:
Interleaving evaluation VS. Batch evaluation (human annotation)
They use tweets as their evaluating documents, and carried out some experiments.
The first table is about retrospective summarization:
They compare their interleaving evaluation with batch evaluation results.

7. 4. Simulation Results Prospective Notification:
No relevant follow ups. A: wrong docs. B: no return.
The second table is about prespective notification. As we can see, the performance under retaining quiet days are poor. While it is much higher under discarding quiet days. So what’s the difference here?
As I mentioned before. There may be some days that has no relevant follow ups. However, the system can not know that before head.
A will be given 0 credit for returning wrong tweets. While B will be given 0 credits too, cause the system cannot predict in the middle of that day, if there will be nothing happened or just not happened yet.

8. 5. Summarize
Interleaving method for 2 different information seeking tasks.
Temporal interleaving strategy.
Validation.