1. HTST Evaluation Notes

2. Outline of Stable Version of HTST Stable version of HTST contains: – HT sense knowledge base – Auxiliary sub-lexicons and data extracted from HT, e.g. highly polysemous words, polyseme density etc. – Context feature (USAS tags) model data extracted from OED word sense definitions. – Main software modules CLAWS USAS VARD HT-OED based components developed in SAMUELS

3. Evaluation HTST is evaluated on six manually annotated test texts. Test data: – Five test texts manually annotated by Fraser; – One EEBO test text manually annotated by Jane. – Full test data set will contain ten texts. Evaluation criteria – General performance in terms of precision – Impact of OED contextual information – Impact of time filtering Further evaluation on full test data is under way

4. HTST Overall Performance Test fileMain HT codesThematic codes Hans182083.15%86.17% Hans200178.78%80.52% Fiction179.51%79.83% Fiction280.28%80.48% History84.37%84.43% 1621-Newes-out- of-France 85.69%86.67% Note: VARD is used for EEBO sample “1621-Newes-out-of- France”, but not used for other test data.

5. Experiment with OED Information Main HT codesThematic codes Test file\codeWith OEDNo OEDWith OEDNo OED Hans182083.15%81.01%86.17%83.93% Hans200178.78%76.36%80.52%78.00% Fiction179.51%76.82%79.83%77.15% Fiction280.28%79.18%80.48%79.38% History84.37%81.43%84.43%81.49% 1621-Newes85.69%87.06%86.67%87.75% Note: OED information helped in most cases (modern English), but decreased precision for EEBO sample. Possible cause is that OED definitions are all written in modern English.

6. Time Filtering on EEBO sample Published in 1621 (Main HT Codes) Year range16501700175018001850190019502000 50084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 55084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 60084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 65084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 70084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 75084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 80084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 85084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 90084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 95084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 100084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 105084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 110084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 115084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 120084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 125084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 130084.70%85.19%85.00%84.70%84.41%84.21%84.11%84.01% 135084.60%85.09%84.90%84.60%84.31%84.11%84.01%83.92% 140084.60%85.09%84.80%84.50%84.21%84.01%83.92%83.82% 145084.70%85.19%84.90%84.60%84.31%84.01%83.92%83.82% 150084.70%85.19%85.00%84.70%84.41%84.11%84.01%83.92% 155084.80%85.29% 85.00%84.70%84.41%84.31%84.21% 160085.19%85.69% 85.39%85.09%84.80%84.70%84.60%

7. Time Filtering on EEBO Sample Published in 1621 (Thematic Codes) Year range16501700175018001850190019502000 50085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 55085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 60085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 65085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 70085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 75085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 80085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 85085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 90085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 95085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 100085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 105085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 110085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 115085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 120085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 125085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 130085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 135085.88%86.37%86.07%85.78%85.58%85.39%85.29%85.19% 140085.98%86.47%86.07%85.78%85.58%85.39%85.29%85.19% 145086.07%86.56%86.17%85.88%85.68%85.39%85.29%85.19% 150086.07%86.56%86.27%85.98%85.78%85.49%85.39%85.29% 155086.17%86.67%86.47%86.17%85.98%85.68%85.58%85.49% 160086.07%86.56% 86.27%86.07%85.58%85.49%85.39%

8. Observation On average, about 82% precision is expected. With proper parameter setting, thematic code tagging can reach nearly 88% on some types of texts. Need further improvement by tuning implemented methods and introducing more reliable methods. OED data contains noise caused by the inconsistent HT versions. If OED entries can be precisely mapped to latest HT codes in future, it should improve the tagger. Larger reliable test data is needed for further development.