1. Tomaž Erjavec1, Adam Kilgarriff2, Irena Srdanović Erjavec3
A large public-access Japanese corpus and its query tool - JapWaC and Sketch Engine -
Tomaž Erjavec1, Adam Kilgarriff2, Irena Srdanović Erjavec3
1Jožef Stefan Institute, Slovenia
2Lexical Computing Ltd. and University of Leeds, UK
3Tokyo Institute of Technology, Japan

2. Overview The case for corpora The case for web corpora
How JapWaC was created
Sketch Engine (SkE)
Demo-ing JapWaC & SkE
Future work
Access to JapWaC & SkE

3. Corpora A sample of a language Useful for studying the language
Language is diverse
Big samples needed, to catch everything
Good tools needed, for large amounts of data
Last 15 years
Big samples are easier to gather
Tools are better
Rapid growth in corpus methods

4. Web corpora Web is huge, free, easily accessible
(Non-)linguists use it for lang. check/research
Skewed?
Keller and Lapata 03:
web results match human judgements well
the large amount of data outweighs the “noise” problem
Web importance as a resource is growing
David Crystal “Language and the Internet” 06:
“new linguistic medium that we cannot ignore”
Web-corpus expertise is growing (WaCky etc.)
as shown by Keller and Lapata, web results match human judgements well, often better than cleaner but smaller corpora
the large amount of data available outweighs the problems associated with using the web
as a corpus (such as the fact that it is noisy and unbalanced).
Preliminary
evidence suggests the ’balance’ of our German
corpus compares favourably with that of a
newswire corpus (though of course any such claim
begs a number of open research questions about
corpus comparability).

5. Steps to compile web corpora (Sharoff, Baroni)
1. Get URL list for required language
~500 most frequent word forms
not function words; for general-purpose corpora, words that do not belong to a spec. domain
queries, 4 words, top 10 URLs
2. Download HTML pages
3. Normalize encoding (to UTF-8)
4. HTML clean-up
boilerplate removal: HTML tags, Java code, navigation frames,…
5. Extract meta-data (URL, title, date,…)
6. Linguistic annotation

6. Steps for JapWaC
URL list of pages in Japanese provided by Serge Sharoff
word, lemma and PoS frequency lists for Japanese, c.f.
Files downloaded and cleaned with BootCat
by Marco Baroni and others from the WaCky project, c.f.
Segmented, tokenised, tagged with Chasen
Translated Chasen tags to English
Converted to Sketch Engine format and loaded

7. Example file The file size is 7038669 kB, showing first 1 kB.
<doc id="
<s>
月々 月々 N.Adv
２ ２ N.Num
６ ６ N.Num
３ ３ N.Num
円 円 N.Suff.msr
で だ Aux
、 、 Sym.c
あなた あなた N.Pron.g
も も P.bind
ブログデビュー ブログデビュー Unknown
し する V.free
て て P.Conj
み みる V.bnd
ませ ます Aux
ん ん Aux
か か P.advcoordfin
？ ？ Sym.g
</s>

8. Basic corpus statistics
49,554 URLs (i.e. HTML files)
16,072 sites (2 domains)
12,759,201 sentences (Chasen)
409,384,411 tokens (Chasen)
7.3 GB filesize
tokens/file:
8,263 Average
5,001 Median
3 Min
170,693 Max

9. URL statistics (top ranking domains, sites and keywords)

10. Chasen POS statistics

11. The Sketch Engine Leading corpus query system Any corpus, any language
Web-based
No software to install
Concordance
Word sketches
“one-page, corpus based account of a word’s grammatical and collocational behaviour”
Thesaurus
Word Sketch Difference

12. Macmillan English Dictionary
Use of Sketch Engine
Macmillan English Dictionary
For Advanced Learners
Ed: Rundell, 2002
Lexicography
Language learning
Linguistic research .

13. extensive comparisons in James Curran‘s thesis (about statistics)
[[Tomaz is right ...  'salience' is just a name we are using (rather presumptuously) for the statistic we use, which is one of the same family as all the others (though we reckon it is the best suited of the family, that's why we use it, see also extensive comparisons in James Curran's thesis).  Stat we currently use is based on Dice coefficient.  This is (basically)    freq(colloc)/(freq(word1) + freq(word2)) (We then do some scaling so the numbers "look nice") We should really list it as "Logdice" rather than "salience" - salience is what you want to measure, but it's presumptuous to imply we have found a measure that succeeds in measuring it.  Stats aren't properly documented yet but will be soon. ]]

14. Creating SkE for Japanese
Load JapWaC into SkE
Write gram relations for Japanese
Chasen POS (as used for jaSlo)
Compile word sketches
Recompute scores in WS
Compile thesaurus

22. Word Sketch examples WS for 女の子 (noun） WS for 冷たい (adjective)
WS for 書く (verb)

24. Thesaurus, WS Diff example(1)
WS Diff for 女の子 and 男の子

25. Thesaurus, WS Diff example (2)
WS Diff for 寒い and 冷たい

26. 「温泉」 example
WS for 温泉

27. Future work More metadata in the corpus: More data cleaning
date, title, author; text typology
More data cleaning
Japanese corpus for HLT research:
sampling only 10 consecutive sentences, 100M
would be available for download with Creative Commons license
For native speakers’ and learners’ use:
original Chasen tags, Chasen kana
Ruby romaji, furigana in examples
Connecting to jaSlo, Natsume system
More advanced relations (MWU etc.), Cabocha?
Load other corpora into SkE (Kotonoha, AB)
avoid copyright problems by sampling only 10 consecutive sentences from many URLs to obtain 100M corpus – make available for download with Creative Commons licence

28. Access to JapWaC & SkE http://www.sketchengine.co.uk Free 30-day trial
Self-registration
Japanese, Chinese, English, French, German, Italian, Spanish, Portuguese, Slovene
Also gives access to WebBootCaT
“instant web corpora”

29. Thank you for your attention!