Representatıvness, balance and samplıng ın a corpus Lınguistıcs

Introduction A corpus is designed to respresent a particular language or language variety. It is impossible to analyse every extant utterance or sentence of a given language. Sampling is unavodiable. How can you be sure that the sample you are studying is representative of the language or language variety under consideration?

One must consider Balance Sampling to ensure representativness

Representativeness in Corpus Linguistics Biber (1993:242) : “Representativness refers to the exent to which a sample includes the full range of variability in a population.” Population: a sample of a language or a language variety in a corpus.

2 factors to determine the representativeness of a corpus The range of genres included in a corpus (i.e. balance). How text chunks for each genre are selected (i.e sampling).

Criteria used to select texts for a corpus External : situational; genre / register Internal: linguistic / text types

Internal criteria is problematic and circular Internal criteria like the distribution of words or grammatical features cannot be the primary parameters for the selection of corpus data. The corpus has been skewed by design.

Sinclair (1995): The texts or parts of texts to be included in a corpus should be selected according to external criteria so that their linguistic characteristics are independent of the selection process.

Another aspect of representativeness: Change over time View of a corpus as a Static (It applies to a sample corpus) Dynamic (It applies to a monitor corpus) language model

Static view of language Helsinki Diachronic Corpus Lancester — Oslo / Bergen Corpus (LOB) Freiburg — LOB (FLOB) Sampling frame LOB : British English, early 1960s FLOB : British English, early1990s

Representativeness of General and Specialized Corpora General Corpora (e.g., BNC) serve as a basis for an overall description of a language or language variety. It should involve samples from a broad range of genres. Specialized Corpora tend to be domain (e.g., medicine or law) and genre (fiction, newspaper texts, academic prose) specific.

Balance The range of text categories included in the corpus determines how balanced the corpus is. The acceptable balance of corpus is determined by its intended uses. A balanced corpus usually covers a wide range of text categories which are supposed to be representative of a language or language variety under consideration. These text categories are typically sampled proportionally for inclusion in a corpus.

Balance There is no scientific measure of corpus balance. A more typical approach to corpus balance is that corpus-builders adopt an existing corpus model when building their own corpus, assuming that balanced will be achieved from the adopted model. BNC is generally accepted as being a balanced corpus. American National Corpus Korean National Corpus Polish National Corpus Russian Reference Corpus

Balance Balance is a more importan issue for a static sample corpus than a dynamic monitor corpus. The builders of monitor corpora finds size of corpora more important than the balance of it. They assume that corpus will balance itself when it reaches a substantial size.

Domain % Date % Medium % Imaginative Book Arts Periodical31.08 Belief and thought Unclassified Misc. published Commerce /finance Misc. published Leisure To-be spoken Natural / pure science Unclassifed Applied science Social science World affairs Unclassified1. 93 Composition of written BNC

Region % Interaction type % Context - goverened % South Monologue Educational /informative Midlands Dialogue Bussiness North Unclassified Institutional Unclassifed Leisure Unclassifed Composition of spoken BNC

Sampling To obtain a representative sample from a population: define the sampling unit and boundaries of population Written texts: sampling units are books, periodicals, or newspaper. List of sampling units is referred to as sampling frame.

LOB corpus Target population: All written English texts published in the United Kingdom in 1961 Sampling frame: British National Bibliograhpy Cumulated Subjet Index , for books Willing’s Press Guide 1961, for periodicals

Sample size Full texts (i.e., whole texts) Text chunks (ideal: sample text fragments, 2,000 running words)

A population can be defined in terms of Language production (demographically oriented) Language reception (demographically oriented) Language as a product (text category / genre of language)

Different sampling techniques Simple random sampling: all sampling units within the sampling frame are numbered and the sample is chosen by the use of table of random numbers Stratified random sampling, first divides the whole population into relatively homogeneous groups (strata) and samples each stratum at random. Demographic sampling (i.e., categorize sampling units on the basis of speaker/writer age,sex and social class) is also a type of stratified sampling.

Stratified Ramdom Sampling: Brown and LOB Corpora The target population for each corpus was first grouped into 15 text categories such as news reportage, academic prose, different types of fiction. Samples then drawn from each text category.

Conclusion In constructing a balanced, representative corpus, Stratified random sampling is to be preferred. For written texts, a text typology established on the basis of external critearia is relevant. For spoken data, demographic sampling is appropriate. It must be complemented by context- goverend sampling so that some context goverend linguistic variations can be included in the resulting corpus.