1. The development and analysis of a corpus of student writing
AILA, Madison, Wisconsin, July 2005
Hilary Nesi
BAWE Project
CELTE,
University of Warwick
BAWE/ Madison, Wisconsin / July 2005

2. A corpus of British Academic Written English
3000+ examples of proficient student writing
Different genres
Different disciplines
Different levels
BAWE/ Madison, Wisconsin / July 2005

3. BAWE/ Madison, Wisconsin / July 2005
Research questions
What are the characteristics of proficient university student writing?
How can this writing be categorised in terms of genres?
What are the similarities and differences between genres produced in different disciplines?
What are the similarities and differences within and between genres produced at different stages of university study?
BAWE/ Madison, Wisconsin / July 2005

4. Four stages of corpus development:
categorizing
contextualizing
collecting
coding
BAWE/ Madison, Wisconsin / July 2005

5. Topic 1: Identifying categories of texts
We are interested in:
the similarities and differences between genres of assessed writing produced in different disciplines
the similarities and differences between genres of assessed writing produced at different stages of university study
BAWE/ Madison, Wisconsin / July 2005

6. BAWE/ Madison, Wisconsin / July 2005
The 4-by-4 matrix
Four broad disciplinary groupings (life sciences, physical sciences, social sciences, humanities)
Four years of study (undergraduate and taught postgraduate)
BAWE/ Madison, Wisconsin / July 2005

7. BAWE/ Madison, Wisconsin / July 2005
The departmental grid
university
discipline
Brookes
Reading
Warwick AH
English
History of Art
Archaeology
Classics
History
Theatre Studies LS
Health & Social
Care
Agriculture
Food Science
Zoology
Biological Sciences
Medicine PS
Computing
Planning
Meteorology
Cybernetics
Engineering
Physics SS
Anthropology
Hospitality
Tourism
Business
Law
Sociology
BAWE/ Madison, Wisconsin / July 2005

8. BAWE/ Madison, Wisconsin / July 2005
The sampling grid (= 3072):
discipline
year AH SS PS LS
1st
64 X 3
2nd
3rd
4th
BAWE/ Madison, Wisconsin / July 2005

9. BAWE/ Madison, Wisconsin / July 2005
Year of study?
Categories: 1, 2, 3, 3*, 4, 4*
3 = 3rd year, no intercalated year
3*= 3rd year, intercalated year
4* = 4th year undergraduate
4 = "normal" postgraduate, e.g. Masters
The categories we will use are 1, 2, 3, 3*, 4, 4*
Years 1 and 2 are relatively straightforward and standard across different degree courses
We will then make a distinction in the contextual information available between third year students who have had an intercalated year and those who haven’t. Yet they will all be classed as third year in terms of the collection of assignments
Examples of four year undergrad courses at Warwick include an MEng in Engineering or MMath in Mathematics
Although students in their fourth year may still be classed as undergrads, they would progress straight to doctoral study in the same way as postgrads, so are at a comparable level of study
This information is captured on the submission sheet we collect from students, along with information about the students’ home department as it is possible that students take modules outside their course or a module that is open to second or third year students, for example
BAWE/ Madison, Wisconsin / July 2005

10. Topic 2: Identifying contexts for the texts
“the social context (place, time, participants) is arguably .. as significant as .. intrinsic linguistic features”
Burnard 2004:8
BAWE/ Madison, Wisconsin / July 2005

11. Perspectives from the discourse community
a. Department documentation
b. Tutor interviews & surveys
c. Student submission forms
BAWE/ Madison, Wisconsin / July 2005

12. Sociology Assignment Types (from interview data)
Essays
Book reviews
Book reports
Projects
Urban Ethnography Assignments
Fieldwork Reports
Dissertations
BAWE/ Madison, Wisconsin / July 2005

13. Tradition vs. innovation
“We’re quite a traditional department in that we still use mainly essays, we’re very conscious that we would like to, and perhaps need to, do something about that”
Mike Neary, Director of Undergraduate Studies, Sociology, Warwick, 26/04/2005
In fact, the change towards more innovative ways of assessing students seems to be a key issue that we are constantly reminded of in our academic interviewing, and the quote here from Mike Neary illustrates that challenging tradition is something needed in the future
It also seems to be something students are starting to expect, and the fact that other departments assess websites and on-line journals is another example of this
BAWE/ Madison, Wisconsin / July 2005

14. Language and educational background
Native vs. non-native English speakers
Submission form data:
1. first language
2. secondary education
Although we may be interested in whether the students are native or non-native speakers, the corpus is not designed to include a balance of non-native speakers – we are primarily interested in writing in British higher education of which non-native speakers play a crucial role
However, one of our contextual categories is the student’s first language, coupled with a history of their secondary education and whether they have been schooled in the UK. This information does not influence corpus design, but allows specific research questions to be answered later
BAWE/ Madison, Wisconsin / July 2005

15. BAWE/ Madison, Wisconsin / July 2005
Is it in or is it out?
Formative vs. summative
Foreign languages
Posters and PowerPoint
Creative writing / Fiction
Through the process of interviewing and collecting data we are increasingly refining our categories
We are collecting proficient written work and have pre-defined this as 65% or Iii and above
A key distinction is made at Warwick between formative, which does not count towards final module grade, and summative assessment, which determines module grade
It was imagined we would be collecting summative assessment only. However our interviews have shown formative assessment is viewed as equally important in terms of student development and may include genres not found in summative work
Therefore we will accept any assignment that has been given grade above our predefined level
Other categories that have influenced inclusion have been assignments written mainly in a foreign language, as the corpus is designed to investigate academic English, however assignments that include foreign language terminology or quotes will be included
Another issue we have discussed is that of posters and PowerPoint presentations, whilst they both have a strong written component this is not assessed in isolation, but either with visual or spoken elements, and so will not be included in the corpus
Creative writing is a grey area, it was initially excluded as it was seen as written for literary purposes rather than pure academic assessment
However, this was challenged by the undergraduate director of studies, Mike Neary, who has introduced an assessment called “Crime fiction” to his “Crime and Deviance” module. This is a crime story assessed by academic criteria, which illustrates how assessment is constantly changing
BAWE/ Madison, Wisconsin / July 2005

16. Contexual Info as Metadata (1)
Student (unique code)
Sex M,F
First Language Eng, Ger, Fre …
UK Education UKA, OSA, UK1, UK2, ..
Course BA English Studies, BSc Maths
And so we have created a list of the contextual information that we plan to include in the metadata that will go into each document header, enabling the assignments to be categorised and analysed
This is information about the student; Go to part 2 (about the assignment)
BAWE/ Madison, Wisconsin / July 2005

17. Contextual Info as Metadata (2)
Assignment (unique code)
Title (text)
Date written (year-month) , , ..
Level (of student) 1,2,3,3*,4, 4*
Module title Critical Theory, …
Module code EN3001, …
Assignment Type Essay, Report, …
JACS Code A, E, D …
Disciplinary Grouping AH, LS, PS, SS
Grade M(erit), D(istinction)
Authors 1,2,3,…
This information will be collected about each assignment in the corpus and from each student, and Paul will now talk more about the collection process
BAWE/ Madison, Wisconsin / July 2005

18. Topic 3: Collecting the texts
checking
Students texts to project
Presentation in core lectures
Reply – accept texts, specify collection point
F2F Collection:
St. forms checked. Money paid.
Cohort s
Investigate: modules (course/department documentation)
establish personal contacts (interview)
Text and data handling
[Facilitate collection of texts to meet sampling policies, timeframe and budget.]
BAWE/ Madison, Wisconsin / July 2005

19. Issues arising from the pilot study
Establish contacts + promote project
Include a monetary incentive
Make collection process user friendly
What (what info,)
Who (approach admin/academic staff/ students directly)
How - mode, - flyers/website/posters/cohort s/ presentations/
BAWE/ Madison, Wisconsin / July 2005

20. BAWE/ Madison, Wisconsin / July 2005
The paper trail
Generate unique identifier for each student/text
Monitoring the submissions – expected and collected
The submission form – student data and ‘disclaimer’ (IP)
Transferring the student data – maintaining links
BAWE/ Madison, Wisconsin / July 2005

21. Topic 4: Coding the texts
Anthropology vs. English Studies assignment
<front> <titlePage> <docTitle> <titlePart type="main">Case Study of the white-throated capuchin monkey (<hi rend="italic">Cebus capucinus</hi>)</titlePart> <titlePart>xxx</titlePart> </docTitle> <figure id="BAWE_3016a-pic1"/> </titlePage> </front> <body>
<front> <docTitle> <titlePart type="main" rend="underline">Discuss the handling of the discourses of religion and the effects of religious and ethical change in the Victorian period</titlePart> </docTitle> </front> <body>
BAWE/ Madison, Wisconsin / July 2005

22. What goes into <front> vs. <body>?
Example of two first pages:
BAWE/ Madison, Wisconsin / July 2005

23. What to do about formulae?
equations (and all kinds of variations of =)
chemical formulae
arithmetic expressions
logical expressions
expressions following some other discipline-specific formalism (e.g. computer code, phonetic transcription etc.)
a part ("term") of any of these (if non-NL)
BAWE/ Madison, Wisconsin / July 2005

24. Insert empty <formula> tag
anything that has been inserted with the MS formula editor (appears as a "field");
any complex formal expression, i.e. that cannot be represented as a simple sequence of characters (e.g. fraction, square root)
I(∆s) = Q
any formal expression separated typographically from running text (new paragraph)
BAWE/ Madison, Wisconsin / July 2005

25. BAWE/ Madison, Wisconsin / July 2005
Example
... The slope of the yield curve can be analysed by looking at the spread between the long-term and the one-period, short-term interest rate, denoted as Snt = Rnt – rt. If we manipulate equation 1, the yield spread, Snt, can be written as the expectation of a weighted average of future changes in short-term interest rates as follows:
Snt = Et Snt *
Snt * = (1/n) [(n-1)Δrt+1 + (n-2)Δrt+2 + …+ Δrt+(n-1)] [2]
<p><s>...</s> <s>The slope of the yield curve can be analysed by looking at the spread between the long-term and the one-period, short-term interest rate, denoted as S<hi rend="italic"><hi rend="sup">n</hi><hi rend="sub">t</hi></hi> = R<hi rend="italic"><hi rend="sup">n</hi><hi rend="sub">t</hi></hi> – r<hi rend="italic"><hi rend="sub">t</hi></hi>.</s> <s>If we manipulate equation 1, the yield spread, S<hi rend="italic"><hi rend="sup">n</hi><hi rend="sub">t</hi></hi>, can be written as the expectation of a weighted average of future changes in short-term interest rates as follows:</s></p>
<p><formula notation="" id="EC0001-form2"/></p>
BAWE/ Madison, Wisconsin / July 2005

26. BAWE/ Madison, Wisconsin / July 2005
Principles of mark-up
Keep the structure of the document as close to the original as possible
Mark up elements relevant to our research
Be cost effective
BAWE/ Madison, Wisconsin / July 2005

27. The final stage: Corpus analysis
Multidimensional analysis will tell us about the language of texts in our 4 x 4 grid. Broadly, a description of register.
SFL Genre analysis will tell us about the generic stages of texts. Broadly, a description of text organization and purpose, with reference to contextual and linguistic features.
BAWE/ Madison, Wisconsin / July 2005

28. BAWE/ Madison, Wisconsin / July 2005
Goodbye Folks!
Thanks for your attention 
Thanks also to:
Signe Ebeling, Richard Forsyth, Sheena Gardner, Alois Heuboeck, Dawn Hindle, Maria Leedham, Paul Thompson, Paul Wickens.
BAWE/ Madison, Wisconsin / July 2005