Six Guiding Principles Sally Fincher Building Research in Australasian Computing Education: Second Workshop 26 th -29 th January 2005: Sydney

2 Remember these …? Pose significant questions that can be answered empirically Link research to relevant theory Use methods that permit direct investigation of the question Provide a coherent and explicit chain of reasoning Replicate and generalize across studies Disclose research to encourage professional scrutiny and critique

3 For this workshop Last year, we concentrated on the first three. For this workshop, we’ll be mostly concerned with the last three: Provide a coherent and explicit chain of reasoning Replicate and generalize across studies Disclose research to encourage professional scrutiny and critique

4 Provide a coherent and explicit chain of reasoning Provide a coherent and explicit chain of reasoning? Sure, yes, of course but what does it mean? And how do we do it?

5 Provide a coherent and explicit chain of reasoning: what does it mean? It doesn’t mean that this sort of approach is inexplicable, or that no-one else does it. In fact, we all have a good idea both of what “coherent” and “explicit” mean. Let’s take an example, to explore this further …

6 Street navigation We all know what we need to have (and to have explained) to navigate through city streets. The coherent and explicit chain of reasoning we need depends on our context For example, in London, streets are named, and signs tend to have a consistent “look and feel” and to be placed about 3 feet off the ground

7 In the US, streets are also identified (although not always with names) and, as a bonus, most towns are laid out on a grid. However the signs tend to be placed above eye level – on posts, or in the case of large streets and highways, suspended over the road itself The navigational chain of reasoning remains clear, but we have to be explicit. Street signs

8

9 Each village, town, city or ward is divided into "machi," the size of large neighbourhoods; each machi is divided into "chome," which usually consist of several dozen blocks. Each "chome" has numbered blocks, and each block has numbered houses. Since only important avenues have names, a house may not be identified by which street it is on, rather by which block, chome, and machi it is in

10 Street signs Although there are no street signs in Japan, there are usually placards on every other telephone pole that have the block address on them. For example, one sign might read "Suginami-ku, Asagaya-Minami 3-14." That means the telephone pole you are looking at is located on the 14th block in the 3rd section of the neighborhood called Asagaya-Minami in Suginami ward. If you have a map book of Tokyo, you can look up the page for Suginami, find Asagaya-Minami, find the 3rd section within it, and the block within that with the number 14 on it. To see which side of the block you are on, look at a telephone pole across the street; if it says "Suginami-ku, Asagaya-Minami 3-22," then you know you are between blocks 14 and 22

11 Coherent and explicit So, Japanese street navigation is totally different from UK or US street navigation – except in the principle of navigation and in the fact that we all understand what has to be accomplished and that there is a systematised way to go about it. That is why we have to be careful to explain all the steps in our reasoning – and what they mean to us – even if they seem “obvious” If we don’t, then it’s all too easy to pervert the reader’s view …

14 Replicate and generalize across studies Well, yes, of course … … but what does it mean? And how do we do it? It’s a twin problem of “input” and “output” Input: what previous work we have based our work on and what we have used to inform our approach Output: giving sufficient information, in the right way, for others to be able to replicate or generalize from our work.

15 Input In BRACE, what have we replicated & generalized? What is important, for instance, in these papers: M. McCracken, V. Almstrum, D. Diaz, M. Guzdial, D. Hagan, Y.B.-D. Kolikant, C. Laxer, L. Thomas, I. Utting, and T. Wilusz. (2001) A multinational, multi-institutional study of assessment of programming skills of first-year CS students. Proceedings of ITiCSE. Nathan Rountree, Janet Rountree and Anthony Robins Predictors of Success and Failure in a CS1 Course (2002) SIGCSE Bulletin vol. 34, no. 4. ETS Kit of Factor-Referenced Cognitive Tests, Educational Testing Service, Princeton, NJ What is important in other papers?

Output: Replication in Chemistry: Angwandte Chemie

17 Angwandte Chemie [a typical paper] … is about three pages long. Almost one page contains experimental detail. Half a page is endnotes. The body of the article is then about one and a half printed pages, of which roughly a third consists of graphics The remarkable density of graphical material, most of it quasi-iconic representations of microscopic molecular structure.

18 Angwandte Chemie “The authors speak, as chemists today do, of molecules that they do not see, but for which they have excellent indirect evidence … I have written of this incredible process, and the way that the chemists’ necessity to move simultaneously in macroscopic and microscopic worlds forces chemists to use a mixture of symbolic and iconic representation of compounds/molecules” Roald Hoffman (2002) Writing (and Drawing) Chemistry in Jonathan Monroe (ed) Writing and Revising the Disciplines, Cornell University

Note especially … A particular feature of Angwandte Chemie is the mandatory inclusion at the end of any experimental paper is an “Experimental Section”, detailing procedures for the experiments carried out

20 Experimental Section “This is a general statement that, in effect, states that anyone, anytime, anywhere who treats the same ingredients in the same way as I did, will make the same chemical compound” And yet, it is based on a single emprical study at one specific time and in one specific place Within the positivist paradigm of physical science this was, and is, acceptable

Contrast with this …

22 Referential footnotes & truth-claims “The note is the correlate of research … The research paper is written replete with referential notes, ideally, in a restricted research paradigm, a note per statement in the principal text. (More subjective moments are confined to a preface or coda or perhaps to notes not serving as references.) … “

23 Referential footnotes & truth-claims “ … to be professional historiography must have notes that provide references for statements that function referentially and make truth-claims.” Dominick LaCapra (2002) Writing History, Writing Trauma in Jonathan Monroe (ed) Writing and Revising the Disciplines, Cornell University

24 For Computer Science Education We are neither Chemists nor Historians, so we don’t have paradigmatic norms for giving replication and generalization information. For BRACE We have two particular issues (common to work of the same nature) And a particular problem (possible unique)

25 Issue 1: “A double crisis of representation and legitimation” Some of what we are reporting is qualitative data.That means we have to accurately present what we know. In doing so, we will make (often implicit) claims for its status as knowledge and its veracity By representing what someone says … or interpreting what a mapping exercise shows … we are embedding what we believe it to represent by its very presentation So, we have to include enough information on our representation of that knowledge to allow someone else to judge it’s legitimacy There are different legitimacies for different types of data and different claims (even within BRACE) Denzin & Lincoln, Handbook of Qualitative Research

26 Issue 1: “fair dealing” It is important to estimate error, and the possibility of unseen error (resting on research design & conduct) It is important to conduct the practice of fair-dealing ‘fair dealing’-wariness of presenting the perspective of one group as if it defined a single truth about the phenomenon, while paying scant attention to other perspectives” Murphy, E. et al, (1988) Qualitative research methods in health technology assessment: a review of the literature Health Technology Assessment 2 (16)

27 Issue 2: How general are generalizations? How can we generalize from any empirical study? Scientific generalization A positivistic, Popperian view: if x happens in y circumstances then z will occur in all cases (recall the “experimental section”, above) Probabilistic generalization Reliant on sample size and statistical power: if x happens in y circumstances then z will occur in about p% of cases (recall we characterised these as “empirical laws” last year) Fuzzy generalization A class of statements which are imprecisely probable: if x happens in y circumstances then z may occur Following the work of Michael Bassey – see references at end

28 Issue 2: Fuzzy generalizations The fuzzy principle state that everything is a matter of degree … Fuzziness has a formal name in science: multivalence. The opposite of fuzziness is bivalence or two-valuedness, two ways to answer each question, true or false, 1 or 0. Fuzziness means multivalence. It means three or more options, perhaps an infinite spectrum of options, instead of just two extremes. It means analogue instead of binary, infinite shades of grey between black and white. It means all that the trial lawyer or judge tries to rule out when she says, “Answer just yes or no”. Bart Kosko (1994 ) Fuzzy Thinking, p.18

29 Issue 2: fuzzy generalisation, the necessary corollary: BET If x happens in y circumstances then z may occur Well, yes, but then equally: If x happens in y circumstances then z may not occur Something else is needed. Bassey suggests Best Estimate of Trustworthiness (BET) which is an additional statement, an “educated guess” based on the experience, insight and professional judgement of the researcher(s) Expressed as a range of likelihood (1-10%, perhaps)

30 Problem: generalization You can think of the BRACE project as a single piece of work collaboratively conducted by 15 researchers You can think of the BRACE project as an aggregation of 15 replicated studies What issues of replication and generalization change with these two views? What part does “institutional context” play? What might the fact that some of the raw data is from a UK institution mean? Is it important that different institutions have contributed different quantities of data?

Conditioned generalisation Domain in which the assertion is true or at least a very good approximation Domain where assertions may be true or is partly true To infinity Domains where assertion is wholly or nearly untrue Stephen Jay Kline (1995) Conceptual Foundations for Multi-Disciplinary Thinking Stanford University Press

32 Disclose research to encourage professional scrutiny and critique Yes, indeed. Warts and all, we’ve got a paper to write. Ladies and gentlemen, start your engines.

33 Michael Bassey references Bassey, M (1999) Case Study Research in Educational Settings Open University Press Bassey, M (2001) A Solution to the Problem of Generalisation in Educational Research: fuzzy prediction Oxford Review of Education 27 (1) pp 5-22 Bassey, M (2003) How general are generalizations? In Joanna Swann and John Pratt, eds Educational Research in Practice Continuum Books, London pp