Determining how well an individual paper satisfies Poppers criteria Popper said good science involves: A substantial theory being put up to test Safe background knowledge to crack it against A test that is severe

Can you isolate a substantial theory that is being put up to test? Note difference between substantial theory and statistical hypothesis I am going to see if any of these variables correlate with each other involves no substantial theory but a number of statistical hypotheses

Can you isolate a substantial theory that is being put up to test? Note difference between substantial theory and statistical hypothesis I am going to see if any of these variables correlate with each other involves no substantial theory but a number of statistical hypotheses coffee improves concentration only because people believe that it does (substantial theory) people will spot more spelling errors out of 30 within half an hour of drinking one rather than zero cups of decaf coffee (statistical hypothesis)

To implement criterion of Falsifiability Have to say *what* (theory) is falsifiable

Safe background knowledge A substantial theory makes predictions by using auxiliary hypotheses (background knowledge). Test substantial theory (caffeine is placebo) by comparing no versus one cup of coffee and of decaf. What auxiliary hypotheses are being used in this case? (Must check all background knowledge is safe!)

Decaf tastes like coffee and will make people expect the same consequences, e.g. increased concentration Is it safe? If we fail to confirm predictions could we just as plausibly reject this auxiliary as reject the substantial theory? If so, the test is not a good one.

Decaf tastes like coffee and will make people expect the same consequences, e.g. increased concentration Is it safe? If we fail to confirm predictions could we just as plausibly reject this auxiliary as reject the substantial theory? If so, the test is not a good one. Need to critically evaluate auxiliaries – e.g. take expectancy ratings, use other ways of disguising the taste of caffeine in caffeine and placebo drinks. List auxiliary hypotheses and check you are happy to treat them as safe.

Compare a caffeine and no caffeine drink and see if people can taste the difference when given a sip.

Can always open up results to further critical scrutiny – maybe the sips tasted the same but caffeine has side effects when taken as a whole drink (e.g. face flushes). Side effects may change expectancies. But comes a time when auxiliary analyses have survived sufficient criticism you decide to treat background knowledge as safe.

To implement criterion of Falsifiability Need to be accept some things as given in order to falsify something else

Test severity Popper: Test is severe if predicted outcome is likely given the theory and unlikely given the rest of background knowledge

Substantial theory predicts one cup versus no cup of coffee will improve performance because of the unconscious power of expectations Test is only severe if prediction is unlikely given background knowledge Does any established background knowledge make the same prediction?

It may be likely on another theory : that if subjects can guess the experimenter's hypothesis and "help" the experimenter get a "good" result they will do deliberately so (the problem of "demand characteristics"). This problem would need to be dealt with for the test to be severe. (For example, maybe drinking the coffee is incidental, not apparently part of the experimental procedure.) NOTE: If prediction follows from other established theories then theory is not really falsifiable in this context: We expect same results whether or not theory is true

Severe: Test is severe if predicted outcome is likely given the theory and unlikely given the rest of background knowledge SO If theory predicts a difference: Significant difference should be likely given theory (= high POWER) If theory predicts no difference: Significant difference should be likely if theory false (high POWER) (= prediction unlikely given rest of background knowledge)

In summary: The test is good if: An explicit substantial theory puts its neck out by Having safe background knowledge in constructing the test Being subjected to a severe test in the sense that the predicted outcome is likely given the theory and unlikely given the rest of background knowledge