1. 31 March 2009 @ Crawford School 1 Measurement – 1 Semester 1, 2009 POGO8096/8196: Research Methods Crawford School of Economics and Government

2. 231 March 2009 @ Crawford School This week Levels of research and unit of analysis Levels of research and unit of analysis Problems of measurement: reliability and validity Problems of measurement: reliability and validity The basic problem of measurement The basic problem of measurement Reliability and validity Reliability and validity Problems of measurement: precision Problems of measurement: precision Precision in measurement Precision in measurement Precision in measures Precision in measures Quantitative vs. qualitative measures Quantitative vs. qualitative measures

3. 331 March 2009 @ Crawford School Levels of research Our ultimate goal is to improve theories in social sciences. The more theories you have, the better your understandings of societies, economies, and politics. Our ultimate goal is to improve theories in social sciences. The more theories you have, the better your understandings of societies, economies, and politics. But we normally test a specific and testable hypothesis derived from a theory. But we normally test a specific and testable hypothesis derived from a theory. A research question is usually formulated at the level of hypothesis, but the question should be relevant to a broader theory. A research question is usually formulated at the level of hypothesis, but the question should be relevant to a broader theory.

4. 431 March 2009 @ Crawford School Example 1 Theory: Economic development results in political development. Theory: Economic development results in political development. Hypothesis: The more industrialized a nation, the greater the level of mass political participation. Hypothesis: The more industrialized a nation, the greater the level of mass political participation. Operational (or “Working hypothesis”): The higher the % of the labor force engaged in manufacturing, according to the United Nations Yearbook, the higher the % of the population of voting age that participated in the most recent national election, according to the Statesman’s Yearbook. Operational (or “Working hypothesis”): The higher the % of the labor force engaged in manufacturing, according to the United Nations Yearbook, the higher the % of the population of voting age that participated in the most recent national election, according to the Statesman’s Yearbook.

5. 531 March 2009 @ Crawford School Example 2 Theory: Socioeconomic status affects political participation. Theory: Socioeconomic status affects political participation. Hypothesis: The higher a person’s income, the more likely he or she is to vote. Hypothesis: The higher a person’s income, the more likely he or she is to vote. Operational (or “Working hypothesis”): The higher a survey respondent’s answer when he or she is asked, “What is your household’s annual income?”, the more likely that person will answer “Yes” when asked, “Did you vote in the last General Election?” Operational (or “Working hypothesis”): The higher a survey respondent’s answer when he or she is asked, “What is your household’s annual income?”, the more likely that person will answer “Yes” when asked, “Did you vote in the last General Election?”

6. 631 March 2009 @ Crawford School Unit of analysis – 1 In deciding what to measure for a concept of your interest, it is very important to understand what is the unit of analysis in your research. In deciding what to measure for a concept of your interest, it is very important to understand what is the unit of analysis in your research. The unit of analysis is the object that the “working hypothesis” describes. It can be a person (in a survey/interview study), a country (in a cross-national comparison), an organization (in a comparative case study of organizational behavior), a year (in a longitudinal analysis of a given country), a policy (in a policy study), or something else. The unit of analysis is the object that the “working hypothesis” describes. It can be a person (in a survey/interview study), a country (in a cross-national comparison), an organization (in a comparative case study of organizational behavior), a year (in a longitudinal analysis of a given country), a policy (in a policy study), or something else.

7. 731 March 2009 @ Crawford School Unit of analysis – 2 The unit of analysis often determines how a variable in a hypothesis is “operationalized” (i.e., measured). The unit of analysis often determines how a variable in a hypothesis is “operationalized” (i.e., measured). All variables in a working hypothesis must have the same unit of analysis. All variables in a working hypothesis must have the same unit of analysis. For a given hypothesis, there are many ways to operationalize depending on how you define the unit of analysis. For example, “The higher the income level, the more likely voters support the Republican party in the US.” How can you operationalize this hypothesis? For a given hypothesis, there are many ways to operationalize depending on how you define the unit of analysis. For example, “The higher the income level, the more likely voters support the Republican party in the US.” How can you operationalize this hypothesis?

8. 831 March 2009 @ Crawford School Answer 1: Individual-level The unit of analysis: Individuals The unit of analysis: Individuals A measure of income: Ask a survey respondent, “What is your household’s annual income?” A measure of income: Ask a survey respondent, “What is your household’s annual income?” A measure of partisanship: Ask the survey respondent, “Did you vote for a Republication candidate in the last presidential election?” A measure of partisanship: Ask the survey respondent, “Did you vote for a Republication candidate in the last presidential election?”

9. 931 March 2009 @ Crawford School Answer 2: State-level The unit of analysis: 50 states in the US The unit of analysis: 50 states in the US A measure of income: The average household income in 2000, according the Statistical Abstract of the United States. A measure of income: The average household income in 2000, according the Statistical Abstract of the United States. A measure of partisanship: The percentage of votes for a Republican candidate received in the last presidential election, according to America Votes. A measure of partisanship: The percentage of votes for a Republican candidate received in the last presidential election, according to America Votes.

10. 1031 March 2009 @ Crawford School What is a concept? What is a concept? (from American Heritage) What is a concept? (from American Heritage) 1. A general idea derived or inferred from specific instances or occurrences. 2. Something formed in the mind; a thought or notion. Concept vs. measure in empirical research Concept vs. measure in empirical research A concept = an un-observable (dependent or independent) variable in a theory or a hypothesis. A concept = an un-observable (dependent or independent) variable in a theory or a hypothesis. A measure = an observable (dependent or independent) variable in a “working hypothesis.” A measure = an observable (dependent or independent) variable in a “working hypothesis.”

11. 1131 March 2009 @ Crawford School Concept vs. measures concept (unobservable) measures(observable) economic growth e.g., a change in the real GDP per capita (purchasing power parity) from 1991 to 2000 (“interval”) corruption e.g., Transparency International (TI) corruption index (“ordinal”) government type e.g., presidential, parliamentary or authoritarian systems (“nominal”)

12. 1231 March 2009 @ Crawford School These are all concepts! Industrialization Industrialization Income inequality Income inequality Poverty Poverty Technological progress Technological progress Profitability Profitability Efficiency Efficiency Accountability Accountability Effectiveness Effectiveness Government size Globalization Democratization Civil society Clean air Basic water needs Sustainability Restoration

13. 1331 March 2009 @ Crawford School The basic problem In any empirical research, we measure things that should correspond to what we want to measure. But any observable variable we would choose to measure a certain unobservable concept corresponds only indirectly to that concept. This is the “basic problem of measurement” in social sciences. In any empirical research, we measure things that should correspond to what we want to measure. But any observable variable we would choose to measure a certain unobservable concept corresponds only indirectly to that concept. This is the “basic problem of measurement” in social sciences. The “basic problem” may result in critical error when making causal inference. (See the next slide for an example.) The “basic problem” may result in critical error when making causal inference. (See the next slide for an example.)

14. 1431 March 2009 @ Crawford School This is the source of a problem. A consequence

15. 1531 March 2009 @ Crawford School What to do? We must ensure, as much as possible, that the relationships between concepts and measures are such that the relationship between the measures mirrors the relationship between the concepts. For this purpose, we must try to use “good” measures. We must ensure, as much as possible, that the relationships between concepts and measures are such that the relationship between the measures mirrors the relationship between the concepts. For this purpose, we must try to use “good” measures. A measure is reliable to the extent that it gives the same result again if the measurement is repeated. (Note: In reality, the measurement is almost always not repeated.) A measure is reliable to the extent that it gives the same result again if the measurement is repeated. (Note: In reality, the measurement is almost always not repeated.) A measure is valid if it actually measures what it purports to measure. A measure is valid if it actually measures what it purports to measure.

16. 1631 March 2009 @ Crawford School Random vs. non-random error Reliability is a necessary but not sufficient condition of validity, because a measure is valid if it is free of both random and non-random errors, while a measure is reliable if it is free of random error alone. Reliability is a necessary but not sufficient condition of validity, because a measure is valid if it is free of both random and non-random errors, while a measure is reliable if it is free of random error alone. If, on average, a given measure of a concept tends to be true, we can assume that any error in the measure is random. If, on average, a given measure of a concept tends to be true, we can assume that any error in the measure is random. By contrast, non-random error is systematic error that tends to, on average, distort a given measure of a concept. By contrast, non-random error is systematic error that tends to, on average, distort a given measure of a concept.

17. 1731 March 2009 @ Crawford School Reliable Valid

18. 1831 March 2009 @ Crawford School Improving reliability Common sources of unreliability Common sources of unreliability clerical errors in official statistics clerical errors in official statistics ambiguous questions in surveys ambiguous questions in surveys dishonest interviewers and interviewees dishonest interviewers and interviewees Careful work (e.g., careful check of data sources, careful check of survey questions, etc.) is the best way to achieve reasonable reliability. Careful work (e.g., careful check of data sources, careful check of survey questions, etc.) is the best way to achieve reasonable reliability. There are several ways (e.g., test-retest check, split-half check) to test the reliability of a measure (even without repeating the measurement). There are several ways (e.g., test-retest check, split-half check) to test the reliability of a measure (even without repeating the measurement).

19. 1931 March 2009 @ Crawford School Improving validity Since all we can observe is always measures, assessing the validity of the measures is quite difficult, if not impossible. It is also difficult to identify common sources of invalidity. Since all we can observe is always measures, assessing the validity of the measures is quite difficult, if not impossible. It is also difficult to identify common sources of invalidity. To achieve reasonable validity, you should conduct a “face validity test” – Does a measure look right to you? To achieve reasonable validity, you should conduct a “face validity test” – Does a measure look right to you? There are several ways (e.g., random-sampling, pilot study) to test/improve the validity of a measure (but not always useful). There are several ways (e.g., random-sampling, pilot study) to test/improve the validity of a measure (but not always useful).

20. 2031 March 2009 @ Crawford School Discussion Suppose that you are interested in knowing the determinants of the accountability of a government. Suppose that you are interested in knowing the determinants of the accountability of a government. How do you measure the level of accountability How do you measure the level of accountability What can be your unit of analysis? What can be your unit of analysis? Is your measure valid? Is your measure valid?

21. 2131 March 2009 @ Crawford School Precision In theory-oriented empirical research, we must deal with not only the relationship between a concept and a measure but also the quality of the measure itself. In theory-oriented empirical research, we must deal with not only the relationship between a concept and a measure but also the quality of the measure itself. There are two kinds of precision with which we shall be concerned: precision in measurement and precision in measures. There are two kinds of precision with which we shall be concerned: precision in measurement and precision in measures. In empirical research (both in quantitative and qualitative research), we first decide how much precise our act of measurement itself should be. Then, we decide how much precise our measured outcomes should be. In empirical research (both in quantitative and qualitative research), we first decide how much precise our act of measurement itself should be. Then, we decide how much precise our measured outcomes should be.

22. 2231 March 2009 @ Crawford School Precision in measurement There are three levels of measurement: nominal, ordinal and interval. There are three levels of measurement: nominal, ordinal and interval. Nominal measurement Nominal measurement Q. Where do you live? Q. Where do you live? A. “Inner North”, “Woden”, “Belconnen”, “Others” A. “Inner North”, “Woden”, “Belconnen”, “Others” Ordinal measurement Ordinal measurement “Lower class”, “Middle class”, “Upper class” “Lower class”, “Middle class”, “Upper class” “Large-sized”, “Medium-size”, “Small-sized” “Large-sized”, “Medium-size”, “Small-sized” Interval measurement Interval measurement Personal income, government expenditures, etc. Personal income, government expenditures, etc.

23. 2331 March 2009 @ Crawford School Levels of Precision Interval m is more precise than ordinal m, which in turn is more precise than nominal m. Interval m is more precise than ordinal m, which in turn is more precise than nominal m. All continuous measures are interval All continuous measures are interval Discrete measures are either ordinal or nominal. Discrete measures are either ordinal or nominal.

24. 2431 March 2009 @ Crawford School Precision in measures Once you choose the level of measurement (i.e., nominal, ordinal or interval), you may want to keep distinctions of measured outcomes (=measures) as fine as possible and practical. Once you choose the level of measurement (i.e., nominal, ordinal or interval), you may want to keep distinctions of measured outcomes (=measures) as fine as possible and practical. Examples Examples Countries are categorized into {Asia, Europe, Africa, Latin America, …} or {East Asia, Southeast Asia, South Asia, …}. Countries are categorized into {Asia, Europe, Africa, Latin America, …} or {East Asia, Southeast Asia, South Asia, …}. The level of corruption is {high, medium, low} or {very high, high, medium, low, very low}. The level of corruption is {high, medium, low} or {very high, high, medium, low, very low}. Person A’s income is {$52,325} or {$52,000}. Person A’s income is {$52,325} or {$52,000}.

25. 2531 March 2009 @ Crawford School Importance Preserving a high level of “precision in measurement” always deserves a strong priority in research. Preserving a high level of “precision in measurement” always deserves a strong priority in research. You should decide a level of “precision in measures” given the subject you are studying. You should decide a level of “precision in measures” given the subject you are studying. When we analyze data, the degree of “precision in measures” is determined by what we wish to do with data. The time to ensure that your measures will be as precise as you want is when you collect data. When we analyze data, the degree of “precision in measures” is determined by what we wish to do with data. The time to ensure that your measures will be as precise as you want is when you collect data.

26. 2631 March 2009 @ Crawford School Quantitative vs. qualitative The discussions thus far can be easily applied in quantitative research. It is important, however, to understand that the same conclusions hold for qualitative research. The discussions thus far can be easily applied in quantitative research. It is important, however, to understand that the same conclusions hold for qualitative research. The differences between quantitative and qualitative measurement involve how data are represented. Qualitative researchers use words like “more” or “less,” “larger” or “smaller,” and “strong” and “weak” and quantitative researchers use numbers. The differences between quantitative and qualitative measurement involve how data are represented. Qualitative researchers use words like “more” or “less,” “larger” or “smaller,” and “strong” and “weak” and quantitative researchers use numbers.

27. 2731 March 2009 @ Crawford School Notes A theory can produce more than one testable hypothesis. Always think as many observable implications of a theory (i.e., hypotheses) as possible. A theory can produce more than one testable hypothesis. Always think as many observable implications of a theory (i.e., hypotheses) as possible. A hypothesis can produce more than one “working” hypothesis. A variable can be measured in a number of different ways with different units of analysis. A hypothesis can produce more than one “working” hypothesis. A variable can be measured in a number of different ways with different units of analysis. It is often recommended to use different units of analysis and different measures to test a given hypothesis (and make a synthetic interpretation). It is often recommended to use different units of analysis and different measures to test a given hypothesis (and make a synthetic interpretation).

28. 2831 March 2009 @ Crawford School Next session Surveys Surveys Differences between interviews and surveys Differences between interviews and surveys Types and modes of surveys Types and modes of surveys Designing a survey Designing a survey Rules of asking questions Rules of asking questions Content Analysis Content Analysis Types of content analysis Types of content analysis Steps in content analysis Steps in content analysis