Quantitative evaluation of texts

Slides:



Advertisements
Similar presentations
The Social Scientific Method An Introduction to Social Science Research Methodology.
Advertisements

Reliability and Validity of Researcher-Made Surveys.
Qualitative text analysis. Why do qualitative text analysis? A number of scholars say you cannot capture the meaning of a text by counting the number.
Developing a coding scheme for content analysis A how-to approach.
Conceptualization, Operationalization, and Measurement
Part II Sigma Freud & Descriptive Statistics
Part II Sigma Freud & Descriptive Statistics
MGT-491 QUANTITATIVE ANALYSIS AND RESEARCH FOR MANAGEMENT
Measurement Reliability and Validity
Three Types of Unobtrusive Research 1.Content analysis - examine written documents such as editorials. 2.Analyses of existing statistics. 3.Historical/comparative.
Developing the Research Question
Agenda for January 25 th Administrative Items/Announcements Attendance Handouts: course enrollment, RPP instructions Course packs available for sale in.
Observing Behavior A nonexperimental approach. QUANTITATIVE AND QUALITATIVE APPROACHES Quantitative Focuses on specific behaviors that can be easily quantified.
Reliability and Validity. Criteria of Measurement Quality How do we judge the relative success (or failure) in measuring various concepts? How do we judge.
CORRELATIO NAL RESEARCH METHOD. The researcher wanted to determine if there is a significant relationship between the nursing personnel characteristics.
RESEARCH DESIGN.
Business and Management Research
RESEARCH IN EDUCATION Chapter I. Explanations about the Universe Power of the gods Religious authority Challenge to religious dogma Metacognition: Thinking.
COLLECTING QUANTITATIVE DATA: Sampling and Data collection
By: Christopher Prewitt & Deirdre Huston.  When doing any project it is important to know as much information about the project and the views of everyone.
Sociological Research Methods and Techniques
Technical Adequacy Session One Part Three.
Content analysis (Holsti)
Final Study Guide Research Design. Experimental Research.
Quantitative Research 1: Sampling and Surveys Dr N L Reynolds.
OBSERVATIONAL METHODS © 2012 The McGraw-Hill Companies, Inc.
Evaluating a Research Report
Analyzing and Interpreting Quantitative Data
Chapter 2 Research Methods in Social Psychology. Chapter Outline  Characteristics of Empirical Research  Research Methods  Research in Diverse Populations.
The What and the Why of Statistics The Research Process Asking a Research Question The Role of Theory Formulating the Hypotheses –Independent & Dependent.
SPSS Example Demonstrate Descriptives and Explore menu.
Validity and Reliability Edgar Degas: Portraits in a New Orleans Cotton Office, 1873.
Chapter 1: The What and the Why of Statistics  The Research Process  Asking a Research Question  The Role of Theory  Formulating the Hypotheses  Independent.
Chapter 10 Analyzing Content: Historical, Secondary, and Content Analysis, and Crime Mapping.
Assumes that events are governed by some lawful order
Quantitative and Qualitative Approaches
Hypothesis Testing A procedure for determining which of two (or more) mutually exclusive statements is more likely true We classify hypothesis tests in.
6. Evaluation of measuring tools: validity Psychometrics. 2012/13. Group A (English)
CHAPTER 12 Descriptive, Program Evaluation, and Advanced Methods.
Copyright  2003 by Dr. Gallimore, Wright State University Department of Biomedical, Industrial Engineering & Human Factors Engineering Human Factors Research.
Conducting and Reading Research in Health and Human Performance.
Validity and Reliability Neither Valid nor Reliable Reliable but not Valid Valid & Reliable Fairly Valid but not very Reliable Think in terms of ‘the purpose.
Scientific Method & Descriptive Research Methods Module 5.
Sampling/ Qualitative Research The Curious Skeptics Still at Work.
Copyright © Allyn & Bacon 2008 Intelligent Consumer Chapter 14 This multimedia product and its contents are protected under copyright law. The following.
OBSERVATIONAL METHODS © 2009 The McGraw-Hill Companies, Inc.
Chapter 6: Analyzing and Interpreting Quantitative Data
How To Analyze a Reading Presented By: Dr. Akassi Content From The Norton’s Field Guide To Writing.
ACM 4063 COMMUNICATION RESEARCH Lecture 7. Content analysis (CA)  A research technique for making references by systematically and objectively identifying.
RESEARCH METHODS IN INDUSTRIAL PSYCHOLOGY & ORGANIZATION Pertemuan Matakuliah: D Sosiologi dan Psikologi Industri Tahun: Sep-2009.
Chapter 6 - Standardized Measurement and Assessment
Validity & Reliability. OBJECTIVES Define validity and reliability Understand the purpose for needing valid and reliable measures Know the most utilized.
How Psychologists Do Research Chapter 2. How Psychologists Do Research What makes psychological research scientific? Research Methods Descriptive studies.
Assistant Instructor Nian K. Ghafoor Feb Definition of Proposal Proposal is a plan for master’s thesis or doctoral dissertation which provides the.
Chp. 2 – Sociological Research
Undergraduate School of Criminal Justice
Measurement Chapter 6. Measuring Variables Measurement Classifying units of analysis by categories to represent variable concepts.
RESEARCH METHODS Lecture 36. NON-REACTIVE RESEARCH.
Literary Genres are a category or certain kind of literature or writing. These categories are identified by examining the characteristics of each piece.
Chapter 26: Generalizations and Surveys. Inductive Generalizations (pp ) Arguments to a general conclusion are fairly common. Some people claim.
Unit 9– Seminar Analyzing Content: Historical, Secondary, and Content Analysis and Crime Mapping Professor Chris Lim, MA, Ph.D.(ABD)
CHAPTER ONE EDUCATIONAL RESEARCH. THINKING THROUGH REASONING (INDUCTIVELY) Inductive Reasoning : developing generalizations based on observation of a.
RESEARCH METHODS Lecture 36
Hypothesis Testing and Confidence Intervals (Part 1): Using the Standard Normal Lecture 8 Justin Kern October 10 and 12, 2017.
Analyzing and Interpreting Quantitative Data
Chapter 13 Quantitative Analysis of Text
Dr. Debaleena Chattopadhyay Department of Computer Science
Two Halves to Statistics
Unit 2 – Methods Objective 1 Describe quantitative and qualitative  methods such as surveys, polls, and statistics used in sociological research.  Objective.
Chapter 10 Content Analysis
Presentation transcript:

Quantitative evaluation of texts Content analysis Quantitative evaluation of texts

Questions about content When we talk about “cop shows” or “news” or “sports” we think about certain kinds of content. Usually, we perceive certain regularities in the content and notice when a single ‘text’ or ‘artifact’ deviates from those expectations. Sometimes we think certain regularities exist, while others dispute our beliefs.

The need for careful analysis Because our own hunches and expectations can be in error, and much of our understanding of the effects, values, and role of telecommunications is dependent upon the nature of the content of television, radio, film, videogames, etc. it is often necessary to more carefully analyzed that content.

Text analyses Many ways of evaluating content/texts are available. We call the entire group of methods “text analysis.” The most heavily quantitative form of text analysis is “content analysis.”

Definition Content analysis: A research technique for making inferences by systematically and objectively measuring specified characteristics of a text.

Content analysis in telecommunications research Content analysis is probably the most common form of research found in scholarly study of telecommunications It demands the least money and resources The downside is that many consider it “an easy publication” and produce very low-quality work

Goals of text analysis To explain the nature of communication. Describe the content, structure, and functions of the messages contained in texts. What does the text mean? How does it achieve that meaning? To describe how communication is related to other variables. Input variables – Outcome variables For example: How does a corporate takeover affect television news coverage? To evaluate texts by using a set of standards or criteria. Must establish a set of standards against which the communication can be compared. Example: Is the text too hard to read for 12-year-olds?

Types of texts Most any fixed symbolic whole—a story, a textbook, a church, a transcribed conversation, a website, and on and on can be considered a ‘text’. Sometimes a whole series of stories (Star Trek, season 2) may be considered a ‘text’.

Acquiring texts Listen to conversations in naturalistic settings Conversations produced in a lab Visit rooms of teenage girls Literary or historical sources (novels or films) Record shows off the air Visit or mirror websites

Procedures Determine the recording units Develop content categories Select the text(s) to be analyzed Determine the recording units Develop content categories Train observers to code units into categories Carry out the coding while monitoring for quality Analyze the data

Sampling in content analysis Population: totality of texts we want to say something about This is often more difficult than it seems All issues of the Herald Leader over a period of a year? All coverage of terrorism in the elite press? We can analyze a census or we can sample The same sorts of sampling techniques used for surveys can be applied here Random v. non-random sampling Many non-random samples chosen for theoretical as well as convenience reasons

Sampling Commonly multiple stages in sampling documents Selecting communication sources Newsweek Prime Time television Sampling documents Pick an issue, particular shows Sampling within documents Front page v. all pages, etc.

Units of observation Chosen first according to theory, then by convenience Articles Broadcasts Books Pictures Movies Letters Conversations

Recording units Recording units are the actual ‘pieces’ of the observational units that are scored according to your category scheme For example, if I were observing a single episode of NCIS, I might score every 5 minutes of the show for the presence or absence of humor. The 5-minute segment would be my recording unit.

Recording units Single word or symbol Theme Character May be too small—large number of data points generated Theme Single assertion about some subject May have overlapping themes Character Person or animal categorized rather than words or themes Sentence or paragraph May have ambiguous or conflicted evidence of one or more categories of content Item Whole book, film, radio program Difficulty coding into single categories Physical size measure Column inches Number of seconds

Coding categories The category scheme is the set of dimensions you use to evaluate your recording units and the available options you have for scoring one recording unit on each dimension For example: Recording unit: Scene Dimension: Emotionality of a scene Scoring options: High/Medium/Low

Coding categories The coding categories must be carefully developed in order to see that when the actual data are generated, they answer your theoretic questions of the text I can’t tell you how many times I’ve had to reject manuscripts because the coding scheme was not adequate to answer the theoretical questions posed in the literature review

Coding scheme Conceptualization  coding categories The code book provides the rules for assigning a coding unit to one or another category It is an actual set of rules for assigning the proper codes (scoring) to each coding unit

Coding rules What are the rules for determining which category a given recording unit should be placed in? How do we know whether a given paragraph is pro-Lunsford, neutral, or anti-Lunsford? This is a crucial part of the coding scheme. A naïve coder who simply applies the rules should get the outcome the theorist/ researcher intended.

Good coding categories Categories should be: Exhaustive Mutually exclusive Derived from a single classification principle Independent Adequate to answer the questions asked of the data

Practice coding In order to see that coders use the instrument as the researcher intended, the researcher holds practice sessions Related content, usually not from the actual sample, is coded and the results discussed

Coding sheet example Coding units Coding categories Length of song (secs) # different words Main topical focus # instruments played Vocal enhancement Oops I did it again We are the world Stairway to heaven Unchained melody

Coding reliability To ensure that the coding scheme is reliable we have to test it Coders score identical content The more often different coders produce the same scores for the identical content, the more reliable the coding scheme is Results are compared using statistical tests for reliability Cronbach’s alpha; Krippendorff’s alpha A rule of thumb is that the coding scheme is reliable if alpha is at least .70

Reliability v. Validity v. Precision The highest levels of reliability are usually found with very simple, extreme codes (true v. false; happy v. sad) but these simple codes often don’t provide the precision we want (clearly true, seemingly true, ambiguous, seemingly false, clearly false) and therefore reduce the value of the results—validity may suffer. The researcher has to consider the tradeoff

Data analyses Descriptive statistics are often used Percentages Mean Standard deviation May compare across texts To test hypotheses, etc. Compare findings to some prediction Relative percentages among categories, between sources on same categories Correlations among categories, with predictor variables, with outcome variables E.g., goriness of violence with measures of audience enjoyment