Chapter 10 Audience measurement pt.2

Use and Misuse Last class we saw the controversy surrounding the PPM. Let’s take a step back. The whole process of ratings is the collection and interpretation of data. "There are three kinds of lies: lies, damned lies, and statistics.” (Mark Twain/Benjamin Disraeli)

Reliability and Validity The collection, interpretation and presentation of audience data is a little tricky, like any other type of research using statistics. The research has to be reliable and valid.

Reliability and Validity Reliability is the consistency of your measurement, or the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects. In short, it is the repeatability of your measurement. A measure is considered reliable if a person's score on the same test given twice is similar. It is important to remember that reliability is not measured, it is estimated.

Reliability and Validity Validity is the strength of our conclusions, inferences or propositions. More formally, Cook and Campbell (1979) define it as the "best available approximation to the truth or falsity of a given inference, proposition or conclusion." In short, were we right? Let's look at a simple example. Say we are studying the effect of strict attendance policies on class participation. In our case, we saw that class participation did increase after the policy was established. Each type of validity would highlight a different aspect of the relationship between our treatment (strict attendance policy) and our observed outcome (increased class participation).

Reliability and Validity Another way to look at it is that if a study is reliable, you can repeat the test and get similar results. If it is valid, you are testing what you want to test. So for the midterm, I gave you a tried and true test that has been taken time and time again, but it is in Greek, it is reliable but not valid because it isn’t testing your understanding of media studies, it is testing your ability to understand Greek AND your understanding of media studies. A more real world example is that if I wanted to test analytic ability and I just asked memorization questions, the test isn’t valid because it is testing the wrong skill.

Reliability Issues In the case of media research, if a particular method yields substantially different results over time, it ma ynot be reliable. People Meter (not PPM) results varied for broadcast TV early on, so questions were raised. Now that enough time has passed, we know that the collection method is fairly reliable (gives the same results). But this does not prove validity.

Validity Issues Ratings are claiming to report on the entire broadcast audience, but they probably best represent a broad middle-range majority of the audience. That means the very rich, the very poor, the very young, the very old and ethnic minorities are sometimes underrepresented (the PPM controversy).

Validity Issues TV ratings are measured in households. We already talked about why that might affect validity. The quality of the sample also affects validity (this also relates to the PPM issue). Sample- A set of individual units, drawn from some definable POPULATION of units, and generally a small proportion of the POPULATION, to be used for a statistical examination of which the findings are intended to be applied to the POPULATION. It is essential for such inference that the SAMPLE should be REPRESENTATIVE. (Norman Marsh:

Validity Issues Also, while the PPM for Radio measures listenership everywhere, the People Meter doesn’t measure ratings outside the house (hotels, sports bars etc…) Neilsen tried to implement a system for out of house viewing, but nobody wanted to pay for the service. They dropped the service in November 2008.

Correction Last class we talked about “Sweeps.” This is 4 times a year, plus up to 3 times more for other markets. This covers Diary based data. People Meter data is 52 weeks a year March 2009March 5 - April 1, 2009 May 2009April 23 - May 20, 2009 July 2009July 2 - July 29, 2009 November 2009October 29 - November 25, 2009

Hyping Hyping is the deliberate attempt to manipulate ratings during sweeps periods by offering promotions or special programs. Hard to catch, but if it is too blatant, Neisen or Arbitron will just toss results. Tampering- manipulating a small part of the sample. This doesn’t happen much either since it is usually obvious and results will be tossed.

Quality v. Quantity What we have been talking about is Quantitative analysis. This is not measuring the quality of the show or particular elements of a show. Because of an over abundance of choices, shows don’t need to get the numbers they used to. So a “quality” show with moderate numbers could still find a home on cable somewhere. Also, some companies are trying to measure “Q-Scores” The Q Score is a way to measure the familiarity and appeal of a brand, company, celebrity, cartoon character or television show. The higher the Q Score, the more well-known and well thought of the item or person being scored is (this is a PR/marketing technique being employed for advertising). The idea is that if a show has smaller numbers overall, but has some quality that is well-liked by a particular group, the show has value.

Schedule 4/2 Chapter 10, Ratings. Presentations. 4/7 Chapter 10, Ratings. Presentations. 4/9 Break 4/21 Quiz Chapter 8 &10. Start Chapter 11 Effects 4/23 Chapter 11 Effects 4/30 Quiz Chapter 11. Start Chapter 12. 5/5 Chapter 12 5/7 Network 5/12 Network 5/14 Final 5/21 Papers due in my mailbox by 12:00pm. I will leave campus for the semester at exactly 12:01pm. If the paper is late, it is an F

Presentations …