How do numbers sing? The uses of statistics in policy environments.

Epistemology and semiotics  Numbers are used to tell stories that decision-makers assume are reality  In and of themselves, numbers are not evidence  Numbers are representations of reality  The story is the derived reality

Transformations of meaning STORY (reality #1) shaping, presentation, translation NUMBERS (reality #2) shaping to determine data elements EVIDENCE (reality #3) determined by the STORY

As representations of reality... Numbers are assembled within metric categories (continuous, categorical, dichotomous) and semantic categories (e.g. “academic year”) Once assembled, their representation is controlled by denominator, that is, the definition of the universe being considered

Quality control at the point of assembly  Evidence is gathered and first turned into numbers at micro levels, and it is at these levels that quality control must function at its highest intensity  Aggregates of data will follow; derived variables will follow.  The further removed from the first construction, the more statistical noise

When numbers become data and then become stories...  They serve a propagandistic function in the sense that they are used to set the conditions of policy  The data can be manipulated by categories and denominators to produce good stories or bad stories  But honest stories are a better policy. Why?

We can tell a bad story, a good story, or an honest story. An honest story...  Is derived, as much as possible, from unobtrusive evidence, e.g. student records  Always tells you who is in the denominator— and who is not in the denominator  Matches its universe against another external reference point  Does not mix people from different planets  Hence, provides more accurate indicators of where problems may lie

Case #1: “Retention” of students from year 1 to year 2 of university study  The statistical definition of “retention” was based on institutions providing data of the proportion of students who started in the fall term of year 1 who were still enrolled in the fall term of year 2 at the same institution  But a national sample of student records found that (a) 20 percent of the students did not start in the fall term, (b) 62 percent attended during summer terms, and © 57 percent attended more than one institution

Case #1 (continued)  The “propaganda” of low retention rates damages the reputation of institutions  The policy issues involve financial support for institutions, principally by state governments  The honest story—based on unobtrusive evidence (student records), taking into attendance patterns into account, and following the student—results in a very high system retention rate, and relieves pressure on institutions

Case #1 (continued)  The honest story goes on to raise another set of questions about the quality of “retention.” Using student records, it asks  How many credits were earned in the 1 st year by students who were “retained”? How many remedial courses did they take? What was their grade point average? And how to these (and other features of student academic history) relate to their course-taking and performance in secondary school

Case #1 (continued again)  The honest data story says that while students can be “retained” in the system at a high rate (88 percent, in fact), that does not mean these students have sufficient momentum to complete degrees  With these data, the honest story can isolate groups of students who can be targeted for special pre-college preparation programs, and/or who can receive better academic advisement in college

Prelude to Case #2: What proportion holds a 1 st university degree?  Population years old 29%  All students entering postsecondary 25% education in 1995 by 2001  Recent secondary school graduates 67% entering 4-year colleges in 1992, by 2000 The answer is in the denominator

Case #2: What proportion of postsecondary students earns a 1 st university degree?  People who want to tell bad stories do not discriminate the student population by age in the term of first attendance  People who want to tell bad stories include students who attend non-degree- granting vocational schools in the denominator

Case #2 (continued)  If student records are used, and  The denominator marks all students who (a) start out in higher education by age 20-21, and, at any time, (b) attend an institution that grants 1 st university degrees...  Then 2 out of 3 students earns a 1 st university degree by age 26/26. That is very good.

Why raise Case #2?  Because the bad stories (low completion rates) dominate the media, they set a tone for the policy environment that offers excuses for reducing public financial support  This is a classic case of the battle between bad and honest stories to determine the policy environment

Case #3: Quality control in data entry: occupational classifications  Computer Assisted Telephone Interviews  Literal strings of words entered  Data entry clerk makes classification  When examined later, 2000 out of 12,000 classifications were wrong  Problems lie in broad categories such as Manager: Mid-Level, and in distinctions between Laboratory Technician/Other Technical/Medical Services.

If we do not classify occupations accurately...  We cannot analyze relationships between education and the labor market with confidence  Vocational/occupational education policy will not be targeted for highest efficiency  Predictions—and the public funding commitments based on predictions—will not be effective

CASE #4: Problems in coding academic course work: what questions do we ask? 4-year colleges 2-year colleges Less than 2-year Telecomm Technology 66% 22% 12% Information Technology Veterinary Medicine

CASE #4: These are not numbers—they are data. What do they tell us? 1) We have made a mistake in placing Telecommunications Technologies and Information Technologies in the same aggregate category of “Engineering Technologies” 2) There were gross errors in classifying “Veterinary Medicine” courses We can fix #2 easily. But #1 is not an easy case.

Confusions in data entry for “Veterinary Medicine”  Courses such as ‘Animal Care’ or ‘Clinical Animal Technology’ are vocational courses for veterinary assistants—Code  Course titles such as ‘Feline Medicine’ or ‘Avian Diseases’ are professional courses for veterinarians---Code  It is not uncommon for a School of Veterinary Medicine to prepare students at both levels, but most vocational offerings are at 2-year colleges

It is worth repeating after these examples... STORY (reality #1) shaping, presentation, translation NUMBERS (reality #2) shaping to determine data elements EVIDENCE (reality #3) determined by the STORY