Highlights from the Program for the International Assessment of Adult Competencies (PIAAC), 2012 Jack Buckley National Center for Education Statistics (NCES) Washington, DC October

What is PIAAC?  International large-scale assessment administered in in 23 countries  16- to 65-year-olds, non-institutionalized, residing in the country, irrespective of nationality, citizenship, or language status  Laptop computer or paper-and-pencil:  In the U.S., 80% took the computer tests and 15% took the paper-and-pencil tests.  Assessment subjects:  Literacy, Numeracy, and Problem Solving in Technology-Rich Environments  Conducted in English in the U.S.:  Background survey in English or Spanish. About 4% could not complete the questionnaire because of language difficulties or learning or mental disabilities, and 1% could not complete it for other reasons. 2

General patterns of U.S. results  Below international average in all subject areas  Ranked better in literacy than in numeracy or problem solving in technology-rich environments  Higher percentage at low proficiency levels than international average  Percentages of top performers similar to or slightly lower than international average, depending on the subject  Performance gap between young and older population smaller than the average gap internationally 3

4 Participating countries Australia Austria Belgium Canada Cyprus Czech Republic Denmark Estonia Finland France Germany Ireland Italy Japan Korea, Rep of Netherlands Norway Poland Slovak Republic Spain Sweden United Kingdom United States Chile Greece Indonesia Israel Lithuania New Zealand Singapore Slovenia Turkey

What PIAAC reports Average Scores: Reported on a scale of for all domains. Proficiency Levels: Reported as the percentages of adults scoring at six performance levels in literacy and numeracy and at four performance levels in problem solving in technology-rich environments. 5

Literacy proficiency levels Locate single piece of information in familiar texts. Read relatively short digital, print or mixed texts to locate single text. Make matches between text and information that may require low level para- phrasing and drawing low-level inferences. Identify, interpret, or evaluate one or more pieces of information and often require varying levels of inference. Perform multiple- step operations to integrate, interpret, or synthesize information from complex texts, and may require complex inferences. Integrate information across multiple, dense texts; construct syntheses, ideas or points of view; or evaluate evidence based arguments. Below Level 1 Level 1 Level 2 Level 3 Level 4 Level 5 6

Literacy example item Below Level 1: Election results  The stimulus is a report of the results of a union election. It consists of several brief paragraphs and a simple table identifying the three candidates and the number of votes they received.  The test taker is asked to identify which candidate received the fewest votes. To do this, the test taker must simply compare the number of votes that each candidate received.  The word “votes” appears only in the question and in the table. Therefore, the task consists of recognizing this direct relationship between the two to infer the answer. 7

Literacy example item Level 4: Library search  The stimulus displays the results of a bibliographic search from a simulated library website.  The test taker is asked to identify a book suggesting that the claims made both for and against genetically modified foods are unreliable. To do this, the test taker needs to read the title and description of each book included in the search results.  Many pieces of distracting information are present. The necessary information must be inferred from the statement that the author “describes how both sides in this hotly contested debate have manufactured propaganda, tried to dupe the public and... [text ends].” 8

Numeracy proficiency levels Perform basic tasks: counting, arithmetic operations with whole numbers. Perform one- step tasks: count; sort; arithmetic operations; understanding simple percent (ex. 50%). Perform 2 or more calculations, simple measurement; spatial representation; estimation; and interpret simple tables, graphs. Understand & work with mathematical patterns, proportions, basic statistics expressed in verbal or numerical form. Perform analysis, complex reasoning, statistics and chance; spatial relationships; and communicat- ing well- reasoned explanations for answers. Understand complex abstract mathema- tical and statistical ideas, embedded in complex texts, draw inferences; arguments or models; justify, reflect on solutions or choices. Below Level 1 Level 1 Level 2 Level 3 Level 4 Level 5 9

Numeracy example item Below Level 1: Price tag  The stimulus for this item consists of four supermarket price tags. The tags identify the product, the price per pound, the net weight, the date packed, and the total price.  The test taker is asked to indicate the item that was packed first by simply comparing the dates on the price tags. 10

Numeracy example item Level 4: Education level  The stimulus for this item consists of two stacked-column bar graphs presenting the distribution of the Mexican population by years of schooling for men and women separately.  The y axis of each graph is labeled “percentage” and includes grid lines for 0, 20, 40, 60, 80, and 100%. The x axis is labeled “year” and presents data for 1960, 1970, 1990, 2000, and A legend identifies three categories of schooling: “more than 6 years of schooling,” “up to 6 years of schooling,” and “no schooling.”  The test taker is asked to indicate what percentage of men in Mexico had more than 6 years of schooling in 1970, choosing from a pull-down menu that has 10 response categories: “0–10%,” “10– 20%,” and so on. 11

Problem solving in technology-rich environments proficiency levels Tasks are well- defined involving use of only one function within a generic interface. Tasks require little or no navigation, and only a few steps to access information for solving the problem. There are few monitoring demands. Tasks require some navigation across pages and applications for solving the problem. Evaluating the relevance, some integration and inferential reasoning may be needed. Task may involve multiple steps and operators, navigation across pages and applications. There are typically high monitoring demands, and evaluation of relevance and reliability of information. Below Level 1 Level 1 Level 2 Level 3 12

Problem Solving in technology-rich environments example item Level 1: Party invitations  This task involves sorting s into pre-existing folders in an inbox.  An interface is presented with five s in an inbox. The test taker is asked to place the s, which are responses to a party invitation, into folders to keep track of who can and cannot attend a party.  The task is performed in a single environment, and the goal is explicitly stated. Solving the problem requires a relatively small number of steps and does not demand a significant amount of monitoring across a large number of actions. 13

Problem Solving in technology-rich environments example item Level 3: Meeting rooms  This task involves managing requests to reserve a meeting room on a particular date using a reservation system.  The task presents two applications: an interface (with a number of s requesting reservations stored in an inbox) and a web-based reservation tool that allows the user to assign rooms to meetings at certain times.  Successfully completing the task involves taking into account multiple constraints (i.e., the number of rooms available and existing reservations). These constraints generate a conflict (one of the demands for a room reservation cannot be satisfied), which has to be resolved by issuing a standard message to decline one of the requests. 14

15 U.S. PIAAC Findings

U.S. average literacy score (270) lower than the international average (273) Lower than in 12 countries : Japan, Finland, Netherlands, Australia, Sweden, Norway, Estonia, Flanders-Belgium, Czech Republic, Slovak Republic, Canada, Republic of Korea Not significantly different than in 5 countries : England and Northern Ireland- U.K., Denmark, Germany, Austria, Cyprus Higher than in 5 countries : Poland, Ireland, France, Spain, Italy 16

17 Below level 1 range: Level 1 range: Level 2 range: Level 3 range: Level 4/5 range: Median Seven countries had higher percentages of adults reaching the highest proficiency level (4/5) in literacy Median

Higher proportion of U.S. adults at the bottom levels of literacy 18

Lower than in 12 countries: Japan, Finland, Netherlands, Republic of Korea, Estonia, Flanders- Belgium, Australia, Sweden, Poland, Czech Republic, Germany, Austria Not significantly different than in 8 countries: Denmark, Slovak Republic, Canada, Norway, France, Ireland, Cyprus, England and Northern Ireland- U.K. Higher than in 2 countries: Spain, Italy U.S. 16- to 24-year-olds below international average in literacy, rank lower than 16- to 65-year-olds overall 19

Only oldest U.S. adults outperformed the international average in literacy 20 *p <.05. U.S. average score is significantly different from PIAAC international average.

Least educated adults below the international average in literacy 21 *p <.05. U.S. average score is significantly different from PIAAC international average.

Employed adults in the U.S. had lower average literacy scores than their peers internationally 22 *p <.05. U.S. average score is significantly different from PIAAC international average.

U.S. White adults had higher average literacy scores than either Black or Hispanic adults 23 *p <.05. Average score is significantly different from White average.

24 U.S. gaps in literacy scores larger than international average by parental education and nativity status

25 In literacy, U.S. gaps larger by educational attainment and skill level of job, but similar to international average by income and employment status

U.S. gaps in literacy scores similar to international average by gender, smaller by age, and larger by health status 26

U.S. average literacy score in 2012 not significantly different from 2003, but lower than in *p <.05. Average score is significantly different from PIAAC.

Summary of literacy findings:  Lower overall literacy scores than international average  Higher percentage of low performers than international average  Gaps between less advantaged socio-economic groups and more advantaged peers higher in U.S. than internationally  No change in overall U.S. literacy scores since

U.S. average numeracy score (253) lower than the international average (269) Lower than in 18 countries: Japan, Finland, Flanders-Belgium, Netherlands, Sweden, Norway, Denmark, Slovak Republic, Czech Republic, Austria, Estonia, Germany, Australia, Canada, Cyprus, Republic of Korea, England and Northern Ireland- U.K., Poland Not significantly different than in 2 countries: Ireland, France Higher than in 2 countries: Italy, Spain 29

Fifteen countries had higher percentages of adults reaching the highest proficiency level (4/5) in numeracy 30 Below level 1 range: Level 1 range: Level 2 range: Level 3 range: Level 4/5 range: Median

31 Higher proportion of U.S. adults at the bottom levels of numeracy

Lower than in 21 countries : Netherlands, Finland, Japan, Flanders-Belgium, Republic of Korea, Austria, Estonia, Sweden, Czech Republic, Slovak Republic, Germany, Denmark, Norway, Australia, Poland, Canada, Cyprus, France, Ireland, England and Northern Ireland- U.K., Spain Not significantly different than in 1 country: Italy U.S. 16- to 24-year-olds below international average in numeracy, rank lower than 16- to 65-year-olds overall 32

U.S. adults in all age groups below international average in numeracy 33 *p <.05. U.S. average score is significantly different from PIAAC international average.

U.S. adults at every education level below the international average in numeracy 34 *p <.05. U.S. average score is significantly different from PIAAC international average.

Employed adults in the U.S. had lower average numeracy scores than their peers internationally 35 *p <.05. U.S. average score is significantly different from PIAAC international average.

U.S. White adults had higher average numeracy scores than either Black or Hispanic adults 36 *p <.05. Average score is significantly different from White average.

U.S. gaps in numeracy scores larger than international average by parental education, not different by nativity status 37

38 In numeracy, U.S. gaps larger by educational attainment, income, and skill level of job, but similar to international average by employment status

39 In numeracy, U.S. gap similar to international average by gender, smaller by age, and larger by health status

U.S. average numeracy score in 2012 lower than in *p <.05. Average score is significantly different from PIAAC.

Summary of numeracy findings:  Lower overall numeracy scores than international average  Higher percentages of low performers than international average  Regardless of educational level or gender, U.S. adults lower than international average  Lower U.S. numeracy scores than in

U.S. average problem solving in technology-rich environments score (277) lower than the international average (283) Lower than in 14 countries : Japan, Finland, Australia, Sweden, Norway, Netherlands, Austria, Denmark, Czech Republic, Republic of Korea, Germany, Canada, Slovak Republic, Flanders-Belgium Not significantly different than in 4 countries: England and Northern Ireland- U.K., Estonia, Ireland, Poland 42

Eight countries had higher percentages reaching the highest proficiency level (3) in problem solving in technology-rich environments 43 Below level 1 range: Level 1 range: Level 2 range: Level 3 range: Median

44 Higher proportion of U.S. adults at the bottom levels of problem solving in technology-rich environments

Lower than in 14 countries: Republic of Korea, Finland, Sweden, Netherlands, Japan, Flanders-Belgium, Czech Republic, Norway, Australia, Germany, Austria, Canada, Denmark, Estonia Not significantly different than in 4 countries: England and Northern Ireland- U.K., Slovak Republic, Poland, Ireland U.S. 16- to 24-year-olds below international average in problem solving in technology-rich environments 45

Only oldest U.S. adults outperformed the international average in problem solving in technology-rich environments 46 *p <.05. U.S. average scores are significantly different from PIAAC international average.

Least educated adults below the international average in problem solving in technology-rich environments 47 *p <.05. U.S. average scores are significantly different from PIAAC international average.

Employed adults in the U.S. had lower average problem solving in technology-rich environments scores than their peers internationally 48 *p <.05. U.S. average scores are significantly different from PIAAC international average.

U.S. White adults had higher average problem solving in technology-rich environments scores than either Black or Hispanic adults 49 *p <.05. Average scores are significantly different from White average.

50 In problem solving in technology-rich environments, U.S. gaps similar to international average by educational attainment, income, employment status, and skill level of job

51 In problem solving in technology-rich environments, U.S. gaps similar to international average by gender and health status, but smaller by age

Summary of problem solving in technology-rich environments findings:  Lower overall problem solving in technology- rich environments scores than international average  Higher percentages of low performers than international average  Gaps between youngest and oldest age groups smaller in U.S. than internationally 52

U.S. PIAAC Findings Summary: 53 U.S. PIAAC Findings Summary:  Lower overall scores than international average in all subjects  Higher percentages of low performers than internationally  Larger gaps between less advantaged and more advantaged peers in literacy and numeracy, but not in problem solving in technology-rich environments  Relatively lower performance of young adults and those with high school education or less  Relatively higher performance of older adults in literacy and problem solving in technology-rich environments

54

For more information Contact: Eugene Owen NCES PIAAC at NCES: ‎ 55