Using regional differences to Estimate The effects of State and mid-level management on educational outcomes Martin Carnoy, Stanford University Higher School of Economics, April 10, 2018

Federal Systems and Education In federal systems, sub-national units have considerable legal jurisdiction over educational policy. Federal governments can, in various degrees, depending on the country, also influence educational policy. However, there is great variation between states in educational outcomes even correcting for SES differences in the students who attend schools in the different states/regions and differences in the resources available to schools in these regions.

Focusing on U.S. States International tests have been applied to random samples of students in U.S. states, and also to states in other federal system countries. PISA data exist for three states in the U.S. (2012), many states in Brazil, Mexico (all states) , Australia (all states) and Germany (all states in certain years). TIMSS has been sampling students in a few U.S. states since 1995, and nine states in 2011. The results in all these countries show considerable variation in students’ performance among states, even adjusted for student FAR.

Analyzing U.S. State PISA Performance

U.S. State TIMSS Performance Over Time

Minnesota and Iowa from the NAEP

Massachusetts and Connecticut from the NAEP

Texas and California from the NAEP

Changes in Average SAEB 8th Grade Portuguese Score, by Mother’s Education, 1995-2011

The Pisa data by state are not as reliable as in countries with ranDom samples in states, but we do have data for the national test—saeb—and here are some state compArisons

Here are some more states—these are all adjusted for SES and other student characteristics

We did a similar analysis of PISA scores in Australia—again controlling for SES

Sao Paulo: State administrative regions, high value added, 5th to 9th grade, 2010-2012 cohorts Mathematics Portuguese Sub-region within the state Years that the sub-region has had high VA Taquaritinga 3 Sao Roque Jose Bonifacio Apiai Itapeva 2 Registro Catanduva Braganca Paulista Angradina Votuporanga Caraguatatuba Mirante do Paranapanema Piraju

Sao Paulo: State administrative regions, low value added, 5th to 9th grade, 2010-2012 cohorts Mathematics Portuguese Sub-region within the state Years that the sub-region has had high VA Osasco 3 Marilia Tupa Garulhos Norte Santo Anastacio Centro Sul 2 Mogi das Cruces Diadema Santo Andre Lins

What does all this tell us? There is important variation in student achievement progress in time among administrative units. This variation exists even after we ”adjust” for student social class, classroom, and school level variables, such as teacher experience and education, —we looked at states in the US, Australia, and Brazil and administrative units within Brazilian states. We have the advantage in many Brazilian states that we can link individual students with their achievement scores and follow their achievement across grades and can link their achievement gains with their teachers’ characteristics. This allows us to estimate which regions and municipalities within a state such as Sao Paulo seem to be consistently increasing student achievement more in 5th to 9th grades, and those that are not doing as well.

Implications These are baseline data—they provide evidence of differences that may be due to the quality of state and municipal management of education. It is another thing to show that middle management is the likely CAUSE of such differences. But, even if we cannot show cause, we can possibly identify differences in management between higher and lower performing administrative units and this is an important topic of further research. We are about to undertake such research in two states of Brazil—Sao Paulo and Pernambuco— and a number of U.S. states.

What kind of data would we need in Russia to do a similar study? The data exist in Russia to compare regional administrative efficiency, but they are not readily available to researchers. If we could get GIA and EGE data by school over the past five or six years in the most populous and many adjoining regions, and gather socio-economic data about the schools and about the schools’ physical locations, these thousands of observations of schools’ academic performance over time could be aggregated to the region level to show relative educational administrative differences between the regions. We could then study the sources of these differences. This is important because it is likely that there are large effects on student achievment of middle level administrative practices.