1. Using regional differences to Estimate The effects of State and mid-level management on educational outcomes
Martin Carnoy, Stanford University
Higher School of Economics, April 10, 2018

2. Federal Systems and Education
In federal systems, sub-national units have considerable legal jurisdiction over educational policy.
Federal governments can, in various degrees, depending on the country, also influence educational policy.
However, there is great variation between states in educational outcomes even correcting for SES differences in the students who attend schools in the different states/regions and differences in the resources available to schools in these regions.

3. Focusing on U.S. States
International tests have been applied to random samples of students in U.S. states, and also to states in other federal system countries. PISA data exist for three states in the U.S. (2012), many states in Brazil, Mexico (all states) , Australia (all states) and Germany (all states in certain years).
TIMSS has been sampling students in a few U.S. states since 1995, and nine states in 2011.
The results in all these countries show considerable variation in students’ performance among states, even adjusted for student FAR.

4. Analyzing U.S. State PISA Performance

5. U.S. State TIMSS Performance Over Time

6. Minnesota and Iowa from the NAEP

7. Massachusetts and Connecticut from the NAEP

8. Texas and California from the NAEP

9. Changes in Average SAEB 8th Grade Portuguese Score, by Mother’s Education, 1995-2011

10. The Pisa data by state are not as reliable as in countries with ranDom samples in states, but we do have data for the national test—saeb—and here are some state compArisons

11. Here are some more states—these are all adjusted for SES and other student characteristics

12. We did a similar analysis of PISA scores in Australia—again controlling for SES

13. Sao Paulo: State administrative regions, high value added, 5th to 9th grade, 2010-2012 cohorts
Mathematics
Portuguese
Sub-region within the state
Years that the sub-region has had high VA
Taquaritinga 3
Sao Roque
Jose Bonifacio
Apiai
Itapeva 2
Registro
Catanduva
Braganca Paulista
Angradina
Votuporanga
Caraguatatuba
Mirante do Paranapanema
Piraju

14. Sao Paulo: State administrative regions, low value added, 5th to 9th grade, 2010-2012 cohorts
Mathematics
Portuguese
Sub-region within the state
Years that the sub-region has had high VA
Osasco 3
Marilia
Tupa
Garulhos Norte
Santo Anastacio
Centro Sul 2
Mogi das Cruces
Diadema
Santo Andre
Lins

15. What does all this tell us?
There is important variation in student achievement progress in time among administrative units.
This variation exists even after we ”adjust” for student social class, classroom, and school level variables, such as teacher experience and education, —we looked at states in the US, Australia, and Brazil and administrative units within Brazilian states.
We have the advantage in many Brazilian states that we can link individual students with their achievement scores and follow their achievement across grades and can link their achievement gains with their teachers’ characteristics.
This allows us to estimate which regions and municipalities within a state such as Sao Paulo seem to be consistently increasing student achievement more in 5th to 9th grades, and those that are not doing as well.

16. Implications
These are baseline data—they provide evidence of differences that may be due to the quality of state and municipal management of education.
It is another thing to show that middle management is the likely CAUSE of such differences.
But, even if we cannot show cause, we can possibly identify differences in management between higher and lower performing administrative units and this is an important topic of further research.
We are about to undertake such research in two states of Brazil—Sao Paulo and Pernambuco— and a number of U.S. states.

17. What kind of data would we need in Russia to do a similar study?
The data exist in Russia to compare regional administrative efficiency, but they are not readily available to researchers.
If we could get GIA and EGE data by school over the past five or six years in the most populous and many adjoining regions, and gather socio-economic data about the schools and about the schools’ physical locations, these thousands of observations of schools’ academic performance over time could be aggregated to the region level to show relative educational administrative differences between the regions.
We could then study the sources of these differences.
This is important because it is likely that there are large effects on student achievment of middle level administrative practices.