Measurement Approaches for Evaluation: We Get What We Measure Diana Eerma Lecturer on Economic Policy University of Tartu, Estonia International Evaluation Conference in Vilnius July 4, 2013

The aim of the presentation and research tasks To make suggestions for measurement in ex post evaluation under the ESF policy - Form a framework for measurement characteristics in social impact assessment. - Identify the challenges of measurement in evaluating the policy achievements. - Formulate lessons learnt from current program period Review some theoretical approaches of measurement. - Recommendations on possible improvements for the next programming period.

Economic impact assessment is an approach to evaluation (Source: Evaluating Socio Economic Development, 2003) Involving the interests of various economic entities (employers, employees, consumers, producers, areas); Introducing an element of time changes and insecurity (comparison of scenarios); Weighting of importance and intensity of economic activities (participation in total employment, participation in GDP); Comparison of different types of effects (direct or/and indirect).

Concerns and challenges of measurement in evaluation of policy achievements How to measure effects of policy and impact of financial instrument on policy goals in the framework of ESF aims? – wide range of issues: employment and social inclusion, education and life-long learning, institutional capacity and efficiency of public administrations. What approach and methods to apply in evaluation of results achieved in order to identify actual impacts after implementation of instruments? Ex post evaluation for corrective action or to make changes in the future programs. Is our measurement system and indicators in it proper to measure outcomes and set objectives? What gets really measured and what should be measured? How application of particular instrument may change welfare? Assessment of additionality in micro and macro level.

Assessing additionality: deadweight, displacement and substitution effects (Source: Evaluating Socio Economic Development, 2003) Assessing the Deadweight – an assessment of the output which would have occurred regardless of the project or intervention. Assessing the Displacement – examining the degree to which the outputs of the project have occurred at the expense of outputs elsewhere in the target area (in factor and product markets). Assessing the Substitution – this effect arises where a firm substitutes one activity for a similar other to take advantage of public sector assistance. Assessing whether Crowding out has occurred – increase in public expenditure cause other variables in the economy to adjust resulting in a decline in private expenditure. Assessing whether Crowding in has occurred – public expenditure encourages private expenditure to increase as a result of adjustments in other variable in the economy.

More systematic approach via social accounting The term – social accounting - has been evolved by several authors during more than last four decades One of the most important in the context is presented by Mook et al (2007:2): ‘A systematic analysis of the effects of an organization on its communities of interest or stakeholders, with stakeholder input as part of the data that is analyzed for the accounting statement.’

Conventional accounting as a subset of social accounting Source: Compiled by author based on O’Connor 2006:6; Grojer and Stark 1977:350) Social success of society (welfare accounting) Social accounting (social and environmental accounting) –economic and non-economic accountability to financial and non-financial stakeholders Conventional accounting – Economic accountability to financial stakeholders

Comparison of the accounting systems

Concepts and methods of social accounting for social success measurement (Source: Compiled by the author) Concepts of social accounting - spheres: education, health, environment, military, industries, etc Welfare Single goal - indicator/indicators Number of goals - indicators/groups of indicators Bookkeeping - corporate accounting - public success accounting Indicator analysis - for internal purposes - for external purposes SOCIAL SUCCESS

Major measuring problems in social accounting (Kaya and Yayla 2007) Disclosures of social and environmental information tend to be a fragmentary; Voluntary reporting does not produce widespread, consistent and systematic practices; Definitions problems of most of the social accounting terms; Presenting values and intentions without supporting details; Reporting only good news; Making inaccurate claims; The difficulties in measuring social ‘externalities’.

A bookkeeping system concept for social accounting Delineation of the concept is following: About content: –Concentration is on welfare oriented social accounting –Developed concept is based on the periodical accounting and a bookkeeping system. It does not deal with extended investment accounting. –Welfare change oriented evaluations with adaptation of existing evaluation methods get applied to the necessity of social accounting that stem from goals, tasks and information available. –The welfare change is measured in terms of particular country. About techniques: Mainly are applied microeconomic theory and techniques, and bookkeeping rules of business administration and simple macroeconomic effects according to the effects and transactions to be evaluated. There does not take place development of economic models of effect measurement on computerized general equilibrium model or location choice model with the equations. The current concept is not dealing with econometrical approaches, but only with simple statistically available information using methods.

Methodological challenges to the development of social accounting bookkeeping concept 1)What has to be assessed? 2)How have social benefits and social costs to be deferred? 3)What booking techniques should be applied? 4)How should social success be measured? 5)How to consider long lasting effects? 6)How to delineate the group whose welfare becomes measured? 7)Which sequence of effects should be taken into account? 8)How to classify social benefits and social costs? 9)How to handle the problem of alternative situations? 10)To what principle should the chart of accounts be oriented?

Methodological challenge (4)

Social Net Benefit (SNB) = consumer surplus + turnover + monetary value of external effects - producer surplus - costs - monetary value of external effects Social Net Benefit (SNB) = (consumer surplus - producer surplus) + (external benefits - external costs) (Additional social accounting) + (turnover - costs) (Commercial profit assessment) Social bookkeeping systems, which comprises: (1) Social accounting = an additional social accounting + commercial bookkeeping (2) Current SNB = additional social success operating statement + commercial profit (loss) account (3) Total social balance = additive social balance + commercial balance

Example: The development of a bookkeeping system for social accounting of the University faculty

Additional operating social success statement of the University faculty (Source: Friedrich and Eerma, 2010)

Recommendations Use/apply more systematic approach for ex post evaluation measurement which meets the following principles: -measures achieved success of society, -identifies success of economic unit based on inside and outside effects, -considers social externalities additionally to financial effects, -takes into account groups whose welfare should be increased.

References Evaluating Socio Economic Development, Sourcebook 2: Techniques and Tools, Friedrich, P. and Eerma, D. (2010). “Application of social accounting for a university in the case of faculty”, in Backhaus, J., Eamets, R., Eerma, D. (eds.), Economics of Education. Issues of Trans­ition and Transformation, Münster: LIT, pp Grojer, J-E. and Stark, A. (1977). Social Accounting: A Swedish Attempt. Accounting, Organizations and Society, Vol 2, No 4, pp Kaya, U. and Yayla, H.E. (2007). Remembering Thirty-five Years of Social Accounting: A Review of the Literature and the Practice, MPRA, München. Mook, L., Quarter, J. and Richmond, J. B. (2007). What Counts. Social Accounting for Nonprofits and Cooperatives. 2nd ed. Sigel Press, London. O’Connor, L. (2006). “Empirical research in social and environmental accounting: a meta-review”, Sloan Management Review, i, pp.1-20.

Thanks for your attention! ACKNOWLEDGEMENT Author acknowledges the support of the Estonian Ministry of Science and Education foundation project No SF s08 and The Research and Innovation Policy Monitoring Program No (TIPS SV 3).