A New Approach in University Evaluation: the Balanced Scorecard Angela Maria D’Uggento, Massimo Iaquinta,Vito Ricci Università degli Studi di Bari Direzione Analisi Statistica – Area Studi, ricerche e programmazione Staff di Rettorato Statistical modelling for University Evaluation: an International Overview Università di Foggia- Mattinata (FG), 5th-6th September 2008

A New Approach in University Evaluation: the Balanced Scorecard Balanced scorecard: a definition The balanced scorecard (BSC) is a strategic planning and management system used to align business activities to the vision and strategy of the organization and to monitor organization performance against strategic goals. It embodies a set of measures that gives top managers a quick but exhaustive vision of the business; in fact, it complements the financial measures with operational measures on: 1) customer satisfaction; 2) internal processes 3) organizational innovation; 4) improvement activities (Morard et Stancu, 2005). It was created by Kaplan and Norton as a performance measurement framework that added strategic non-financial measures to traditional financial ones (Kaplan, Norton 1992).

A New Approach in University Evaluation: the Balanced Scorecard The BSC has evolved from its early use to a full strategic planning and management system (Kaplan, Norton 1996). Born in the business managerial field, in the last decades it has drawn the attention of public administrators, becoming a useful tool to carry on evaluation during years. Some foreign Universities have adopted the BSC: Edinburgh (UK), San Diego, California (USA), Bond (Australia). Any Italian University has not a BSC but it’s currently used either by private companies or by Public Administration (Health Care, Local Government, etc).

A New Approach in University Evaluation: the Balanced Scorecard Balanced scorecard: the 4 traditional Perspectives Financial: it concerns with economic and financial aspects; Stakeholders: this perspective allows the Organization to fit outcome metrics to “customers” needs and expectations; Internal process: it’s the perspective of operational processes; d) Learning and Growth: it includes employee training, empowerment, informative systems.

A New Approach in University Evaluation: the Balanced Scorecard BSC: traditional perspectives’ relations A New Approach in University Evaluation: the Balanced Scorecard

A New Approach in University Evaluation: the Balanced Scorecard Balanced scorecard: main steps Specification of strategic objectives and goals Definition of a Strategical Map to highlight cause and effect relationships among strategic objectives Selection of the KPA (key performance actions) and the KPI (key performance indicators) Implementation of the BSC dashboard for monitoring and evaluation

A New Approach in University Evaluation: the Balanced Scorecard The Balanced scorecard of the University of Edinburgh/1 Four perspectives and performance indicators

A New Approach in University Evaluation: the Balanced Scorecard The Balanced scorecard of the University of Edinburgh/2 Four perspectives and performance indicators

The Balanced scorecard of the University of Edinburgh/3 Specification of the single indicator (1)

The Balanced scorecard of the University of Edinburgh/4 Specification of the single indicator (2)

A New Approach in University Evaluation: the Balanced Scorecard A BSC for an Italian University: the revised Perspectives (Del Sordo et al, 2007) d Students (Academic Courses) (needs and expectations) Community (Research) Learning and Growth (employee training, empowerment, informative systems.) Internal processes (operational processes) Economic and financial (efficency & efficacy) Stakeholders Learning & Growth Internal Processes Financial

A New Approach in University Evaluation: the Balanced Scorecard A BSC for University: Our proposal Introduction of MiUR five Guidelines and Indicators about University Planning 2007-2009 (MiUR, DM n.362/07 e DM n. 506/07) into the BSC standard frame; 2) Translation of the six guidelines of Bari University Planning into the four BSC traditional dimensions.

A New Approach in University Evaluation: the Balanced Scorecard A BSC for University: Our proposal The perspective Stakeholders would be divided into 4 sections (Figure 1): Academic courses, Development of scientific research, Relationship with territory, Internationalization programs so representing all the possible “players”: students and their families, researchers, local, domestic and foreign Institutions and firms.

A New Approach in University Evaluation: the Balanced Scorecard Figure 1 - BSC Perspectives and Guidelines of University planning

A New Approach in University Evaluation: the Balanced Scorecard A BSC for University: MiUR Guidelines & Evaluation Fields (from DM 506/2007) Academic courses (3 indicators) Scientific Research (5 indicators) Services to Students and Researchers (5 indicators) Internationalization Programmes (4 indicators) Teaching and non-teaching staff policies (4 indicators) Relationships with territory (Bari Plan 2007-2009)

A New Approach in University Evaluation: the Balanced Scorecard A BSC for Bari University Strategic objectives, Initiatives/actions & targets Key Performance Indicators University Plan 2007-2009 21 MiUR indicators + 21 Performance indicators

A New Approach in University Evaluation: the Balanced Scorecard A BSC for University: some performance indicators of Bari University

A New Approach in University Evaluation: the Balanced Scorecard A BSC for University: work in progress If implemented in Italian Universities, BSC could represent: a standardized model and as a reporting tool for decision-makers and stakeholders; a framework to measure the multidimensional performance of the University (intellectual & relational capital), whose core business is producing and offering knowledge intensive services and to maximize students’ satisfaction; a modern and integrated approach, particularly appreciated by the University management which, sometimes, hardly succeeds in monitoring, at the same time, several variables which are usually untied; an instrument to rationalize data collection and to minimize information overload

A New Approach in University Evaluation: the Balanced Scorecard A BSC for University: work in progress There is no doubt that one criticism of the BSC is its construction. Sometimes, the conceptualization is very hard to apply in a university environment, but a modeling of the system is required as a starting point (how many Italian universities have “mapped” their internal processes ?). As to the methodological approach, in recent years the research directions include the structural equation modeling, encompassing Partial Least Squares models, and the multivariate approach, in addition to the traditional statistical techniques.

A New Approach in University Evaluation: the Balanced Scorecard A BSC for University: work in progress Structural equation model provides the estimation of multiple interrelated dependent relations between variables and also the investigation of latent variables. PLS regression, generalizing and combining features from principal component analysis and multiple regression, is particularly useful to predict a set of dependent variables from a very large set of independent variables.

A New Approach in University Evaluation: the Balanced Scorecard A BSC for University: further developments the introduction of a BSC concerning the whole Italian Universities System (using mean or median of single Universities’ indicators with also standard deviation). It could be a useful tool for the Italian Ministry of Higher Education; a standardized instrument of benchmarking among Italian Universities; after measuring and managing performances, an instrument to evaluate them, either at the whole University level or at Faculty or Department level; to apply a structural equation modeling in a rationalization tentative of BSC (Morard & Stancu, 2005)