1 Is Europe Making a difference: Assessing Impact in Information Society Technologies Isabelle Collins Technopolis Ltd

2 European Research Landscape

3

4 History of IST Research Programmes

5

6 The IST programme €1 billion per annum 23 Operational domains Basic Research, Components, Systems, Applications Health, Public Services, Inclusion, Business, Learning, etc

7 Evolution of the IST programme

8 Methodological Development Key performance indicators Logic models and causality Impact Assessment

9 Key performance indicators Main tool - objectives hierarchy

10 Main issues and outcomes Confusion between data and indicators Connecting indicators to objectives Outcome - set of data being collected on 4 factors: –Patents –Publications –Thingy –Whatnot

11 Assessing causality Taking the objectives hierarchy one step further Constructing multi-level logic models Looking for causality chains

12 Key problems and results Missing middle Evaluation as archaeology No involvement or buy-in Aggregation

13 WING Assessing Impact –IST FP5 and increasingly FP6 –23 Domains and overall Programme –3 year rolling plan of work –More than 12 organisations in special domain teams

14 Levels of analysis Domain – This is where the brunt of the empirical work is located. Includes, if necessary or required, examination of specific work areas / sub-domain areas, especially if these have very different technological, market or social characteristics Horizontal – A much more limited proportion of the overall effort is dedicated to the exploration of methodological approaches designed to tackle cross cutting issues Aggregated – This includes the aggregation of domain level results and, if available, the results emerging from the horizontal studies

15 Study process

16 Methodology Questionnaire “TIMES” analysis Interviews & other data collection Survey team Domain Experts CENTRAL Quality Domain expert WING Mgt Report Synthesis Analysis Work final draft FINAL DOMAIN REPORT AGGREGATED REPORT HORIZONTAL ISSUES Media macro-economic impacts other horizontal themes etc as required Composition analysis

17 Key steps Clarify the objectives Analyse key changes in the domain technological environment Analyse changes in the domain socio-economic environment Identify, as far as possible, the links between the domain activity and these changes (attribute impact)

18 Clarify the objectives Make explicit the programme objectives from the strategic objectives of the Lisbon agenda, through the objectives of the FP to the specific objectives of the domain Put together hierarchy of objectives Identify the key planned impacts Identify the main external factors

19 Technology environment Describe state of the art at beginning of period (start of FP5) Describe state of the art today Identify likely key future developments Map domain research activities against this landscape

20 Economic and social environment Describe economic environment at beginning of period –Size, nature, structure, regulatory environment etc –Position of EU in world market Describe environment today Identify likely key future developments Map domain research activities and key players against this landscape

21 Impacts Scientific and technological impacts Economic impacts Policy and regulation impacts Social impacts Environmental impacts and Sustainability

22 Key methodological issues Programme related –Lack of clear objectives –Complexity Timing –Domain studies –Aggregate studies Management related –Client study management –Managing dispersed teams –Scientists v evaluators Aggregation –Indicators Attribution