UGDIE PROJECT MEETING Bled September WP6 – Assessment and Evaluation Evaluation Planning  Draft Evaluation plan

UGDIE PROJECT MEETING Bled September WP6 – Evaluation planning Objectives of Evaluation planning: –To drive project activities towards the effective completion of the assessment process Instruments of Evaluation planning –The Evaluation Plan (Deliverable D12) Responsibilities of Evaluation planning –Users’ needs correctly addressed throughout the project –Proper assessment objectives and methods definition –Robustness and clearness of results at the end of the project

UGDIE PROJECT MEETING Bled September WP6 – Evaluation Plan (D12) The final EP (also referred to as “validation plan”) has to be issued before validation activities actually start, and should include all details defined for data collection, measurements, indicators, reference case etc. (in other words the assessment methods)  deadline for F-MAN Evaluation Plan: 04/03 The Draft EP must be issued much more earlier, as it is supposed to impact various project activities (test sites definition, key applications to be validated, main decision makers to be involved in validation, etc.)  (1 st ) F-MAN Draft Evaluation Plan: issued and delivered to all partners on 06/02

UGDIE PROJECT MEETING Bled September WP6 – Evaluation Plan (D12) Who is responsible for preparing (and issuing) the Evaluation Plan?  the Evaluation Team Who else is involved in the definition of the Evaluation Plan?  WP1 concerned partners  Determination of users needs  Identification of decision makers, assessment objectives, expected impacts, (reference case)  WP2 concerned partners  Determination of functional requirements  Definition of system architecture  WP3/WP4 concerned partners  Resources availability, validation methods applicability  WP5 concerned partners  Definition of verification sites  Definition of validation methods Briefly… all F-MAN partners (+ ROG members)

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan Contents:  Definition of users needs  Description of applications  Phases of development  Assessment objectives  Pre-assessment of expected impacts  Categories of assessments and evaluation methods  Study design

UGDIE PROJECT MEETING Bled September (to be defined) Draft Evaluation Plan Current status:  Definition of users needs (compliant with Deliverable D3)  Description of applications (compliant with Deliverable D5?) To be discussed  Categories of assessments and evaluation methods  Study design  Phases of development  Assessment objectives  Pre-assessment of expected impacts

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan Definition of users needs  Who is who in the rail freight transport process? Customers : final users of transport services (industries, SMEs) Order Managers : liaison with customers (belonging to COC, FF) Fleet Managers : responsible for providing solutions to customers orders (belonging to COC, …?) Maintenance Managers : responsible for wagons maintenance in co-ordination with FM (belonging to COC, …?) Train Operating Companies : public/private undertakings providing traction (TOCs are also main COCs) Infrastructure Managers : authorities or undertakings granting access to railway infrastructures

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan Definition of users needs  Summary of users needs Customers : reduced response and journey time, improved supervision systems Order Managers : improvement of information quality and availability Fleet Managers : improved availability, timeliness and quality of information on managed wagons; support to productivity control; user friendliness of new systems Maintenance Managers : improved maintenance planning/scheduling, enhanced diagnostic Train Operating Companies : better exploitation of owned rolling stock; same needs of FMs when TOCs are also COCs, + gradual and low cost implementation of new systems Infrastructure Managers : too many needs!… just contribute to exploit infrastructures with no additional costs or efforts for them!

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan Definition of users needs  Focusing on F-MAN targets F-MAN will address the needs of FMs and COCs –Because fleet management is the bottleneck in the process –Because the fulfilment of FMs and COCs needs provides solution (at least partial) to the needs of the other concerned actors thus, F-MAN targets are –Development of a Tracking&Tracing system –Development of a Data Processing and information management system granting distributed access to wagons information –Development of a rail cars Asset Management system –Definition of a system architecture based on the most suitable Information Society Technologies –Definition of a new ownership/operation scheme enabling maximum exploitation of the system (F-MAN Pool)

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan Description of applications  F-MAN will develop the following main applications: –Tracking and tracing of rail freight cars (TSM) –Data processing and pro active information on wagons availability (DPM) –Rail cars asset management (AMM)  For each application, key characteristics and elements for validation are presented: –Technologies and approach used –Functionality (according to CONVERGE Telematics function list) –Main decision makers (key to definition of assessment objectives) –Users groups within and outside the project (to be involved in application validation) –Verification site example

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan Phases of development  Scheduling of definition, development, verification and assessment phases for the applications to be validated :  Main concern: –To monitor activities in order to assure that verification will be carried out leaving enough time and resources (e.g. simulation hours, man hours, etc.) for validation

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan The Assessment Objectives  Defining Assessment Objectives –AOs determine what validation activities will aim at –AOs should reflect the criteria adopted by the relevant decision makers in making judgements and decisions determining the adoption and/or large scale implementation of a given application –AOs for decision makers are not necessarily the same as the application design objectives –The basis for the identification of F-MAN AOs is the analysis of users needs, as the most relevant decision makers concerned in F-MAN are users themselves (COCs, FMs, FF,…) example

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan Pre-assessment of expected impacts Preliminary assessment of the impacts expected from the adoption of F-MAN applications on their potential users, operators, indirect users and other relevant appraisal groups  Main goals: –To identify nature/type of potential benefits (or drawbacks) to be considered as objectives (or risks) for the applications; project results will be evaluated against such objectives –To identify operators, users and/or other appraisal groups to be targeted by validation activities (data collection, interviews, questionnaires, …) –To estimate the expected magnitude of applications impact on the relevant appraisal groups (to focus validation activities on the most significant to be assessed) –To determine if the project has the resources necessary for assessing any crucial impact Based on Assessment Objectives and pre-assessment of expected impacts, the impacts actually to be assessed through validation activities will be selected example

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan  Categories of assessment and evaluation methods  Categories of assessment:  Technical assessment (system performances, reliability, …)  Impact assessment (qualitative and quantitative assessment of expected impacts)  Users acceptance assessment (users’ opinions, preferences, willing to pay, …)  Costs/benefits analysis  Market assessment (demand/supply)  Financial assessment (initial and running costs, rate of return, payback period)  Evaluation Methods  Measurements & simulations  Comparison with reference case  Interviews and questionnairs  ….

UGDIE PROJECT MEETING Bled September CONCLUSIONS Read carefully the document Make remarks Integrate the missing topics Co-operate for next steps Think of the Evaluation Plan as a reference document throughout the project development

UGDIE PROJECT MEETING Bled September Draft Evaluation Plan  Study design  Detailed definition of assessment methodologies –Definition of indicators for performance (e.g. reliability, time to query results,, data traffic, … ?) and impact (e.g. productivity, user friendliness, … ?) –Definition of reference case (e.g. current values of the indicators) –Definition of data collection methodologies and measurement conditions –Statistical considerations –Measurement plan