1. Measuring reputation in Testbeds Chrysa Papagianni, Symeon Papavassiliou National Technical University of Athens

2. Outline Measuring Testbeds’ Reputation FTUE Framework
Motivation and Objectives
Federated Trust and User Experience framework
FTUE Framework
Overview
FTUE Framework: Evaluation and Results
Integration into the Fed4FIRE federation and Information Flow

3. Motivation and Objectives
Trust is the subjective probability by which an entity, A, expects that another entity, B, performs a given action1
Trust and Reputation has been widely used in P2P network
Reputation based Trust Management Systems
Measuring testbeds’ reputation
A Reputation Based Trust Management system can help experimenters select resources based on a testbeds’ reputation
Entities rate each other
Opinions aggregated
Reputation Score
The term trust trust has been used in various contexts in literature, however, one prevailing definition is the one from Gambetta that contributes to the notion of treating trust as an opinion, an evaluation and thus a belief. Hence it can also be defined from a malicious point if view. Trust and Reputation has been widely used in P2P networks. In this context Reputation-based systems are a sub-category of trust management systems. Reputation-based P2P systems were introduced to build trust among peers. These systems try to evaluate the peers and associate a reputation value to each one.
The main idea is to let different entities rate each other, after the completion of an interaction, collect the feedback for each entity, aggregate the provided information in a distributed or centralized manner, and produce a reputation score, which can assist other entities in future trust decisions
In such systems, we assume service providers and provided services are not trusted. Service requesters select service providers based on their reputation values thus reputable service providers are selected to provide the service.
1 D. Gambetta, ”Can we trust trust?”, in Trust: Making and Breaking Cooperative Relations, D. Gambetta, Basil Blackwell, 2000, pp

4. Federated Trust and User Experience (FTUE)
But how to build reputation scores for testbeds?
Federated Trust & User Experience (FTUE) Framework1
Subjective: Experimenters’ QoE
Objective: Monitoring information
What is a testbed service?
Non technical/technical services x
Experimenters evaluate federated testbeds’ services
Added value service for testbed owners and users
Empowering users to select resources
1 Kapoukakis, A.; Kafetzoglou, S.; Androulidakis, G.; Papagianni, C.; Papavassiliou, S., "Reputation- Based Trust in federated testbeds utilizing user experience," Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), 2014 IEEE 19th International Workshop on , vol., no., pp.56,60, 1-3 Dec. 2014

5. FTUE – Framework Overview (1)
User’s Credibility is updated - comparison between the opinion and monitoring data
Opinions are weighted with Credibility and Quality values and aggregated to form the Reputation Scores
In F4F, testbeds may advertise aggregated services via the RS, where an aggregated service is a technical or non technical services pertaining one or more testbeds. For example node availability is a technical aggregated service. A non-technical service corresponds to the non-quantifiable experience of the user on conducting an experiment such as the Overall Experience of the user conducting the experiment.
The experimenter, upon completing an experiment provides feedback (an Opinion) regarding the quality of experience in using the aggregated service. The ratings of the experimenters are given in terms of a Mean Opinion Score (MOS). The framework delegates each service opinion to the respective testbeds that are used in the experiment and advertise the particular aggregated service.
After collecting the provided feedback, each allocated opinion is compared with the corresponding monitoring data, as retrieved from the monitoring infrastructure. Depending on the outcome and whether there is a match between the opinion and the monitoring data, the credibility of the user is accordingly altered.
Opinions from users are weighted with the Credibility and Quality values and then aggregated to form the reputation scores for each aggregated service of each testbed
Experimenter provides QoE feedback (Opinion) for the Service
Services Advertised via the Reputation Service
e.g. SA1 is service of type S1 for tesbed A

6. FTUE – Framework Overview (2)
Scenario 1
Opinions match the monitoring data
Truthful
Increase Credibility
Scenario 2
Opinions differ from the monitoring data for every testbed
Malicious
Decrease
Credibility
Scenario 3
Conservative Opinions and high or low quality of service
Truthful –
Moderate
Increase Credibility
Scenario 4
At least one testbed has different behavior
Truthful or malicious in disguise
Increase Credibility

7. FTUE: Performance Evaluation
Goal: Adaptability of our framework in changing conditions.
Simulation Setup
100 Users/ 10 experiments each
2-4 Testbeds in the federation (A to D)
Testbeds advertise 1 technical Service and 1 non technical service e.g. Overall Experience
User Opinions / Monitoring Data: Uniformly Distributed [0,1] based on
1-500 experiments: Smooth Operation - 80% Truthful /20% Malicious
experiments: Technical Problems for Testbed B - 20% Truthful (Moderate) /80% Truthful or Malicious in disguise
User Classes
Scenario I
Truthful
Scenario II
Malicious
Scenario III
Truthful (moderate)
Scenario IV
Truthful or
Malicious in disguise

8. Performance Evaluation: Adaptability in changing conditions
Successful constraint of malicious users
Quick Adaptation

9. Reputation Service in Fed4FIRE
Experimenter Tools
Experimenter Tools
Ruby based implementation
Reputation Service Repository
REST
Reputation Service
Retrieve Reputation Scores for each testbed service from RS
Provide Ratings/Opinion and Quality for experiment
Reputation Service
Retrieve Monitoring data for experiment
XML-RPC / REST
Manifold
Data Broker
Update Credibility Values
Update Reputation Scores
OML server
SFA
OML
Aggregate Manager
REST
Testbed Monitoring

10. Thank you!
Questions: