1. Exploiting Unintended Feature Leakage in Collaborative Learning
Luca Melis∗ UCL
Congzheng Song∗
Cornell University
Emiliano De Cristofaro
UCL & Alan Turing Institute
Vitaly Shmatikov
Cornell Tech

2. Collaborative machine learning
Dataset 1
Participant 1
Model 1
Dataset 2
Participant 2
Model 2
Dataset 3
Participant 3
Model 3
Periodically exchange model parameters
Training data never leave participants’ machines

3. Collaborative machine learning
synchronized gradient updates

4. Collaborative machine learning
Federated learning with model averaging

5. Key idea Any useful ML model reveals something about the
population from which the training data was drawn
inferring “unintended” features
that hold for certain subsets of the training data

6. Inferences Membership Inference Passive Property Inference
Active Property Inference

7. Threat Model K participants (1 adversary, 1 target) Algorithm 1 K = 2
observes gradient updates computed on a single batch of the target’s data
K > 2
observes an aggregation of gradient updates from all other participants
Algorithm 2
the result of two-step aggregation:
(1) every participant aggregates the gradients computed on each local batch
(2) the server aggregates the updates from all participants.

8. Threat Model

9. Threat Model - Embedding layer Non-numeric Discrete Inputs Sparse
Low-dim vector representation
Treat embedding matrix as a parameter
Sparse gradient
Infer information from non-zero gradient

10. Membership Inference IN? - Interpretation data model - Importance
Disease record
- Implementation in

11. Membership Inference - Experiment (a) idea
Test Bag of Words (BoW) : the input to be inferred
Batch Bag of Words (BoW): the target’s data in each batch
subset
(b) dataset
Yelp-health : vocabulary containing 5,000 words
FourSquare: 30,000 locations
(c) result

12. Passive Property Inference
- Interpretation
Not necessarily in all class
Not necessarily related with training object
Detect properties in a single batch
Detect properties in a participant’s entire dataset
Bob’s photo -- gender classification – whether Alice also appears
whether people wear glasses
when a property appears
- Assumption
Data labeled:
- idea
generate aggregated updates based on the data with the property and updates based on the data without the property.
train a binary batch property classifier
and feeds it

13. Passive Property Inference
- idea

14. Single batch Property Inference
- Experiment
ex1

15. Single batch Property Inference
t-SNE projection of the features from different layers
- Experiment
ex1

16. Single batch Property Inference
- Experiment
ex2
Main task: review-score classification
Inference: specialty of doctors
ex3
Infer some people

17. Dynamic Property Occurrence Inference
determine if people in the image are of the same gender
infer whether and when a certain person appears in the other participant’s photos

18. Inference against well-generalized models
Main task: sentiment
Inference: infer authors’ gender
dataset: annually expanded
student-written essays and reviews
Truthful/Deceptive OR Positive/Negative
labeled with attributes of the author (gender, age, sexual orientation, region of origin, personality profile)
the document (timestamp, genre, topic, veracity, sentiment)

19. Active property inference
Let the main model learn separable representations for the data with and without the property.
adversary performs additional local computations and submits the resulting values into the collaborative learning protocol
Main task: gender classification
Inference: presence of ID 4

20. Multi-party experiments
A. Synchronized SGD

21. Multi-party experiments
B. Model averaging

22. Multi-party experiments
B. Model averaging

23. Defense
A. Sharing fewer gradients
B. Dimensionality reduction

24. Defense
C. Dropout
D. Participant-level differential privacy

25. Limitations A. Auxiliary data
More targeted inference attacks require specialized auxiliary data that may not be available
B. Number of participants
some federated-learning applications involve thousands or millions of users
C. Undetectable properties
It may not be possible to infer some properties from model updates.
D. Attribution of inferred properties
may not be able to attribute these inputs to a specific participant in multi-party scenarios

26. Thanks!