1. Validation of an Automated Gating Pipeline for ICS Assays in Vaccine Trials
Greg Finak, Ph.D., Senior Staff Scientist
Marylou Ingram ISAC Scholar
Vaccine and Infectious Disease Division
Fred Hutchinson Cancer Research Center

2. Flow Cytometry At the Fred Hutch
Intracellular Cytokine Staining Assay is the bread- and-butter assay for the HVTN.
1000s of patient samples / year.
Manual analysis and current tools limit the number of cell populations that can be explored.
10 colors
12 colors
8 colors
17 colors
27 colors
2002
Today
In development
CD4 / CD8
T-cells &3 cytokines
TfH, Memory,
NK & 7 cytokines
16 colors

3. Challenges for automated gating
Reproducibility
Validation
Ease of use

4. OpenCyto General framework for reproducible automated analysis.
Built around R/BioConductor.
Automated hierarchical analysis.
Flexible and powerful but…
Requires expert knowledge (R, programming, statistics, machine learning)
Makes it difficult to use for the uninitiated.
Finak G, et al. PLoS Comput. Biol. (2014)

5. Cell Population Definition
OpenCyto: How it works
Abstract Gating Template
Gating Algorithms
(12+ available)
Pick one!
Tuning Parameters
Pick these too!
Cell Population Definition
Name
Parent
Channels
+ or -
Import, compensate, transform, automated gating, summarize and export
Raw FCS files
Summary Tables, Plots, Analysis Reports

6. Choosing algorithms and tuning parameters requires
OpenCyto: How it works
Gating Algorithms
(12+ available)
Pick one!
Tuning Parameters
Pick these too!
Choosing algorithms and tuning parameters requires
translating expert knowledge from one domain to another.

7. If you’re not a computational person…
How to use automated gating tools

8. OpenCyto: a simple example
Cytokine gating routine
Three tuning parameters.
Affect the position of the gate
Defaults are not always optimal.
Manual tuning not feasible for hundreds of gates and samples.

9. Learn Gating Parameters from Data
Manual gates are a great source of expert knowledge.
Scan through each manual gate.
Heuristics guess at the type of gate.
Short-list algorithms.
Optimize tuning parameters to maximize average accuracy, F-measure, relative to training data.
Choose algorithm with best performance

10. Inference of OpenCyto Templates from Manual Gates
Build templates for hundreds of cell populations.
Push the problem of QC downstream post-gating (where it’s much easier).
Examine gate thresholds / population statistics across samples to identify and correct outliers automatically.
Gate Imputation
Look for “nearest” sample that passed QC
Copy its gate to the outlier.
QC at each gate.
Prevent errors from propagating through gating tree.
Gates are still data-driven. Use data to pick gating algorithm and tuning parameters.
Need relatively few gated samples (as few as 15).

11. Some Results

12. HVTN Trial ICS Data Study of 82 subjects in 3 treatment groups.
5 stimulations (4 HIV proteins, 1 negative control - 2x replicate).
384 samples.
8.2 Gb of data.
165 manually gated cell subsets.
Data processed with openCyto
Learned template from 15 samples.
Gated remaining data using the template.

13. Estimated Bandwidth Parameters for Cytokine Gates

14. Trained gates have reasonable accuracy down to tree depth 7
F-measure as a metric of training success.
Rare cell populations: smaller F-measure doesn’t necessarily mean gate is wrong.
Sensitive for small populations.

15. How do the gates compare?
We are dealing with exceedingly rare cell subsets

16. Training: Cell population statistics are consistent with manual gates
Tells us which populations should be trusted and which ones should be carefully reviewed (or the template edited by hand).

17. Training: Cell Counts are consistent with manual gates

18. Test Set Results on (N = 379)
56,092 cell populations
Absolute error << relative error
Relative error in rare cell subsets
Does it matter?

19. Key Question: Do the differences affect inference?
Fit a linear model to the background-adjusted proportions for each of the 408 (6 cytokines x 4 stims x 17 parents) cell populations.
Assess difference in magnitude between treatment arms.
Compare effect size between manual and automated gating.

20. Consistent Inference between Automated an Manually Gated Data
Excellent agreement
20 cell subsets / stimulations differ between treatment arms.

21. Visualize Background Adjusted Cell Population Proportions

22. Summary and ongoing work
New tool for learning openCyto templates from manual gates
Few examples needed for training
Good agreement with expert manual gates
Performs well for rare cell subsets
Ongoing
Continue validation on retrospective data sets
Implement as a regular pipeline component.
Standardize interface and release R package.

23. OpenCyto Command Line Tools
A command-line interface to OpenCyto Beta release Do simple things well… Import FJ workspaces process FCS files (compensate, transform, annotate) gate data using openCyto templates export population statistics

24. Acknowledgements RGLab HVTN VISC FlowJo VRC Raphael Gottardo
Steve De Rosa
Mike Jiang
Julie McElrath
Phu Van
Peter Gilbert
Evan Greene
Kirsten Cohen
VISC
FlowJo
Paul Obrecht
Mike Stadnisky
Jay Almarode
VRC
Mario Roederer
Kathy Foulds