1. Psychopathology Network Analysis Workshop
Sacha Epskamp
Eiko Fried
Department of Psychological Research Methods
University of Amsterdam
Utrecht University, October

5. Network Replicability crisis ?

6. Network stability
Why could replicability be an issue in psychopathological network research?
Let's write a brief network paper together to find out!
Data: 180 women with PTSD diagnosis, 17-item screener
Data from DOI /a , freely available at

7. Dataset 1
Strong positive edge: 5-11, 3-4, 16-17
Negative edge: 10-12

8. Dataset 1
Strong positive edge: 5-11, 3-4, 16-17
Negative edge: 10-12

9. Dataset 1 Paper Strong positive connections between 3—4, 5—11, 16—17
Strong negative edge between 10—12
Most central nodes: 3, 16, 17 ⟶ consider as targets in intervention study

10. Dataset 1
Paper published … partytime!

11. Dataset 2
Now imagine we find another dataset, same sample size, female PTSD patients
First dataset, n=180
Second dataset, n=179

12. Dataset 2
Now imagine we find another dataset, same sample size, female PTSD patients
First dataset, n=180
Second dataset, n=179

13. Dataset 2
First dataset, n=180
Second dataset, n=179

14. Dataset 2
First dataset, n=180
Second dataset, n=179

15. Network stability
To avoid a replicability crisis, we need to investigate and report how accurate & stable our parameter estimates are
Especially relevant because our research may have clinical implications for thousands of patients
E.g.: what are the most central symptoms that ought to be treated?

16. Network stability Two main questions: Stability of edge weights
Stability of centrality indices

17. Network stability Two main questions:
Is edge 3—4 meaningfully larger than edge 3—11?
Is node 17 substantially more central than node 16?

18. R-package bootnet

19. edge weight stability

20. Boostrapping edge weights
Is edge 3—4 (0.42) stronger than edge 3—11 (0.14)?
Obtain CI by bootstrapping
Predictions?

21. Edges
Edge weights

22. 3—4
3—11
0.42
0.14
Edges
Edge weights

23. 3—4
3—11
0.42
0.14
Edges
Edge weights

24. 3—4
3—11
0.42
Edges
0.06
Edge weights

25. 3—4
3—11
0.42
Edges
0.06
Most edges are not meaningfully different from each other because their CIs overlap.
This is not really surprising: we are estimating 136 edge parameters with only 180 observations.
Edge weights

26. Centrality stability

27. Subset bootstrap
We now want to understand how stable the estimation of centrality indices is: e.g., is centralty of node 17 (1.16) substantially higher than the centrality of node 16 (0.99)

28. Subset bootstrap
Unfortunately, bootstrapping CIs around centrality estimates is not possible
Costenbader, E., & Valente, T. W. (2003) DOI: /S (03)

29. Subset bootstrap
Obtain centrality for data (s17 > s3 > s16...)

30. Subset bootstrap Obtain centrality for data (s17 > s3 > s16...)
Subset data by dropping 10% of the people
Obtain centrality for -10% subset (s17 > s7 > s4...)

31. Subset bootstrap Obtain centrality for data (s17 > s3 > s16...)
Subset data by dropping 10% of the people
Obtain centrality for -10% subset (s17 > s7 > s4...)
Subset data by dropping 20% of the people
Obtain centrality for -20% subset (s16 > s7 > s3...)
...

32. Subset bootstrap So what we get is centrality for
Full data (s17 > s3 > s16...)
N -10% data (s17 > s7 > s4...)
N -20% data (s16 > s7 > s3...)
N -30% data (s17 > s3 > s16...)
N -40% data (s17 > s3 > s16...)
N -50% data (s16 > s3 > s7...)
N -60% data (s17 > s3 > s7...)
N -70% data (s17 > s3 > s16...)
N -80% data (s3 > s6 > s17...)
N -90% data (s7 > s3 > s16...)

33. Subset bootstrap

34. Subset bootstrap We can also subset nodes instead of people
95 quantile range of the estimated centralities in the subset bootstraps

35. Take home message
For most statistical parameters or test statistics, it is very useful to understand how precisely they are estimated
Different ways to do that, one way is to bootstrap confidence intervals around the point estimates
Investigating the stability of network parameters like edge weights will help us to understand how likely our networks generalize
bootnet is a very first & preliminary step: develop your own methods that help the field investigate the stability of networks

36. Take home message
Stability also helps guide the question how many observations we need to obtain stable networks
n=180 for k=17
n=3812 for k=10

37. Eiko Fried Sacha Epskamp
Sacha Epskamp
Department of Psychological Research Methods
University of Amsterdam

38. https://arxiv.org/abs/1605.09288

40. Latent Network Modeling

42. Residual Network Modeling

44. BFI Example

45. BFI Example

46. https://arxiv.org/abs/1510.06871

48. Time-varying Networks

50. https://arxiv.org/abs/1604.08045