1. 851-0585-04L – Modeling and Simulating Social Systems with MATLAB
L – Modeling and Simulating Social Systems with MATLAB
Lecture 10 – Statistical Tests, Scientific Writing and Presentation of Results
Karsten Donnay and Stefano Balietti
Chair of Sociology, in particular of Modeling and Simulation
© ETH Zürich |
© ETH Zürich |

2. Schedule of the course Introduction to MATLAB
Schedule of the course
Introduction to MATLAB
20.02.
27.02.
05.03.
12.03.
19.03.
26.03.
02.04.
23.04.
30.04.
07.05.
14.05.
21.05.
28.05.
Introduction to social-science modeling and simulations
Working on projects (seminar thesis)
Handing in seminar thesis and giving a presentation

3. Final presentation schedule
Project presentation 15’ + 5’ (for Q&A)
2 days available:
Tuesday, May 29th
Wednesday, May 30th
Registration for final presentation is binding; if you do not want to obtain credits, do not register!
Sign up for slots begins next week in class

4. Final presentation schedule
Reminder:
Your reports are due midnight of Friday May 25th 2012
Submission is through GIT:
simply upload your report, source code, videos etc. before the deadline and we will retrieve it

5. Goals of Lecture 10: students will
Goals of Lecture 10: students will
Refresh knowledge on optimization in MATLAB acquired in lecture 9, through a brief repetition of important points.
Understand the importance and use of statistical tests when comparing your model to empirical data.
Receive an introduction to scientific writing* and practical advice on the framing of the final report.
Get some specific instructions on the expected format of the final report and a content “checklist” to follow when writing up the report.
See an example of the level of sophistication we expect in visualizing your model results.
* thanks to Heiko Rauhut and Volker Grimm for providing content

6. Repetition
the structure and simplicity of your code is the most important factor for the performance of your program
make use of the MATLAB specific techniques to improve performance when writing routines; use functions over scripts and compartmentalize
test your program for performance, in particular the parts of your code that carry the main computational load; MATLAB’s profiler is very useful for that!
example of vectorizing: can be faster than looping but not necessarily; the only way to know is to test your program under the expected load of the simulation.
parallelization very easy and useful for parameter sweeps

7. Statistical Tests When do we need statistical tests?
Quantify the agreement between simulated and empirical data
How similar are a simulated distribution and the corresponding empirical distribution?
Can we be “certain” that we have reproduced a stylized fact of the empirical data?
Optimizing your model for empirical data
Identify model parameters for which the agreement with empirical data is maximal
Statistical tests tell us how sure we can be that our simulation reproduces an empirical observation!

8. Statistical Tests How do we use them?
Identify a quantity that is meaningful and can be both observed in the simulation and the data
Record this data in the simulation and bring it in a distributional form for which it can be compared
Perform an APPROPRIATE statistical test to establish the level of agreement
Note: it is often difficult to identify a meaningful quantity AND choose an appropriate test for comparison!

9. Statistical Tests Example: Power Law
Statistical Tests
Example: Power Law
The occurrence of inter-state wars as a function of their size (casualties) follows a power-law
“war size” in a model of inter-state wars should follow the same distribution!
appropriate test here:
Kolmogorov-Smirnov (KS) test
(not R2 test!)
Cederman (2003). “Modeling the Size of Wars: From Billiard Balls to Sandpiles”. APSR 97(1):

10. Statistical Tests Tests in MATLAB:
Statistical Tests
Tests in MATLAB:
Many hypothesis tests already efficiently implemented:
Pearson’s correlation: corr()
(as options: Spearman and Kendall)
Kolmogorov-Smirnov test: kstest() and kstest2() for one-sample and two-sample test respectively
t-test: ttest()
A comprehensive list of all hypothesis tests implemented in MATLAB may be found at l

11. Statistical Tests Are statistical tests important for your projects?
If you are working with empirical data you may want to consider using a statistical test to quantify your agreement with the data…
… it is certainly not mandatory, though!
Please get in touch with us if you are interested in model optimization/validation with empirical data.

12. Scientific Writing Clear, easy to read, concise, and precise.
avoid metaphors; use examples instead
keep sentences short
Main purpose: communicate your work so that others can replicate it, modify it and object to it.
It is still a form of creative writing!
create a narrative
guide the reader through your project and results

13. The Hour Glass Structure

14. What are your research questions?
In the introduction of your reports, clearly specify the research questions, e.g.:
How does crime influence population growth?
Will the outcome change if mechanism A is changed to mechanism B in the model? How?
Are the results of the model reproducible if the complicated function f(x)=exp(x+y)+y2 is replaced by the simpler g(x)=x+y ?
In the “Results” section come back to your research questions.

15. Interpretation of Results
Interpretation of Results
Remember: Correlation does not imply causation
Do not over-interpret your results
Propose an underlying mechanism
xkcd.com/552

16. Correlation vs. Causality
Correlation: corrcoef(A,B) =
Causality: A correlates with B, but how does the causality look like? A C A B B A B A B

17. Reproducibility of Results
It is an important criterion for the quality of scientific work that your results are fully reproducible
Make sure to set and store the random seed of every simulation run to ensure reproducibility
STREAM = RandStream.getDefaultStream returns default random number stream with its seed value
RandStream.setDefaultStream(RandStream('mt19 937ar','seed',SeedValue) sets the random seed with the (new) value SeedValue

18. Data Dump – storing your results
Make sure that the results of your simulations are stored at the end of a simulation run
You can for example simply save the results in form of the array structure (matrix, cell array) of your simulation using MATLAB’s save() command
Or you may export them to comma separated files using csvwrite() (and csvread() to recover data)
Documentation on csvwrite() may be found here

19. Visualization of Results
Visualization of Results
Graphical formats: use PDF or EPS in the print version. JPG (better PNG) just for quick visualizations
Increase the font sizes for readability: set(gca, ‘FontSize’, 16);
ALWAYS use labels on axes: xlabel(‘Time (s)’, ‘FontSize’, 16); ylabel(‘Number of Agents’, ‘FontSize’, 16);

20. Visualization of Results (2)
Visualization of Results (2)
Put a box around the figure: box on
Show grid: grid on
Use thicker lines: plot(x, y, ‘LineWidth’, 2);
Set axis limits: xlim(x0, x0); ylim(y0, y0);

21.
Example
As an example: Let us say that we have a project about simulating happiness of agents.
We have the result of 10 different simulation runs in y, and the corresponding time vector in x.
x = t0:dt:t1; y = [happiness values from simulation 1; happiness values from simulation 2; happiness values from simulation 10];

22.
plot(x,y(1,:))

23. xlabel(‘Time(days)’) ylabel(‘Fraction of happy agents’)
plot(x,y(1,:))
xlabel(‘Time(days)’)
ylabel(‘Fraction of happy agents’)

24. xlabel(‘Time(days)’) ylabel(‘Fraction of happy agents’)
plot(x,y)
xlabel(‘Time(days)’)
ylabel(‘Fraction of happy agents’)

25. xlabel(‘Time(days)’) ylabel(‘Fraction of happy agents’)
plot(x,y)
set(gca,’FontSize’,16)
xlabel(‘Time(days)’)
ylabel(‘Fraction of happy agents’)

26. errorbar(x,mean(y),std(y)) set(gca,’FontSize’,16) xlabel(‘Time(days)’)
errorbar(x,mean(y),std(y))
set(gca,’FontSize’,16)
xlabel(‘Time(days)’)
ylabel(‘Fraction of happy agents’)

27. errorbar(x,mean(y),std(y),’k’,’LineWidth’,2) set(gca,’FontSize’,16)
errorbar(x,mean(y),std(y),’k’,’LineWidth’,2)
set(gca,’FontSize’,16)
xlabel(‘Time(days)’)
ylabel(‘Fraction of happy agents’)
xlim([0 10])
ylim([0 .5])

28. add caption in LaTeX! errorbar(x,mean(y),std(y),’k’,’LineWidth’,2)
set(gca,’FontSize’,16)
xlabel(‘Time(days)’)
ylabel(‘Fraction of happy agents’)
xlim([0 10])
ylim([0 .5])
add caption in LaTeX!

29. 05.07.2018 errorbar(x,mean(y),std(y),’k’,’LineWidth’,2)
set(gca,’FontSize’,16)
xlabel(‘Time(days)’)
ylabel(‘Fraction of happy agents’)
xlim([0 10])
ylim([0 .5])

30. Important Graphs semi-logarithmic plot (semilogx() or semilogy())
Important Graphs
semi-logarithmic plot (semilogx() or semilogy())
α =
y = exp(α x)
α sets the slope of the curve

31. Important Graphs power law plot (loglog()) y = xα
Important Graphs
power law plot (loglog())
α = -1.2
y = xα
α sets the slope of the curve

32. Important Graphs power law plot with exponential cutoff (loglog())
Important Graphs
power law plot with exponential cutoff (loglog())
α = -1.2
y = xα exp(β x)
α sets the slope of the curve
β sets the exponential cutoff

33. Phase Diagrams
To show how the result depend on two different parameters, phase diagrams can be used:
D. Helbing and W. Yu, PNAS (2009)

34. Phase Diagrams
code snippet that uses surf(); here matrix M has as entry (i,j) the model fit for every combination of two parameters (here: a, b)
[a,b]=meshgrid(0:0.01:1);
hold on;
view(0,90);
surf(a,b,M,'EdgeColor','none');
colorbar;
set(gca,'FontSize',14)
xlabel('parameter a');
ylabel('parameter b');

35. Phase Diagrams

36. Normalized Parameters
It is usually advantageous to rescale any parameter in your model to a uniform range, for example [0,1]
Normalized parameters simplify the visualization and analysis of the model results
Ideally, the parameter range of model parameters and their scaling to the uniform range is chosen such that all relevant variations of the model are visible within that range

37.
Create Videos
It is helpful to illustrate your simulations with videos during your presentation:

38.
Project Report
Have a look at last year’s projects to make sure that you do not repeat the same results!
Use the report template.
Try to explain your results and your motivation behind the project.
Include your MATLAB code in the appendix
If you have any problems or concerns with your projects, we are here to help you!

39.
Project Report
There are a number of things that will certainly negatively affect your grade:
no clearly stated research question
no conclusion regarding the work you have completed and how it relates to your research question
graphs which have no legend, axis labels or caption, are not properly readable or the labels are too small
no references to the literature to validate and back-up your statements/model etc.
submitting you report late or incomplete

40. Project Report References are very important:
Project Report
References are very important:
cite the sources you are using
be aware that plagiarism is taken very seriously at ETH, if you are unsure how to cite properly follow ETH’s plagiarism guidelines
we accept both natural science and social science citation conventions but please use them consistently
it is important that your references are correct and correctly formatted!

41. Project Report Documentation of your code is important
Project Report
Documentation of your code is important
You can, for example, use the following tool to automatically generate html files from comments in your code
Please also pay attention to the MATLAB code style guidelines provided here to guarantee that your code is flawless and easy to understand

42. Final checklist A typical scientific report should include: Title
Abstract
Introduction
Materials / Methods
Results
Discussion
References
Appendix (if needed)
ReportTemplate.zip

43. References MATLAB documentation on plotting
MATLAB code style guidelines
Tool to convert comments to html files
J.-L. Dumont. “Trees, maps and theorems”. Principiae (2009).
Russey, Ebel, Bliefert. “How to Write a Successful Science Thesis”. Wiley (2006).
R. Murray. “How to Write a Thesis”. Open University Press (2006)