Download presentation
Presentation is loading. Please wait.
Published byClaude Lloyd Modified over 9 years ago
1
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 1 Aggregates, output and other basic tools
2
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 2 Topics 1.Aggregates simple groupby 2.Output console CSV charts 3.Misc 1.Simulation - "init phase" 2.(periodic) Globals 3.Macros 4.Functions local variables
3
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 3 Output
4
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 4 count([condition]) count() count(age >= 18 and not gender) sum(expr[, filter=condition][, skip_na=True]) avg, std, min, max, percentile, median, gini avg(age) avg(age, filter=gender) gini(income, filter=age >= 18, skip_na=False) all(condition1[, filter=condition2]) any all(income >= 0, filter=age >= 18) any(age 120) Aggregates - Simple
5
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 5 groupby(expr1[, expr2, expr3,...] [, expr=expression] [, filter=filterexpression] [, percent=True], [, pvalues=possible_values]) group individuals by their value for the given expressions, compute an expression for each group groupby(agegroup, gender) groupby(gender) groupby(agegroup, gender, workstate, percent=True) groupby(agegroup, gender, filter=MARRIED) groupby(agegroup, gender, expr=avg(income)) groupby(agegroup, gender, expr=avg(income), filter=age >= 18) Aggregates - Groupby (aka “pivot table”) agegroup | gender | | | False | True | total 0 | 165 | 177 | 342 5 | 263 | 264 | 527 10 | 278 | 290 | 568 15 | 293 | 347 | 640 20 | 294 | 303 | 597 … | … | … | … 40 | 375 | 359 | 734 45 | 362 | 408 | 770 50 | 681 | 694 | 1375 60 | 529 | 532 | 1061 70 | 428 | 340 | 768 80 | 310 | 182 | 492 90 | 83 | 30 | 113 100 | 3 | 1 | 4 total | 5049 | 4951 | 10000
6
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 6 Output
7
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 7 show(expr1[, expr2, … ]) evaluates the expression(s) and shows the result show(groupby(age, gender, filter=age <= 10)) show(count(age >= 18)) show("not dead", count(not dead), "\n", "average age", avg(age, filter=not dead)) "skip_shows: True" simulation option => hide them all qshow(expr1[, expr2, … ]) evaluates the expression(s) and shows the result next to the expression(s) qshow(count(), avg(age), std(age)) Output - console
8
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 8 dump(expr1[, expr2, …, filter, missing, header]) creates table with the given expressions evaluated over some individuals of the dataset filter - take a subset of individuals missing - value to transform ‘nan’ values into (default: no transformation) header - True/False, whether or not to add column names show(dump(age, partner.age, gender, filter=id < 10)) id | age | partner.age | gender 0 | 27 | -1 | False 1 | 86 | 71 | False 2 | 16 | -1 | True 3 | 19 | -1 | False 4 | 27 | 21 | False 5 | 89 | 92 | True 6 | 59 | 61 | True 7 | 65 | 29 | False 8 | 38 | 35 | True 9 | 48 | 52 | True Output - dump
9
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 9 csv(expr1[, expr2, …, suffix, fname, mode]) write to CSV-files fname - (path and) name of file to save to suffix - equivalent to fname = '{entity}_{period}_{suffix}.csv' mode - 'w' (default) or 'a' (overwrite or append) csv(avg(age)) csv(period, avg(age), fname='avg_income.csv', mode=‘a’) csv(dump(age, gender), suffix='dump') csv(groupby(age, gender, filter=age <= 10)) Output - CSV
10
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 10 Longitudinal not (currently) supported Slow on large data usually a good idea to use groupby instead of plotting millions of points Arguments fname - (path and) name of file to save to if not specified, shows interactive viewer supports.png &.pdf suffix - equivalent to fname = '{entity}_{period}_{suffix}.png' colors: ['r', 'g', 'b'] (see matplotlib colors documentation) grid: True | False Output - charts - generic
11
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 11 bar(groupby(agegroup, eduach)) Output - charts - bar
12
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 12 plot(groupby(agegroup, expr=count(not gender)), 'o--', groupby(agegroup, expr=count(gender)), 's-.', groupby(agegroup), '*-', colors=['r', 'g', 'b']) Output - charts - plot
13
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 13 stackplot(groupby(age, eduach)) Output - charts - stackplot
14
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 14 pie(groupby(eduach), labels=['Lower secondary', 'Upper secondary', 'Tertiary'], explode=[0.1, 0, 0]) Output - charts - pie
15
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 15 boxplot(groupby(eduach, expr=age, filter=eduach != -1), notch=True) Output - charts - boxplot
16
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 16 - area: 3.1415 * (4 + 1.5 * children.count()) ** 2 - scatter(age, salary, c=eduach, s=area, grid=True) (dataset with 200 persons) Output - charts - scatter
17
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 17 Miscellaneous
18
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 18 simulation: init: - household: [init_region_id, household_composition] processes: - person: [ageing] […] start_period: 2016 periods: 2 Init execute the processes in start_period - 1 (here 2015) can be used to initialize variables, output files, … can be used to do static (parts of the) simulation init functions often contain "init" in their name Simulation - "init phase"
19
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 19 Building csv files progressively entities: person: processes: create_files: - csv('period', 'average', 'std dev', fname='age_aggregates.csv') output: - csv(period, avg(age), std(age), fname='age_aggregates.csv', mode='a') simulation: init: - person: [create_files] processes: - person: [output] Output with init phase example periodaveragestd dev 200242.037429.97504 200335.7295131.73392 200431.5097132.06869 200528.6107431.88439 200626.4910231.49843
20
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 20 entities: person: processes: retirement: - workstate: if(age >= 65, RETIRED, workstate) Globals variables that do not relate to any particular entity periodic globals can have a different value for each period in CAPITAL letters by convention (periodic) globals globals: periodic: # PERIOD is implicit - RETIREMENTAGE: float entities: person: processes: retirement: - workstate: if(age >= RETIREMENTAGE, RETIRED, workstate)
21
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 21 Macros (easier to read & maintain) Macros defined on entity level re-evaluated wherever they appear can contain any expression/value (but not other macros) in CAPITAL letters by convention processes: ageing: - age: age + 1 csv_output: - csv(period, count(age = RETIREMENTAGE) / count(age >= 15 and age < RETIREMENTAGE), fname='dependency_ratio.csv', mode='a') processes: ageing: - age: age + 1 csv_output: - csv(period, count(not ACTIVEAGE) / count(ACTIVEAGE), fname='dependency_ratio.csv', mode='a') person: fields: - age: int […] macros: ACTIVEAGE: (age >= 15) and (age < RETIREMENTAGE)
22
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 22 processes: ageing: - age: age + 1 - juniors: 5 * trunc(age / 5) - plus50: 10 * trunc(age / 10) - agegroup: if(age < 50, juniors, plus50) Local variables e.g. juniors, plus50 in the ageing function no need to declare them temporary and not stored: only available in the function they are defined Functions local variables
23
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 23 Exercises (demo03.yml)
24
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 24 1. Open demo03.yml in the editor 2. Open the input dataset (demo.h5) in ViTables (F9) check which fields we have for the person entity 3. Take one individual (e.g. id == 42) and check how he evolved during the simulation (simulation.h5) e.g. his age increased by one each year 4. In the input dataset (demo.h5), find all persons aged 65+ who are still working (workstate equals 1) use == to test for equality (as in a LIAM2 model) use parentheses around conditions in ViTables queries use "&" to combine conditions (instead of "and" in a LIAM2 model) e.g. (condition1) & (condition2) can usually be done using LIAM2 interactive console not in this case (cannot use fields not declared). Exercises - ViTables
25
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 25 5. initialize agegroup in 2015 and check the result with ViTables (F9) 6. simulate 20 years, instead of 2 in period 2020… 7. … display the number of persons by agegroup 8. … display the number of persons old enough to be retired by sex 9. … for persons old enough to be retired, display the percentage of each gender 10. … display the proportion of males by agegroup 11. … display a bar chart of those proportions Exercises - init and groupby
26
Autumn School Dynamic MSM16-18 November 2015 | L-Esch-sur-Alzette Slide 26 12. demo03.yml already creates "dependency_ratio.csv" add two columns to that file with the child dependency ratio (0-14 / 15-64) and the aged dependency ratio (65+ / 15-64) for each year 13. create a new csv file called "average_agediff.csv" with the difference of average age between men and women for each year Exercises - progressive csv files
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.