1. Large Datasets and Path Mapping in Tableau A V Flinsch Information Visualization Fall 2015

2. WSPRnet High frequency radio wave propagation statistics Shows current data only Data can be visualized at the wsprnet.org website, but it is limited for a 24 hour period. I wanted to see what happened in the past…

3. Data Availability Old data is archived Monthly files Large datasets 5-150 Mb zipped Around 750Mb unzipped 5M+ rows/month Too large for Excel

4. Data Issues Transmitting and receiving locations are in an unusual format Maidenhead gridsquares are a shorthand method for representing latitude/longitude pairs

5. Process Download data Load into MySQL table Convert grid squares to latitude/longitude pairs

6. Getting the data Create a shell script to – Download file –Unzip –Load to MySQL table

7. Data Conversion CPAN saves the day –Perl module to convert grids from/to lat/lon Problem ….

8. Data Conversion CPAN saves the day –Perl module to convert grids from/to lat/lon Problem …. –I don’t know Perl....

9. Hack it or Fake it Most languages are similar enough that I can figure something out Create a conversion table that will allow me to translate the grids to lat/lon pairs

10. Hacking it… #!/opt/local/bin/perl use Ham::Locator; my $locator = new Ham::Locator; $f1="A"; $f2="A"; for ($a=0; $a<18; $a++) { for ($b=0; $b<18; $b++) { for ($c=0; $c<10; $c++) { for ($d=0; $d<10; $d++) { # get lat and long here $grid="$f1$f2$c$d"; $locator->set_loc($grid); my ($latitude, $longitude) = $locator->loc2latlng; print ("$grid,$latitude,$longitude \n"); } $f2++; } $f1++; $f2="A"; }

11. More MySQL Create a table of all possible grid squares Update the wspr data with lat/long pairs for transmitting and receiving stations

12. Simple Update update spots s set s.tx_longitude = (select m.longitude from maidenhead m where m.grid_square = left(s.tx_grid,4)), s.tx_latitude = (select m.latitude from maidenhead m where m.grid_square = left(s.tx_grid,4)), s.rx_longitude = (select m.longitude from maidenhead m where m.grid_square = left(s.rx_grid,4)), s.rx_latitude = (select m.latitude from maidenhead m where m.grid_square = left(s.rx_grid,4));

13. Next Problem Creating a path in Tableau requires plotting individual points At minimum a start and end are required

14. Even more MySQL Split each row into individual pairs Easily done by creating a view

15. Even more MySQL Split each row into individual pairs Easily done by creating a view 2 rows/ entry now means that I have effectively doubled the data

16. Step 1 Open Tableau

17. Step 2 Connect to MySQL in Tableau

18. Step 3 Select database view and other tables

19. Step 4 Create some filters

20. Step 5 Create a map

21. Step 6 Connect the dots

22. Step 7 Define the path

23. Step 8 Set start and end points

24. Step 9 Give the paths some size

25. Final View Add an additional filter on TX station – AB2RC Set a date, band and time