Large Datasets and Path Mapping in Tableau A V Flinsch Information Visualization Fall 2015

WSPRnet High frequency radio wave propagation statistics Shows current data only Data can be visualized at the wsprnet.org website, but it is limited for a 24 hour period. I wanted to see what happened in the past…

Data Availability Old data is archived Monthly files Large datasets Mb zipped Around 750Mb unzipped 5M+ rows/month Too large for Excel

Data Issues Transmitting and receiving locations are in an unusual format Maidenhead gridsquares are a shorthand method for representing latitude/longitude pairs

Process Download data Load into MySQL table Convert grid squares to latitude/longitude pairs

Getting the data Create a shell script to – Download file –Unzip –Load to MySQL table

Data Conversion CPAN saves the day –Perl module to convert grids from/to lat/lon Problem ….

Data Conversion CPAN saves the day –Perl module to convert grids from/to lat/lon Problem …. –I don’t know Perl....

Hack it or Fake it Most languages are similar enough that I can figure something out Create a conversion table that will allow me to translate the grids to lat/lon pairs

Hacking it… #!/opt/local/bin/perl use Ham::Locator; my $locator = new Ham::Locator; $f1="A"; $f2="A"; for ($a=0; $a<18; $a++) { for ($b=0; $b<18; $b++) { for ($c=0; $c<10; $c++) { for ($d=0; $d<10; $d++) { # get lat and long here $grid="$f1$f2$c$d"; $locator->set_loc($grid); my ($latitude, $longitude) = $locator->loc2latlng; print ("$grid,$latitude,$longitude \n"); } $f2++; } $f1++; $f2="A"; }

More MySQL Create a table of all possible grid squares Update the wspr data with lat/long pairs for transmitting and receiving stations

Simple Update update spots s set s.tx_longitude = (select m.longitude from maidenhead m where m.grid_square = left(s.tx_grid,4)), s.tx_latitude = (select m.latitude from maidenhead m where m.grid_square = left(s.tx_grid,4)), s.rx_longitude = (select m.longitude from maidenhead m where m.grid_square = left(s.rx_grid,4)), s.rx_latitude = (select m.latitude from maidenhead m where m.grid_square = left(s.rx_grid,4));

Next Problem Creating a path in Tableau requires plotting individual points At minimum a start and end are required

Even more MySQL Split each row into individual pairs Easily done by creating a view

Even more MySQL Split each row into individual pairs Easily done by creating a view 2 rows/ entry now means that I have effectively doubled the data

Step 1 Open Tableau

Step 2 Connect to MySQL in Tableau

Step 3 Select database view and other tables

Step 4 Create some filters

Step 5 Create a map

Step 6 Connect the dots

Step 7 Define the path

Step 8 Set start and end points

Step 9 Give the paths some size

Final View Add an additional filter on TX station – AB2RC Set a date, band and time