Big Data Using Big Data for Cultures and Communities Jeremy Reffin Simon Wibberley CASM, University of Sussex Carl Miller CASM, Demos July 2014

Overview Emergence of “Big Data” “Open Data” initiatives Open source tools and processor power Analytics culture in decision making

Overview Emergence of “Big Data” Environment becoming saturated with digital devices that record data Social lives becoming increasingly digital (e.g. 500 million tweets / day) Access to big data now widespread through internet

Overview “Open Data” initiatives Key indices / measures have long been collected but sequestered Access to key data now widespread through internet Culture of “what gets measured gets managed” led to collection of more indices / measures Political initiatives to make data more widely available (“open”) as a market tool

Overview Open source tools and processor power Sophisticated data processing and visualisation tools now in the hands of most users

Overview Analytics culture in decision making 1980s: rise of data-driven approach to decision-making in business management Trend increasingly influenced formation of public policy Now potentially useful data accessible to community-level initiatives

Overview Emergence of “Big Data” “Open Data” initiatives Open source tools and processor power Analytics culture in decision making SOURCES TOOLS

Sources Social MediaCommunity Information Local Government Central Government Corporate

Sources Social MediaCommunity Information Local Government Central Government Corporate Availability Subject Matter UbiquitousIsolated SocialFactual Focus GeneralSpecific

Sources Social MediaCommunity Information Local Government Central Government Corporate Availability Subject Matter Focus Degree of Structure Degree of Oversight UbiquitousIsolated SocialFactual GeneralSpecific UnstructuredStructured Not curatedOften curated

Sources Social MediaCommunity Information Local Government Central Government Corporate Availability Subject Matter Focus Degree of Structure Degree of Oversight High Volume Open Access UbiquitousIsolated SocialFactual GeneralSpecific UnstructuredStructured Not curatedOften curated DependsNoSometimes NoSometimes Rarely

Sources Social MediaCommunity Information Local Government Central Government Corporate Availability Subject Matter Focus Degree of Structure Degree of Oversight UbiquitousIsolated SocialFactual GeneralSpecific UnstructuredStructured Not curatedOften curated DependsNoSometimes NoSometimes Rarely High Volume Open Access

Tools Extract & process Analyse Visualise

Sources x Tools Social MediaCommunity Information Local Government Central Government Corporate Extract & process Analyse Visualise

Sources x Tools Social MediaCommunity Information Local Government Central Government Corporate Extract & process Analyse Visualise

Sources x Tools Social MediaCommunity Information Local Government Central Government Corporate Extract & process Analyse Visualise APIs / Downloadable data sets Analytics tools (Excel, Google Analytics)

Sources x Tools Social MediaCommunity Information Local Government Central Government Corporate Extract & process Analyse Visualise

Wordle Wordle is a tool for generating “word clouds” from text that you provide. Word clouds give greater prominence to words that appear more frequently in the source text The word clouds can be customised with different fonts, layouts, and color schemes See: wordle.net

Text is Beautiful This resource creates interactive word clouds, concept webs, and correlation wheels -In a concept web, the position of concepts matter; those more related appear near each other. Related concepts are grouped into themes, denoted by colour -A correlation wheel visualises concepts that are correlated with each other. Two concepts are correlated if they appear together in the text often and appear apart rarely See: textisbeautiful.net

Circos Circos extends the “correlation wheel” concept into much more sophisticated data visualization using a cicular layout. Originally designed for genomic data, it produces attractive graphics that are particularly effective for displaying pairwise interactions in general and ”flows”in particular See: circos.ca

Circos (continued) Circos is often used to convey complex data in a condensed format Circos is relatively simple to use but is not ‘point and click’ It requires comfort / familiarity with ‘scripting’ approaches See: circos.ca

Sources x Tools Social MediaCommunity Information Local Government Central Government Corporate Extract & process Analyse Visualise

gephi Gephi visualises graphs. A graph is a representation of a set of objects where there are pairwise relationships between some of these objects. Any network can be usefully represented as a graph – as can most data with structural relationships between elements Gephi sits on the interface between data visualisation and data analysis. It has sophisticated tools for exploring data and analysing, filtering, clustering, manipulating and exporting it See: gephi.github.io

gephi: example output

tableau Tableau provides sophisticated tools for analysis and visualisation of data stored in a spreadsheet or similar data format. It can create interactive See: tableausoftware.com

Sources x Tools Social MediaCommunity Information Local Government Central Government Corporate Extract & process Analyse Visualise

Integrated analysis: DataShine Provides interactive mapping visualisation of UK Census data Source: datashine.org.uk

Sources x Tools Social MediaCommunity Information Local Government Central Government Corporate Extract & process Analyse Visualise

Social MediaCommunity Information Local Government Central Government Corporate Extract & process Analyse Visualise Unstructured Not curated High Volume Open Access General Method51

Twitter Boolean term scraper Relevancy Classifier Pattern Classifier (1) Pattern Classifier (2) Output 1 Output 2 Extract & process Analyse Visualise Method51 is a framework for collecting, analysing, and understanding Twitter data sets It helps users to locate tweets relevant to a precise topic of interest (i.e. separate the ‘wheat from the chaff’) and to gain the best possible insight into what is being said about that topic This is achieved using chains of classifiers – devices for placing tweets into different categories based on the words that they contain