Data and Big Data in MPI David Austin and Gerald Rys Ministry for Primary Industries, 10 September 2016 To Land and Water, National Science Challenge Worksop on Big Data

What is Big Data Big data refers to datasets that are so large or complex that traditional data processing applications are inadequate to deal with them.  Challenges with big data include analysis, capture, data curation, search, sharing, storage, transfer, visualization, querying, updating and information privacy. Accuracy in big data may lead to more confident decision-making, and better decisions can result in greater operational efficiency, cost reduction and reduced risk. First research done in NZ commissioned by MoRST EEP Education and Employment policy and CC on Industry and Environment nearly 20 percent of papers ministers requested more evidence

Big Data implications for MPI and the Primary and Food Sectors Larger and more diverse datasets provide opportunities for MPI and the broader sector. These include: Exploring economic impact of local events and broader international trade. Improved food traceability (e.g. labelling of products) Predicting biosecurity risk (e.g. targeting high risk imports) Managing the environment (e.g. fisheries, water, land management)

Data Classification Data classification is key to efficiently harnessing the potential of Big Data. MPI is actively promoting a number of data classification initiatives and adopting multiple approaches to harness the broader potential of Big Data. New Zealand Organisms Register – a standard organism taxonomy of all New Zealand species. Farm Data Standards – common data vocabularies for efficient data exchange and support a move to precision agriculture based on sensors and data driven decisions. New Zealand Business Number – streamline business data inter-change. Grant Management System (GMS) – provides a central location and consistent classification for managing data on the majority of MPI’s science and research projects. Research, Science and Innovation Domain Plan – working with MBIE to improve the quality and alignment of this data across government and research organisations.

Core Infrastructure Data warehouses and GIS systems – to provide consistent data across MPI. Building staff capability in predictive analytics and data visualisation, supported by tools tailored for working with larger data sets.

Reusing data Purchasing data products built with Big Data technology – eg LIDAR analysis for forest management. Reusing export certificate data to understand economic activity and support response activity. MPI has set-up data sharing agreements with industry bodies, notably Beef+Lamb and DairyNZ. This is to reduce data collection burden while improving MPI access to good-quality on-farm data.

Improving efficiency Targeted interventions – using predictive analytics to schedule biosecurity inspections. Using fisheries catch-effort data to predict and manage fish stock. 

Collaboration Joint Border Analytics Pilot – working with border sector agencies to improve analytic capabilities and make agency border activity more efficient and effective. Using live data feeds to track shipping and fishing fleets. Working with StatsNZ, other agencies, and industry to improve data collection across the National Exotic Forest Description and Agricultural Production Survey.

Governance Developing processes and procedures to guide how MPI and its partners safely share and manage data. Improving data quality across MPI’s data collection systems.

Examples of Leveraging Big Data MPI has also undertaken detailed analysis of the primary industry workforce using the broader government official data through the IDI (Integrated Data Infrastructure).  This has enabled us to explore workforce characteristics on demographics, qualifications, and diversity to inform our policy work on meeting the future skill needs of the primary industries. MPI is also leveraging big data to inform the Consumer Insights Programme. Big data allows us to identify and quantify emerging market trends in consumer purchasing, including the product characteristics they look for, the places where they access information about products, and the retail channels they use.