Choose Your Own (Data) Adventure: Perils and Pitfalls, and Lots of Promise April L. Zenisky University of Massachusetts Amherst Address at NERA - Trumbull, CT: October 2015 1

2

3

4

5

6

7

Use the Data! Lots of talk lately about data and data use in educational settings Data is good, and using data is better! My way in - communication and use of assessment data Disclaimer: I like data! In educational settings the expectation of ‘use the data’ is interesting to me 8

Setting the (historical) stage Reporting data and results has largely been an enterprise straight out of the school of “you’ll take what we give you and you’ll like it” A Report was generated by agency charged with reporting, disseminated to users intact Control of the data, and hence control of the story about how the data should be interpreted, resided with the agency who owned the data Here’s where to look, what you’ll see, and here are prepped answers to the questions you’ll probably ask 9

The winds of change Mechanisms for the ‘use the data’ expectations are online dashboards, tools, and portals Paradigm shift from “Here’s the Story” to “You’re in Charge” Allow users to customize which results to display based on interests / needs Some blurring of line between reporting and secondary analysis Not disseminating results, but data (& access to it) 10

Disseminating data, not results Drawing from my work in score reporting Potential pitfalls in assumptions about Use, Access, and Understanding What is it that we hope to achieve when data is shared publicly through portals, tools, and dashboards? Is ‘use the data’ actually a reasonable expectation? 11

12 12

Use the data in educational settings This means different things to different people Often depends on where you fit in “Education” Administrative / managerial capacity (leaders) Instructional / delivery capacity (teachers) Personal capacity (families) Peripheral capacity (researchers, The Public) We all have thoughts about how others should use data 13

Assumption 1: Use the data It connotes action Go forth with numbers and do stuff Action (which includes inaction) is a function of who they are Very specific, given why they are there Use needs to take users into account 14

Avoiding the Use pitfall Needs analysis (Research!) Consideration of the users What do they want to know? What, if any, are the actions they want to take? What can data managers do to enable users? Balance of keeping tools flexible but supporting specific real-life use cases Necessary challenge for research and practice 15

Assumption 2: Access the data I have instructors and families squarely in mind here Issue 1: equity concerns Issue 2: opportunity concerns “I don't think it matters how easy they are to use.... time is the issue. Time to read them, time to figure out how to integrate results into classroom practice with multi levels and where differentiated teaching is necessary.” Issue 3: familiarity concerns 16

Avoiding the Access pitfall Access needs to be recognized as a problem! Equity, opportunity, familiarity - these impact data use in ways that are real, not theoretical No easy solutions here - problems often present themselves at the local level One strategy is use-case sharing and collaboration to build a toolbox of approaches Your ideas? 17

Assumption 3: Understand the data A lot gets made of a lack of relevant types of “literacies” Quantitative, graphical, data, assessment Step back from labels Working with data is like anything else - it requires investment of time and energy Cost-benefit ratio of effort to perceived outcome Hope is that if data makes sense = has value! 18

Assumption 3: Understand the data Understanding builds on process and knowledge Can users obtain results and work within the data to answer their questions to their own satisfaction? Are statistical caveats and limitations of data communicated clearly to support appropriate interpretations (and curtail errors)? How do we promote understanding in this choose- your-own-adventure approach, where control resides in the hands of the users? 19

Avoiding the Understanding pitfall Coming at this from the communication and systems side Quality dashboards, portals, and tools are a must Collaboration in systems development Consideration of use cases must be at the forefront 20

So, where does that leave us? I’ve described plenty of pitfalls related to use, access, and understanding Focusing on assumptions feels … negative. Go back to Slide 5. What is it that we hope to achieve when data is shared publicly through portals, tools, and dashboards? Is ‘use the data’ actually a reasonable expectation? 21

Disseminating data isn’t going away… Reflective of evolution in how people interact with information today These orientations toward information must inform the choices we make in advancing public use of data Explanatory and Instructive Exploratory and Constructive 22

23

24

“Use the data” is not unreasonable Must be accompanied by good choices behind the scenes, to empower users in their choices “Use the data” can be more than a buzzword Let’s build tools to support users and advance research and practice - that’s the real adventure! 25

Enjoy the rest of the conference! Thanks! Enjoy the rest of the conference! 26