Lies, Damn Lies, and Statistics Or How NOT to Do Data Governance Metrics

What is Data Governance Anyway? Data Management = Doing things right Data Governance = Doing the right thing Doing the right thing is about making sure that your data is collected, used and stored according to (1) laws and regulatory regimes; (2) ethical standards; (3) the greatest benefit of the organization that collected it and the consumers that organization serves.

How Do You Measure Governance Effectiveness? It’s Easy to Make Mistakes Proofiness: How You’re Being Fooled By the Numbers by Charles Seife. ”Causistry” “Potemkin Numbers” “Apple Polishing” The goal of metrics is to provide insight, not validation. Causistry implies a statistical linkage between two variables when there is no linkage at all, much less a causal one. Potemkin numbers is the act of using phony measurements to suggest an end that you had already determined. Apple polishing is manipulating data to make them seem more attractive than they actually are.

Mistake #1: Focusing Only on Stellar Metrics The purpose of metrics is to drive an organization towards improvements. You don’t always want a stellar metric. If your performance metrics are always perfect, then you probably don’t have very good performance metrics. It is tempting to present great numbers to management to maintain support and budgets, but this isn’t what metrics are supposed to do.

Mistake #2: Ignoring the behaviors that your metrics are driving What gets measured, gets done. Metrics must be defined--and incentive structures well-thought out-- in advance of system development Another Example: Continental Airlines metrics on fuel consumption to cut costs.

Mistake #3: Believing The Numbers BAD METRICS: Trying to use statistics to measure something that is inherently subjective (beauty, happiness, taste, enthusiasm). EXAMPLE: Happiness=P+(5xE)+(3xH) P=Personal Characteristics E=Existence (reflects health measures) H=Higher Order Needs (recognition)

Starting With the Right Question My responses are limited. You must ask the right questions.

Do my data governance processes address privacy concerns and document breaches? The Health Insurance Portability and Accountability Act, the federal law that details requirements for medical service providers to protect patient medical records. Example: Investigation of CVS Pharmacies by Health and Human Services Governance Issue: Failure to properly protect patient privacy under federal regulations Metrics: Are systems tagged with appropriate warning notices on applicable laws and regulations; are personnel trained; is there a policy for sanctioning employees who do not comply?

Do my processes ensure my data is used for appropriate purposes? “Consumers care about their privacy and should have a say in how their personal information is used, especially when it comes to who knows what they’re doing online.” Example: Verizon Wireless use of “supercookies” Governance Issue: Data collected to target Verizon advertising, but insufficient attention paid to the “hijacking” of the information by third parties Metrics: What data you collect (volume, type, description), how many people access it, what their roles are, existence and updating of access control policy

Some other questions to consider… Do we know what our industry regulatory regimes are and do our data governance processes gather the data that regulatory reporting requires? Do my data governance processes ensure data is accessible to those who put it to use for the benefit of my organization and consumers? Are my governance metrics driving the kind of behavior that my organization wants to encourage in our members or employees? Question 1: often applicable to financial industry, academic institutions Question 2: often applicable to government agencies and non profits Question 3: A question for mature programs requiring experience and higher level critical thinking about the overall strategy of the organization.

And Finally…Don’t Ignore Qualitative measures Every good feedback loop should include qualitative measures as well as quantitative. When you are trying to make a point to a management level audience, stories of impact almost always get a better response than simple numbers and charts. data professionals "need to be really effective storytellers...it's not just about accessing data, but about the ability to link that to strategy, to share it with our communities." --Julia Richman, Chief Innovation and Analytics Officer, City of Boulder, CO