Measurement Myths in Health Social Media Erica Holt www.ericaholt.com Twitter: @ericaholt Social Media Measurement No industry standard period, let alone for health care industry. My background includes health care (non profit medical center), health non profits and federal public health efforts – these observations are based on my experience implementing and measuring social media efforts with varying objectives for these types of organizations, and dealing with hopes pinned on unrealistic result expectations. I’m going to talk about some of the myths that have led to disappointment in health care social media and how to, especially for this field, not “believe everything you read” by original thought leaders in social media and focus on measuring what matters for your program and goals. Image: via Flickr, Zahra786 CC 2.0

What’s Working? Work Back From Measureable Outcomes Original Roadmap For Every Case Intrinsic Value What’s Not? Engagement Driving Goals Misconceptions of ROI Measuring the Moment State the problem VERBAL EXAMPLE We already know about strategy and setting (realistic) goals. Defining ‘outcomes’ measurement – means different things to different parts of field – behavior change, social marketing, actual health outcomes, sm as a learning tool But then we go to measure – and it always comes back to engagement Hopes pinned on unrealistic results Definitions Theory of change / logic model – make decisions about inputs, measure outputs, try best to correlate outputs to outcomes (pretty much impossible to prove causality in any case let alone SM) Measurement Inputs Outputs Outcomes Segmentation not aggregation Examples in health care and public health The obsession with measuring all things social is indicative of thinking inside rather than out. Inside thinking focuses on justification of time, energy and efforts all aimed at creating results. Outside thinking focuses on learning market needs and intents. Do you measure market needs and intents? The irony of measuring all things social is that the measures are put for the moment. Given the dynamic nature of how people use social technology, how suppliers are advancing the technology and the subsequent dynamics everything is but for a moment.  If things are in a dynamic state then measuring a moment becomes irrelevant to what is happening the next moment.

No social media measurement standards Too soon for best practices Special challenges for health care and public health Quick example of hc measurement challenge Erica Holt background: Global Standards, Six Different Ways Lisbon have set an intriguing, if perhaps impossible, agenda: “Moving towards a global standard of social media measurement.” I wish them luck. six (yes, six) other groups to develop similar guidelines, and I know exactly how difficult setting standards can be. Each one of those groups has a different definition of standards: Within the IAB and WAA crowd the debate is whether we should be using “unique visitors” or “unique browsers” to quantify the “reach” of any given post.  (I go with unique browsers.)  Among a number of Linked-in groups the question of a social media equivalent of AVEs is still open for discussion. (Here’s a not-so-subtle hint about the answer: It’s not going to happen.)  Media content analyzers are debating:  whether an item from a content farm should be included (it shouldn’t), and whether a story that shows up twice in 72 hours is a duplicate (it is), and  whether “balanced” should be a component of sentiment and if so would it be considered desirable or undesirable? (It should be and it is desirable.)  Within the Sentiment Analysis crowd the debate is around an acceptable standard of accuracy.  And for still other groups the question is, “What is a real definition of engagement or influence?” I’ve been working on these sorts of things for the better part of the last two decades, and the one thing I’ve learned is that there is no “standard” metric. The best you can hope for is that everyone agrees to the measurement equivalent of “Brush Your Teeth Every Day,” “Be Nice to Your Mother,” and “Thou Shalt Not Kill.”   Don’t get me wrong. I’m a big fan of standards. And while I may have once believed that they didn’t go far enough, I am now convinced that their adoption is a critical milestone in the history of public relations.  We have to recognize that every group jumps into the standard-setting fray with a slightly different agenda: Monitoring companies want a standard that is easy to implement and is acceptable to clients. Agencies want a standard that will make them look good in the eyes of the client and is acceptable to the COO or whoever controls the budget. Web analytics managers want a standard that is easily repeatable and defensibleto the marketing types. Marketing types want a standard that is easily understood by product managers. Meanwhile, the client just wants most of this stuff to go away and leave them alone.They want an easy answer.  Unfortunately, like any situation where people’s views, opinions, and agendas are colliding,there is no easy answer.  If you take a strong stand, believe in your principles, and call for boycotts for those who don’t comply with your standards, then you draw the ire of those with other agendas. If you just go along with whatever the client wants, then you end up never changing anything because big client organizations are notoriously resistant to change. And if you go with the flow and hope it all comes out okay in the end, chances are it won’t. As with any big overarching goal, it probably isn’t going to get solved in a day or two with one or two people and some grandstanding headlines. The real work gets done in living rooms, conference rooms, board rooms, and back rooms, over months of listening, accepting, rejecting, and bargaining

Engagement is measureable But is it what you want to measure Engagement is measureable But is it what you want to measure? Define Engagement Engagement does not necessarily correlate with or lead to outcomes (i.e. behavior) Most people agree that comments, likes, subscribes, indicates more engagement than a simple page view. What's measurable. Engagement is measurable -- but is that what you want to be measuring? There's a drive to measure "engagement" but what does that mean?   Getting people involved with a campaign or application does not mean you've succeeding with a marketing or health improvement objective. It may indicate learning, but we don’t have evidence of this just yet either. It’s a good place to start. Image: via Flickr, CC 2.0

No! There’s not one formula for ‘ROI’ ROI ≠ whatever you get for your efforts ROI is Not That Simple As I’ve said elsewhere recently, there is no single easy answer to social media ROI measurement. Right now there are at least half a dozen august bodies trying to define a standard. To make matters much worse, the term ROI is very commonly misused to mean “whatever you get for your efforts.” One more time for the record: ROI is a standard accounting calculation meaning “percent return on a financial investment.” The disheartening tendency to misdefine the term ROI causes confusion and miscommunication. Especially between the people who use it incorrectly and the board room. Who definitely do not use it incorrectly. (See, for instance, “The Dumbing-Down of "ROI:" Shel Holtz Makes Sense of It”.) A recent study by Tom Watson, Professor of PR at The Media School at Bournemouth University in the UK, found that 66% of PR people he surveyed used the term regularly but only 14% used it in the context of financial objectives. (Download WatsonROIpaperIPRRC2011) In fact, among the PR professionals he surveyed, 64% didn't think there should be a standard definition of ROI! Even worse, about 10% said they used an AVE-based formula. Talk about menaces! Those who try to dumb down social media ROI do themselves and the industry no favors. For a particularly egregious example, take Mia Iverson’s attempt to tackle social media ROI.  Her “formula” for ROI calculation is not even remotely related to real ROI. And to put initials around it doesn’t change that fact.  (Honestly, Mia, “SMROI?” It sounds like something you would eat around a campfire. I’d love to see your COO or CFO’s reaction.)  Mia uses tweets and likes for the “R” in ROI. She suggests that you take the “desired action” (the likes or click throughs) and divide by the total impressions. To quote: “...you can take your number of followers [impressions] and the number of click throughs [desired actions] and calculate your ROI accordingly. Problem solved.”  WTF? My point here is not to dump on Mia or blame her or anyone else struggling to simplify metrics. My point is that it’s just not that easy. The elusive ROI Magic Formula is not going to be discovered by just picking up a couple of numbers that are lying around the Internet. As with anything else, it’s Garbage In, Garbage Out. If you aren’t willing to do some hard work up front, you will get garbage for metrics.  Image: via Flickr: http://www.flickr.com/photos/kobie/307630928/sizes/l/  Some rights reserved by k_o_b_i_e

More Myths More Data is Better More Measurable = More Effective Others’ Case Studies Apply Myth: More Data is Better, Data deluge, most of it doesn’t matter Myth: More measurable = more effective Myth: Others’ case studies can apply to you. What worked for other, might not work.

Then What? Create Your Own Roadmap Pitch Value and Realistic Outcomes Start With Outcomes and Work Back Correlate Outputs to Outcomes Segmentation Not Aggregation MEASURING IS NOT THE TOUGH PART!: GARBAGE IN, GARBAGE OUT – IT’S THE UPFRONT WORK THAT’S MORE DIFFICULT. It’s agreeing on what metrics will matter beforehand. Setting goals that are measurable in the first place. Understanding that impact doesn’t always mean something goes up (like sales or eyeballs). Sometimes it can be that something goes down (like costs of customer service or traditional marketing costs). Benchmarking, which means measuring where you are NOW relative to your goals so you can track future progress and impact. This takes time, but you can’t ever determine results if you don’t know where you started. Understanding that social media may not be the outcome channel itself, but that there are a pile of ways it IMPACTS outcomes, and measuring those is key (such as learning). Knowing that determining ROI is ultimately about doing the math between dollars in and dollars out. Learning the art and science of correlation of data, so you can tie your efforts in one area of business to the results and impacts they have elsewhere. Realizing that software can give you the data and even help you crunch numbers, but you need to engage your brain to make it valuable to your work. There is no substitute for human analysis, ever.

Prioritize scarce resources Pursue a happy middle Finally Prioritize scarce resources Pursue a happy middle Take the pulse Then measure what you care about Digital media gives you a wealth of measures. Measure what matters. Engagement is measurable. Put performance in perspective of program goals. It is hard to bridge online with offline but you can establish digital goals.