1. Why and how you value your plumber: establishing and conveying value in the outcomes of a data virtual organization. Peter Fox (TWC/RPI) ESIP, Jan 4. 2011 Peter Fox (TWC/RPI) ESIP, Jan 4. 2011

2. What you will hear  A bit of history – plumbers (PG13)  Why – VOs: Outcomes and value  How  Evaluating – really, really?  Provenance for a virtual organization?  Opportunities for ESIP and Type Is  A rough attempt to pull it all together  A bit of history – plumbers (PG13)  Why – VOs: Outcomes and value  How  Evaluating – really, really?  Provenance for a virtual organization?  Opportunities for ESIP and Type Is  A rough attempt to pull it all together

3. ESIP history  11 th + year – the honeymoon is well over, past the 7-year itch, now it’s time for ESIP and those that fund it to get comfortable  Plumbers – I promise not to over-do this analogy but it arose (surprise) from the early years of DODS, nka OPeNDAP  The way to make data matter is to get it from those that have to those that want it  Inlets and outlets  Infrastructure is required  Computers and people  Fittings, pipes and yes plumbers  11 th + year – the honeymoon is well over, past the 7-year itch, now it’s time for ESIP and those that fund it to get comfortable  Plumbers – I promise not to over-do this analogy but it arose (surprise) from the early years of DODS, nka OPeNDAP  The way to make data matter is to get it from those that have to those that want it  Inlets and outlets  Infrastructure is required  Computers and people  Fittings, pipes and yes plumbers

4. Virtual Organizations, i.e. ESIP  ‘A VO is a group of individuals whose members and resources may be dispersed geographically and institutionally, yet who function as a coherent unit with a common goal.’ (Beyond Being There 2008) or a long- term common interest (Ahuja)  ESIP has VOs within VOs, i.e. some organizational structure; executive committee, committees, working groups, clusters  Many goals, different levels of coherency  ‘A VO is a group of individuals whose members and resources may be dispersed geographically and institutionally, yet who function as a coherent unit with a common goal.’ (Beyond Being There 2008) or a long- term common interest (Ahuja)  ESIP has VOs within VOs, i.e. some organizational structure; executive committee, committees, working groups, clusters  Many goals, different levels of coherency

5. Reminder: roles and relationships  ‘The virtual research organization: in which members of various corporate and academic research units voluntarily come together to advance a technology on an ongoing basis.  These members assume well defined roles and status relationships within the context of the virtual group that may be independent of their role and status in the organization employing them’ (Ahuja et al., 1998).  I.e. do your goals match ESIPs?  ‘The virtual research organization: in which members of various corporate and academic research units voluntarily come together to advance a technology on an ongoing basis.  These members assume well defined roles and status relationships within the context of the virtual group that may be independent of their role and status in the organization employing them’ (Ahuja et al., 1998).  I.e. do your goals match ESIPs?

6. Contrasting a Virtual Information Organization with a Traditional one

7. Successes focus on value

8. Evaluation of a VO (1)?  An assessment of the overall effectiveness of an effort, ideally yielding a numeric measure by which informed cost-benefit analysis of investment decisions can be made.  An assessment of the degree to which the effort fulfills its specification in terms of whatever measures were pre- specified.  An assessment of the overall effectiveness of an effort, ideally yielding a numeric measure by which informed cost-benefit analysis of investment decisions can be made.  An assessment of the degree to which the effort fulfills its specification in terms of whatever measures were pre- specified.

9. Evaluation of a VO (2)?  An assessment of whether the effort fulfills the purpose for which it was intended  An assessment of whether the artifacts or processes embodied in the effort have been proved to be superior to an alternative, where that alternative is frequently the traditional solution  An assessment of whether funds allocated to an effort have been productively used, yielding useful generalizable results  An assessment of whether the effort fulfills the purpose for which it was intended  An assessment of whether the artifacts or processes embodied in the effort have been proved to be superior to an alternative, where that alternative is frequently the traditional solution  An assessment of whether funds allocated to an effort have been productively used, yielding useful generalizable results

10. Evaluation of a VO (3)?  An assessment of whether the effort proves acceptable to the intended end-users  An assessment of whether end-users continue to use it in their normal work  An assessment of where the effort fails to perform as desired or as is now seen to be desirable  An assessment of the relative importance of the inadequacies of the effort  An assessment of whether the effort proves acceptable to the intended end-users  An assessment of whether end-users continue to use it in their normal work  An assessment of where the effort fails to perform as desired or as is now seen to be desirable  An assessment of the relative importance of the inadequacies of the effort

11. (Orthogonal) Dimensions of evaluations StructuredLess structured QuantitativeQualitative SummativeFormative Controlled experimentsEthnographic observations Formal and rigorousInformal and opportunistic http://janus.ucc.nau.edu/edtech/etc667/proposal/evaluation/summative_vs._formative.htm

12. Formative and Summative  Evaluation carried out for two reasons:  giving feedback = formative evaluation  grading the outcome = summative evaluation  “When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative." (Stakes)  What you use to evaluate (formative) is often different from what others use (summative)  Evaluation carried out for two reasons:  giving feedback = formative evaluation  grading the outcome = summative evaluation  “When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative." (Stakes)  What you use to evaluate (formative) is often different from what others use (summative)

13. Summative == Metrics  Things you can measure (numerical)  Things that are categorical  Could not do before  Faster, more complete, less mistakes, etc.  Wider range of users  What others for ESIP?  Measure or estimate the baseline before you start  Things you can measure (numerical)  Things that are categorical  Could not do before  Faster, more complete, less mistakes, etc.  Wider range of users  What others for ESIP?  Measure or estimate the baseline before you start

14. Formative and Summative 14

15. Evaluation questions: To what extent does ESIP Interviews/ Focus Group SurveysDocument Analysis Observation Activities enhance end-user access to and use of data to advance science and education needs? XXXX Activities enable higher levels of semantic capability and interoperability such as explanation, reasoning on rules, and semantic query? XXXX Contribute to the development and support of community resources, data systems and provision of results from diverse observing systems using suitable technologies? XXXX 15 Evaluation questions and associated data collection methods

16. Evaluation questions:Interviews/ Focus Group SurveysDocument Analysis Observation What factors impede or facilitate progress toward ESIP goals? XX What progress has been made toward sustaining and ‘scaling up’ ESIP activities? XXX Does ESIP contribute to the reports on modern data frameworks, user interfaces, and science progress achieved? XXXX Does ESIP incorporate user experiences in the evolution of its organization? XXXX 16 Evaluation questions and associated data collection methods

17. VOs - artifacts or processes?  Both – i.e. what is created, documented, saved as well as the things that were done along the way and by whom, and how, etc.  E.g. community building, inter-agency working activities, tutorials, …  VOs tend to deal better with artifacts than processes  Both – i.e. what is created, documented, saved as well as the things that were done along the way and by whom, and how, etc.  E.g. community building, inter-agency working activities, tutorials, …  VOs tend to deal better with artifacts than processes

18. Recall VO- coupled elements Technology Communication Patterns Organizational Structure

19. Some of this is, er…  Provenance - Origin or source from which something comes, intention for use, who/what generated for, manner of manufacture, history of subsequent owners, sense of place and time of manufacture, production or discovery, documented in detail sufficient to allow reproducibility (or assess value!)  Yes, apply this to ESIP – VO provenance  Provenance - Origin or source from which something comes, intention for use, who/what generated for, manner of manufacture, history of subsequent owners, sense of place and time of manufacture, production or discovery, documented in detail sufficient to allow reproducibility (or assess value!)  Yes, apply this to ESIP – VO provenance

20. And the reality? It’s about the questions that are being asked, e.g.  What exactly happened during that technical workshop last summer and how has it helped our agency?  Exactly what value do you (or we) get by you being the chair of a Committee (Working Group, Cluster)? It’s about the questions that are being asked, e.g.  What exactly happened during that technical workshop last summer and how has it helped our agency?  Exactly what value do you (or we) get by you being the chair of a Committee (Working Group, Cluster)?

21. The ecosystem?  These are what enable anyone to explore/ confirm/ deny their value perceptions or get quality answers to direct questions… Accountability ProofExplanationJustificationVerifiability ‘Transparency’ (the illusion of it) Trust Provenance - Internal/ External Identity

22. Why an illusion?  It’s not that the word transparency is wrong, it is what it is being used for –  “If I let you see everything, you can get answers to your questions”  Nope, not unless you are very lucky…  It depends on  Who is asking the question (and why)  What the answer will be used for  CONTEXT and ROLE (poorly represented)  It’s not that the word transparency is wrong, it is what it is being used for –  “If I let you see everything, you can get answers to your questions”  Nope, not unless you are very lucky…  It depends on  Who is asking the question (and why)  What the answer will be used for  CONTEXT and ROLE (poorly represented)

23. So … translucency  See just what is necessary and sufficient  Practical definition  As close to the relevant data, information and knowledge artifacts presented in an appropriate form  Yes, I mean curation for evaluations and value assessments  Methodological means - many  See just what is necessary and sufficient  Practical definition  As close to the relevant data, information and knowledge artifacts presented in an appropriate form  Yes, I mean curation for evaluations and value assessments  Methodological means - many

24. And some …  Identity  YOUR identity  Friends, organizations  Communities  Peer and non-peer relations  Accountability  By whom, to whom  When and how often  Documentation – are you happy Ted?  Identity  YOUR identity  Friends, organizations  Communities  Peer and non-peer relations  Accountability  By whom, to whom  When and how often  Documentation – are you happy Ted?

25. Just remember  Plumbers fix things, they show up, the do the work, they leave and then you get the bill… they work in an analog world  Thus the value proposition is different (you all know that) in a digital world  So, we need a …  Plumbers fix things, they show up, the do the work, they leave and then you get the bill… they work in an analog world  Thus the value proposition is different (you all know that) in a digital world  So, we need a …

26. We need a Knowledge Base VO Provenance Descriptions of the artifacts Specific terms/ language 26 Questions ? Who? What/w hen/wh y/ how? Answer: explain value of outcomes

27. Populating the KB: Communication  A key feature of virtual organizations is a high degree of informal communication  Because of a lack of formal rules, procedures, clear reporting relationships, and norms, more extensive informal communication is required  -> as a result a lot of VO provenance is either  Not collected  Collected but fragmented, disconnected  Not identified as provenance (when it is created)  A key feature of virtual organizations is a high degree of informal communication  Because of a lack of formal rules, procedures, clear reporting relationships, and norms, more extensive informal communication is required  -> as a result a lot of VO provenance is either  Not collected  Collected but fragmented, disconnected  Not identified as provenance (when it is created)

28. Opportunities  To undertake both formative (ESIP) and summative (Type Is) evaluations  Give some level of formalization to the meaning/ terms and relations – progress  An incentive to identify provenance of the VO that is ESIP  ‘Rules’ to define the combinations of evidence required – value – ESIP and Is  Type Is - some use cases (state+value)  A few test cases, I/O metrics  To undertake both formative (ESIP) and summative (Type Is) evaluations  Give some level of formalization to the meaning/ terms and relations – progress  An incentive to identify provenance of the VO that is ESIP  ‘Rules’ to define the combinations of evidence required – value – ESIP and Is  Type Is - some use cases (state+value)  A few test cases, I/O metrics

29. Comment: Translucent VOs  Along the spectrum from opaque (bad) to transparent (unsustainable) is a balance  Applies to leadership, communication, feedback  Posit: balance arises out of flowing requirements consistently to the working level of the VO and percolating direct experience and value of outcomes up  Usually depends on the right people – not everyone can evaluate (or be evaluated)  Along the spectrum from opaque (bad) to transparent (unsustainable) is a balance  Applies to leadership, communication, feedback  Posit: balance arises out of flowing requirements consistently to the working level of the VO and percolating direct experience and value of outcomes up  Usually depends on the right people – not everyone can evaluate (or be evaluated)

30. Bringing this together  A persistent presence and record of artifacts, activities and processes is needed: state and metrics/ value  No, this is currently not being done (or feasible)…  Facilitate sustained activities - support the notion of nested VOs by establishing their value (may change funding model)  But really - take ESIP from an organization to a ‘brand’ (like AMA) – then you know who they are gonna call (w/ apologies to Ghostbusters – that was the other story I could have told)  A persistent presence and record of artifacts, activities and processes is needed: state and metrics/ value  No, this is currently not being done (or feasible)…  Facilitate sustained activities - support the notion of nested VOs by establishing their value (may change funding model)  But really - take ESIP from an organization to a ‘brand’ (like AMA) – then you know who they are gonna call (w/ apologies to Ghostbusters – that was the other story I could have told)

31. Thanks  pfox@cs.rpi.edu pfox@cs.rpi.edu  http://tw.rpi.edu http://tw.rpi.edu  pfox@cs.rpi.edu pfox@cs.rpi.edu  http://tw.rpi.edu http://tw.rpi.edu