1. Authority and Data Dr. Scott Renner sar@mitre.org 26 August 2008 DRAFT For discussion only For Limited External Release

2. © 2008 The MITRE Corporation. All rights reserved Authority and Data What could the term “authoritative data” mean in an enterprise like the DoD? There are four distinct meanings in use today –Agreed data vocabulary –Appropriate source system –Authorized data producer –Approved data for decision-making All four concepts are necessary in a very large enterprise To avoid confusion, we need uniform terms of reference –Each concept clearly defined –Distinct terms for distinct concepts This briefing presents those concepts and proposes terms 2

3. © 2008 The MITRE Corporation. All rights reserved 3 Preliminary Terms of Reference ProducersConsumers information

4. © 2008 The MITRE Corporation. All rights reserved 4 Preliminary Terms of Reference Consumers information data Producers

5. © 2008 The MITRE Corporation. All rights reserved 5 Preliminary Terms of Reference desktop computers handheld devices weapon systems sensor devices Information is produced by people operating…

6. © 2008 The MITRE Corporation. All rights reserved 6 Preliminary Terms of Reference application shared information space …with some sort of application posting information into a shared information space

7. © 2008 The MITRE Corporation. All rights reserved 7 Preliminary Terms of Reference application shared information space application … pulled or pushed to other applications for the benefit of other people operating various things

8. © 2008 The MITRE Corporation. All rights reserved 8 operated by mission organizations built by acquisition organizations Preliminary Terms of Reference application shared information space application operated by mission organizations built by acquisition organizations controlled by mission organizations Data Producers / Information Owners Data Sources Consuming Applications Consumers / Operators / Users Infospace

9. © 2008 The MITRE Corporation. All rights reserved 9 Preliminary Terms of Reference application Data Producers / Information Owners Data Sources Consuming Applications Consumers / Operators / Users Infospace Common Vocabulary: All things describing semantics and syntax Data models, elements, taxonomies, ontologies, …

10. © 2008 The MITRE Corporation. All rights reserved Key Characteristics of the Enterprise No effective central authority –Over producers, consumers, builders, and funding Instead, we have an “enterprise of enterprises” –Common purpose –Limited autonomy Often no single sovereign over all participants 10 What could the term “authoritative data” mean in an enterprise like the DoD?

11. © 2008 The MITRE Corporation. All rights reserved What Is The Enterprise? 11 ChinaSyriaFranceGermany UN StateTreasuryEPAJustice US NavyArmy DoD AFMCAFSOCAMCAETCACC Air ForceAustraliaUK Commercial NGOs This enterprise is formed from several sub-enterprises, all having limited autonomy, and some purpose in common View in slideshow mode

12. © 2008 The MITRE Corporation. All rights reserved Key Characteristics of the Enterprise No effective central authority –Over producers, consumers, builders, and funding –Several enterprises, common purpose, limited autonomy –No single sovereign over all participants Very large enterprise –Many people, large conceptual domain Some processes and information needs are not stable –Processes vary from highly-structured to ad hoc –Flexibility is often essential Information needs range from highly structured data to completely unstructured data 12 What could “authoritative data” mean in an enterprise like this?

13. © 2008 The MITRE Corporation. All rights reserved Authoritative = Agreed Data Vocabulary Information / data sharing won’t happen unless people understand the data in the same way –Establish a semantic match –Cope with representation mismatch When we write down this common understanding, we call it a “common vocabulary” Agreement doesn’t have to be voluntary for everyone –Some people can create the agreement –Others, subject to their authority, can be told to adopt it –(There are limits on how far this will work) 13 When organizations agree on semantics and representation, we call the result an authoritative data vocabulary vocabulary

14. © 2008 The MITRE Corporation. All rights reserved Example Each COI develops its authoritative data vocabulary –Glossaries –Taxonomies –Data models and data elements –Schemas –Ontologies –Whatever they need to record, use, and teach their common understanding of data COI members are expected to understand and use it 14

15. © 2008 The MITRE Corporation. All rights reserved A data source is an application (aka a system, or Program of Record) that provides data via services Usually this data is stored internally Authoritative = Appropriate Data Source 15 application data service interface

16. © 2008 The MITRE Corporation. All rights reserved Authoritative = Appropriate Data Source A data source is an application that provides data via services Other applications copy this data for local processing –Today, via web service (XML over HTTP) –Yesterday, in comma-separated files over FTP 16 3 2 5 4 6 1

17. © 2008 The MITRE Corporation. All rights reserved 9 Authoritative = Appropriate Data Source The fateful day –Instead of getting data from the original source… –An application obtains data from a copy –Or a copy of a copy 17 8 1 3 5 4 6 7 2

18. © 2008 The MITRE Corporation. All rights reserved 17 16 15 14 13 12 11 10 Authoritative = Appropriate Data Source Over time, the result is chaos –Copies of copies, latency, error “corrections” –Applications give different answers for same question –Nobody knows which applications are still needed 18 8 9 1 3 25 4 67

19. © 2008 The MITRE Corporation. All rights reserved Authoritative = Appropriate Data Source 19 ADS When the appropriate source system is established, we call it the authoritative data source

20. © 2008 The MITRE Corporation. All rights reserved Controlled Redundancy Can Be OK By the way, redundancy isn’t always bad –Controlled redundancy can be useful –If you can keep the sources consistent 20 ADS

21. © 2008 The MITRE Corporation. All rights reserved Each COI has its authoritative data vocabulary The DoD Metadata Registry (MDR) is the authoritative data source for these vocabularies Federated registries provide redundant, consistent access Example 21 Federated Registry Federated Registry MDR design time runtime

22. © 2008 The MITRE Corporation. All rights reserved Authoritative = Authorized Data Producer Data producers are mission organizations that create and maintain data as part of their assigned responsibilities –Systems are not producers –Systems are acquired for and operated by producers –Individual human operators are not producers –Individuals are part of a producer organization 22

23. © 2008 The MITRE Corporation. All rights reserved Authoritative = Authorized Data Producer Data producers are mission organizations that create and maintain data as part of their assigned responsibilities Data producers are accountable to the information owners Information owners have rights/duties for data –What data will be collected –What level of quality to maintain –Who is allowed to access or update –How these rights are delegated/assigned –This is a mission role, not acquisition Commander (not program manager) has authority Delegating and assigning these rights creates an information owner hierarchy 23

24. © 2008 The MITRE Corporation. All rights reserved Authoritative = Authorized Data Producer Data producers are mission organizations that create and maintain data as part of their assigned responsibilities Data producers are accountable to the information owners Information owners have rights/duties for data, exercised within an information owner hierarchy Many producers can be authorized/assigned to create and maintain the same kind of data –Each JFACC is authorized to create its own ATO A single data source may support several data producers –Every JFACC uses the same application to create ATOs 24 When a mission organization has the right & responsibility to produce some kind of data, we call it an authoritative data producer

25. © 2008 The MITRE Corporation. All rights reserved Example Each COI has its authoritative data vocabulary The DoD Metadata Registry (MDR) is the authoritative data source for these vocabularies Each COI is the authoritative data producer for its own vocabulary MDR COI #1

26. © 2008 The MITRE Corporation. All rights reserved Authoritative = Approved For Decisions Consumers use data to make decisions Commanders responsible for those decisions have authority to determine the source of that data –They don’t let just anybody provide the data –Rules and procedures specify information needs, sources & producers –Commander as infospace authority establishes the rules Different commands may have different rules/procedures –Sources and producers approved by one may not be approved by another 26 When a mission organization directs their deciders to use data from a particular source and producer, we call that the approved data, approved source, and approved producer

27. © 2008 The MITRE Corporation. All rights reserved Example: Air Mobility Command (AMC) AMC obtains certain aeronautical navigation data from the National Geospatial Agency (NGA) –NGA is the authoritative data producer –Some NGA system is the authoritative data source AMC provides corrections of this data to NGA –NGA incorporates these, over time; however… –AMC also maintains its own corrected data –AMC is the authoritative producer of these corrections –AMC’s Table Management Distribution System (TMDS) is the authoritative source for this corrected data AMC commander directs AMC consumers to use TMDS data –TMDS data is the approved data for AMC consumers –TMDS is the approved source 27 Source: Tina Woodbury (AMC), Neil Custer (CSC) – March 2005

28. © 2008 The MITRE Corporation. All rights reserved Summary Authoritative data vocabulary –Semantics and representation established by agreement among “important” organizations Authoritative data source –Application designated to provide specified data Authoritative data producer –Organization with right & responsibility to create and maintain specified data Approved data source and producer –Source/producer directed for use within an organization 28 Four concepts of “authoritative data”, four distinct terms

29. © 2008 The MITRE Corporation. All rights reserved OK, So What? Let’s see how these concepts and terms help describe CPM Data Governance CPM data governance begins with a data problem –Some consumer needs certain data, and can’t get it –Portfolio stakeholders cooperate to solve the problem Two ways to distinguish these data problems –Does the needed data exist? –Is the solution a matter for system developers only? This results in four categories of solution 29

30. © 2008 The MITRE Corporation. All rights reserved 30 Kinds of Data Problems and Solutions Net-enable the existing systems Build or change systems to collect the missing data Make data sharing agreements Change TTP No Users and Operators Yes Only Acquirers Who makes the change? Does the needed data exist? HARDER

31. © 2008 The MITRE Corporation. All rights reserved 31 Kinds of Data Problems and Solutions Net-enable the existing systems Build or change systems to collect the missing data Make data sharing agreements Change TTP No Users and Operators Yes Only Acquirers Who makes the change? Does the needed data exist? Infospace authority validates the data “need”, advocates for required change, and directs use of the approved data sources that are created

32. © 2008 The MITRE Corporation. All rights reserved 32 Kinds of Data Problems and Solutions Net-enable the existing systems Build or change systems to collect the missing data Make data sharing agreements Change TTP No Users and Operators Yes Only Acquirers Who makes the change? Does the needed data exist? Acquirers net-enable and/or modify the authoritative sources

33. © 2008 The MITRE Corporation. All rights reserved 33 Kinds of Data Problems and Solutions Net-enable the existing systems Build or change systems to collect the missing data Make data sharing agreements Change TTP No Users and Operators Yes Only Acquirers Who makes the change? Does the needed data exist? Authoritative data producers change their activities to begin producing the needed data

34. © 2008 The MITRE Corporation. All rights reserved 34 Kinds of Data Problems and Solutions Net-enable the existing systems Build or change systems to collect the missing data Make data sharing agreements Change TTP No Users and Operators Yes Only Acquirers Who makes the change? Does the needed data exist? Authoritative producers make sharing agreements with infospace authority

35. © 2008 The MITRE Corporation. All rights reserved Inferences People sometimes ask: Who owns the data element? Usually not a good question; however… –It belongs to the COI with authority over the vocabulary –Not to a data producer Many producers can create that kind of data –Not to a data source Several sources may share that kind of data Data producer to data source is a many-many relationship There can sometimes be several sources and producers for a given kind of information –A single source is desirable, not always feasible –Often desirable to have many producers Consumers may sometimes choose among several authoritative sources and/or producers 35

36. © 2008 The MITRE Corporation. All rights reserved Conclusion All four concepts are necessary in a very large enterprise Without distinct terms, confusion will ensue Within the data/info management community, this is our own “shared vocabulary” problem –Doesn’t really matter if we use the proposed terms –Does matter if we understand the concepts and have distinct terms for each 36

37. © 2008 The MITRE Corporation. All rights reserved References Net-Centric Information Management (2005) http://handle.dtic.mil/100.2/ADA464442 DoD Directive 8320.2, Data Sharing in a Net-Centric Department of Defense http://www.dtic.mil/whs/directives/corres/pdf/832002p.pdf DoD 8320.2-G, Guidance for Implementing Net-Centric Data Sharing (2006) http://www.dtic.mil/whs/directives/corres/html/832002g.htm My Two Cats Are a Community of Interest (2006) http://dodccrp.org/events/2006_CCRTS/html/papers/157.pdf http://www.dodccrp.org/events/2006_CCRTS/html/presentations/157.pdf Net-Centric Information Sharing (2005) A Style Guide for Common Vocabularies (2005) https://wwwd.my.af.mil/afknprod/DocView.asp?DocID=701841 37

38. © 2008 The MITRE Corporation. All rights reserved 38

39. © 2008 The MITRE Corporation. All rights reserved 39 Applications Post & Pull Through Services Data SourceData Consumer post servicepull service shared information space (physical) This is how most people envisioned the shared space in the early days of NCDS; however… controller

40. © 2008 The MITRE Corporation. All rights reserved 40 Applications Post & Pull Through Services service request/response Data SourceData Consumer shared information space (virtual) People now often envision the infospace as a virtual construct, which exists as the collected policy decisions of the infospace authority controller