1. Intelligence, Surveillance, and
U.S. Combatant Command
Background:
This case study is about a specific combatant command that is unique for several reasons: the nature of its missions; its focus on irregular warfare; its global area of responsibility; and its need for interoperability with DoD, IC, and International partners.
This command does not have a large enterprise culture and they have no formal mandate to share data. But, intelligence is key to their success. There are units within the Command that are among the most proficient mission-specific users and managers of data in DoD, yet there is no imperative to share data outside of the immediate circle of trust of that unit nor consider the strategic value of data across the Command. This becomes apparent when compared to the highly effective initiatives behind standardization and interoperability of secure high-capacity wired, SATCOM, and radio communications channels.
But, what is common among all Combatant Commands is the ongoing struggle to move from ‘need to know’ to ‘need to share’ mind-set. And, to do this task securely while also managing a wide variety of data, business processes, programs, people, and roles. As such, there are significant challenges associated with implementation of a cohesive, command-wide knowledge management (KM) and information sharing (IS) approach.
For the past ten years the Intelligence, Surveillance, and Reconnaissance (ISR) community has focused on improving information sharing across commands. A key part of achieving this goal is ensuring a robust enterprise data layer that serves as the platform for collecting, storing, and sharing data.
Findings: Knowledge Management Gaps
Before MarkLogic began working with this Command, we met with key stakeholders who provided a number of insights that helped guide the direction of the project.
The discussions revealed many common perceived capability gaps:
No enterprise-wide search for all documents whether in SharePoint portals or shared drives
Duplication of effort and systems. Every program is paying for its own database, search engine, hardware, ancillary software, and integration
No way to leverage a Command taxonomy to drive better precision and recall in search and alerting
Inability to leverage desired social media-type functions including tagging, rating, following, posting
Inability of certain communities of interest to leverage library services
Inability to perform business intelligence (BI)-type discovery and analysis on a combination of structured and unstructured data
Lack of Search and Alerting as a Service across the enterprise (Cloud Deployment)
The main point is that the Command is as much a collection of information management components as it is an operational force. Without an enterprise approach to Knowledge Management and Information Sharing, each component within the Command is at risk of becoming a ‘silo of excellence.’ Or, even worse, the Command’s missions is put at risk through reduced interoperability, duplication of effort, and IT systems that cost too much and lack agility.
Intelligence, Surveillance, and
Reconnaissance (ISR)

2. Enterprise data layer for U.S. forces
Before MarkLogic
Inflexible data aggregation
No support for sensor feeds, UAV feeds, cyber security data, human terrain, and social media
Discoordinated technologies
Limited data types
No DIL support
Expensive to maintain
After MarkLogic
Complex Data Integration All Multi-INT data: UAV feeds, sensor feeds, cyber security event data, social media data—and any other SIGINT, HUMINT, or FMV data
Support for DIL Operations Easy distribution of data without worrying about security vulnerabilities or disconnected, intermittent, and latent networks
Increased Productivity Faster development with an agile database that adapts to change
Trusted Platform Government-grade security on a hardened platform that ensures data resiliency
Reduced Cost Requires less storage and runs in the cloud (ie. less hardware)
Before ML Narrative:
This Command wanted to develop a cohesive, command-wide knowledge management (KM) and information sharing (IS) platform to support the Intelligence, Surveillance, and Reconnaissance (ISR) community. To accomplish this, they needed an Enterprise Data Layer (EDL) to serve as a centralized system architecture that would aggregate various types of information from multiple sources into a consistent, well-defined view. However, their legacy technology, Oracle and Endeca, was not flexible or agile enough to handle the data aggregation beyond straightforward text and imagery, and would not support overseas operations in areas with disconnected, interrupted, and low bandwidth (DIL).
More Detail on Specific ISR Challenges:
Explosion of Sensors Recently the number and type of sensors used in the battlefield have greatly increased. Stationary towers, UAVs and other airborne platforms are constantly being upgraded with the latest detection or imagery technology. Too often each new platform is delivered with its own exploitation and processing capability. Additionally, new data formats require constant re-engineering of existing systems.
Explosion in Volume New high resolution sensors and long endurance Multi-INT airborne platforms will generate unprecedented amounts of data in constantly changing data formats. Ingesting, processing, storing and disseminating such volumes of information cannot be achieved with the current infrastructure.
Information Overload As the number of Multi-INT derived products explodes, it gets harder for the war fighter to find the mission critical intelligence he or she is looking for. During mission planning and execution more and more time is spent searching, reviewing and correlating intelligence products spread around multiple ISR or Command and Control (C2) systems. This results in waste of valuable resources and a constant risk of using outdated information.
After ML: The MarkLogic ISR solution revolutionizes how U.S. Government organizations share, discover, and exploit Intelligence, Surveillance, and Reconnaissance (ISR) information. Compared to legacy solutions, it provides advances in performance, scalability, and functionality, requires less hardware, and reduces operation and maintenance costs.
Acronyms:
SIG-INT (Signal intelligence) – refers to the interception of signals, either text or voice, electronic or human (
HUM-INT (Human intelligence) – refers to information gathered from interpersonal, human contact (
FMV (Full motion video) – full color or infrared video data and associated metadata.

3. X X Before MarkLogic Limited data integration Connected Operations UAVZ X X Z
DIL Operations
Whiteboard Built-Out Script:
1. Problem Statement
Historically the Command’s systems allowed to share and discover imagery and textual intelligence. With the emergence of asymmetrical warfare:
1.1. Sensor feeds, UAV feeds, cyber, human terrain and social media could not be exploited
1.2. Disconnected Interrupted and Low bandwidth DIL) special operations could not be supported
1.3. Counter insurgency, counter terrorism, stability and other missions could not supported
2. Before State
Legacy infrastructure was Oracle stack and Endeca
1.1. Data sources limited to Imagery and Text
1.2. No support for unstructured and fast changing data Sources.
1.3. No DIL Support
1.4. Counter insurgency, counter terrorism, stability missions could not be supported
Enterprise Data Layer

4. After MarkLogic Complex data integration More advanced features
Connected Operations
UAV
UAV Z Z Z Z
DIL Operations
DIL Operations Z
Whiteboard Built-Out Script:
3. After State
1.1. Support ingest of sensor feeds, UAV feeds, cyber, human terrain and social media
1.2. Support for DIL operations
1.3. Increased analysts and operators productivity
1.4. Provided mission and technical agility
Longer Narrative:
The Enterprise Data Layer is a centralized, searchable system that aggregates data from disparate ISR and INT silos to provide a complete, actionable view of all mission-critical data. It makes information discoverable and accessible to the war fighters and existing COP applications:
Decouples data sources from information users (human or systems)
Disseminate mission oriented products regardless of sources and original formats
Transparently integrates Multi-INT sources
Decouples consumers from change in sources data formats – maximizes agility
The EDL is configured to support multiple ISR Information Domains such as imagery or signal intelligence, to deliver data fusion and collaboration applications supporting various Communities of Interest (COIs). The EDL provides automatic mechanisms to create, provision, manage, and secure the information domains. Information domains are automatically created, provisioned, and deployed from the domain definitions contained in the metadata catalog. Each domain provides:
A data fusion and collaboration portal allowing users to search, edit, collaborate, and disseminate information associated with a corpus of information defined in the metadata repository.
Web services to ingest, search, and retrieve information.
The ability to participate in federated search with other information domains.
A secure environment based on the access permissions defined in the metadata catalog.
The EDL Metadata Catalog extends the functionality of a metadata catalog. It not only describes the ISR information and where to find it, but also defines how it is secured and automatically managed. It holds the policies and parameters, allowing the provisioning of the information domains and all SOA data services. Metadata is used to capture user activities and operations on the mission data. This approach allows organizations to minimize rework, track individual activities, and minimize the number of copies of the same dataset.
Enterprise Data Layer
More advanced features

5. 59 times faster than RDBMS 57 percent reduction in disk space used
100+ million documents
70 servers in 10 clusters
59 times faster than RDBMS
Performance
Raytheon was commissioned to independently evaluate the performance of the MarkLogic DIB MetaData Catalog (MDC) against traditional RDBMS-based solutions. Following are their results and conclusion:
Performance 20 million record test
• MarkLogic MDC compressed the footprint of the ingested raw data by 33% even with indexing turned on, and used 43% the disk space that of APEX/DIB 1.3.
• MarkLogic MDC outperformed APEX/DIB 1.3 and was the only test configuration to successfully complete Test 4 (120 Users).
• Examining the APEX/DIB 1.3 RDBMS during the test with 6 users, disk I/O was shown to be the limiting factor. This is consistent with RDBMS vendor observations in the DIB 2.0 tuning guide. However, MarkLogic was able to complete the 6 user test in 56 minutes with the same disk I/O capability.
• MarkLogic completed the same amount of work 59 times faster than the RDBMS (56 minutes vs 55 hours).
• MarkLogic’s average response time was 32 times faster than that of the RDBMS-based MDC.
These tests provided convincing evidence that MarkLogic server provides superior performance as the “engine” for the DIB MetaData Catalog (MDC), superior to the RDBMS implementation under the increasing user loads and data volumes that are present in an enterprise environment. This clear performance edge is another powerful argument for use of MarkLogic as part of any DIB-enabled DCGS node.”
57 percent reduction in disk space used

6. Flexible replication to globally share information
Getting data when and where you need it can be challenging—and even risky or impossible. Global organizations often run into latency issues trying to replicate data from centralized data centers to remote areas. And, for military operations in particular, replicating data to the field is often problematic because of cybersecurity concerns and technical roadblocks created by adversaries armed with signal jamming equipment.
Flexible Replication solves these problems by allowing you to easily distribute portions of your data without worrying about security vulnerabilities or disconnected, intermittent, and latent networks.
MarkLogic has full database replication, which organizations use for disaster recovery. But, flexible replication is an ideal choice when organizations need the capability to:
Enable field agents and remote workers to access data in austere environments
Replicate filtered data from a centralized database to thousands of edge nodes around the world
Collect data on edge nodes and have it automatically added to the main database when reconnected
Provide improved performance by conducting queries and updates locally rather than over a network
Address issues with disconnected, intermittent, and latent networks, often using radio or satellite
Prioritize security and maintain the ability to use Role Based Access Control (RBAC)
In the case of this combatant command, they can use flexible replication to get the full capabilities of an enterprise database out to the field on laptops or portable server carry-cases. They can get just the data they need, and then when they are back at headquarters, they can reconnect and synchronize all their data to the main system.