OMOP CDM on Hadoop Reference Architecture Target audience: technical or IT staff currently using or hoping to use OHDSI OMOP CDM Goals: Show design for CDM using ‘big data’ tools that are Apache open source as alternative to Oracle or Postgres Show ingestion paths or approaches for various data types and data sources Map current (Nov 2016) most commonly used tools fit for various business purposes based on market share but not exclusive to Cloudera Glossary of acronyms OHDSI: Observational Health Data Science & Informatics OMOP: Observational Medical Outcomes Partnership CDM: Common Data Model HDFS: Hadoop Distributed File System RDBMS: Relational Database Management System SQL: Structured Query Language Revision history Original: November 29, 2016 Send questions to sdolley at cloudera dot com Original technical authors: Derek Kane, Tom White N.B. This is offered as one opinion on the current most selected set of choices to fill technical capabilities, and is not inclusive of all approaches.

UPDATEABLE, ANALYTIC STORAGE OMOP CDM on Hadoop Reference Architecture BIG DATA SUPERSET ARCHITECTURE FOR A DATA LAKE BATCH PROCESSING SPARK HIVE SQOOP MAPREDUCE PIG ANALYTIC SQL IMPALA SEARCH ENGINE SOLR MACHINE LEARNING SPARK REAL-TIME PROCESSING SPARK STREAMING KAFKA FLUME UPDATEABLE, ANALYTIC STORAGE KUDU WORKLOAD MANAGEMENT YARN WORKFLOW MANAGEMENT OOZIE SECURITY SENTRY, RECORD SERVICE HDFS Encryption, TLS/SSL, Kerberos FILE STORAGE PARQUET Source HDFS FILESYSTEM HDFS ONLINE NOSQL HBASE Source Source Legend: Box titles in black bold such as “BATCH PROCESSING” are the technical capability required to meet some business requirement(s). Tool names in dark gray such as “SPARK” or “Kerberos” are Software/Apache projects that can be implemented to meet the technical capability need. Boxes near the bottom of the architecture tend toward storing data; middle boxes tend toward managing or organizing the data; boxes at the top tend toward enabling users to access the data. NB: alternate, less common tool options can be found later in this document.

UPDATEABLE, ANALYTIC STORAGE Minimum software stand up for CDM in Hadoop MINIMUM SOFTWARE TO STAND UP CDM IN HADOOP BATCH PROCESSING SPARK HIVE SQOOP MAPREDUCE PIG ANALYTIC SQL IMPALA SEARCH ENGINE SOLR MACHINE LEARNING SPARK REAL-TIME PROCESSING SPARK STREAMING KAFKA FLUME UPDATEABLE, ANALYTIC STORAGE KUDU WORKLOAD MANAGEMENT YARN WORKFLOW MANAGEMENT OOZIE SECURITY SENTRY, RECORD SERVICE HDFS Encryption, TLS/SSL, Kerberos FILE STORAGE PARQUET Source HDFS FILESYSTEM ONLINE NOSQL HBASE Source Source HDFS Legend: Box titles in black bold such as “BATCH PROCESSING” are the technical capability required to meet some business requirement(s). Tool names in dark gray such as “SPARK” or “Kerberos” are Software/Apache projects that can be implemented to meet the technical capability need. Boxes near the bottom of the architecture tend toward storing data; middle boxes tend toward managing or organizing the data; boxes at the top tend toward enabling users to access the data.

Hadoop Ingestion Paths Ingestion method varies by data source & complexity of transformations needed Three approaches, select one or many Flat File Ingestion CSV/TSV/TXT Hive RDBMS Ingestion from relational databases (MySQL, PostgreSQL, Oracle, SQL Server, etc.) RDBMS SQOOP Hive Complex Ingestion (JSON, XML, Nested sources, custom, etc.) Source file Spark Hive Note: All data is most often physically stored in HDFS. This graphic shows the most commonly used tools to manage getting data into HDFS.

Hadoop Ingest/Egress, Hive & Impala GETTING DATA IN DATA STORED & MANAGED GETTING DATA OUT Ingestion method varies by data source & complexity of transformations needed. Three approaches, select one or many Flat File Ingestion CSV/TSV/TXT Hive Hive Catalog Impala SQL from SQL Render OHDSI apps HDFS Other SQL, other apps (optional) RDBMS Ingestion from relational databases (MySQL, PostgreSQL, Oracle, SQL Server, etc.) RDBMS SQOOP Hive Hive Catalog Impala SQL from SQL Render OHDSI apps HDFS Other SQL, other apps (optional) Complex Ingestion (JSON, XML, Nested sources, custom, etc.) Source file Spark Hive Hive Catalog Impala SQL from SQL Render OHDSI apps HDFS Other SQL, other apps (optional)

for ETL and Batch processing for fast, in-memory analytics SQL on Hadoop Hive and Impala share the same metadata repository (data dictionary) Data ingested into HDFS and made available in Hive… HDFS Hive for ETL and Batch processing Metadata (in Hive Catalog) Impala ODBC/JDBC tools for fast, in-memory analytics … is also available in Impala.

Programming framework Streaming Workflow Engine Cloudera options Batch data ingest Programming framework Streaming Workflow Engine Spark MapReduce Kafka Spark Streaming Sqoop Flume Oozie Security Interactive Database (SQL) Search Other Sentry Kerberos RecordService HDFS Encryption Impala SOLR / Cloudera Search Pig Hive on Spark Hive on MapReduce Data storage file format Storing data (key value) Storing data (columnar) Storing data (files) Hadoop File System AWS S3 AWS EBS EMC Isilon OneFS Kudu Parquet Hbase Avro

With other options listed Batch data ingest Programming framework Streaming Workflow Engine Kafka Spark Streaming Storm Spark MapReduce Oozie Nifi (aka Hadoop Data Flow) Sqoop Flume Security Interactive Database (SQL) Search Other Impala Hive on Spark Hive on MapReduce Hive on Tez Hawq Presto Sentry Kerberos RecordService HDFS Encryption Ranger Knox Pig SOLR / Cloudera Search Data storage file format Storing data (key value) Storing data (columnar) Storing data (files) Parquet Hbase Cassandra Accumulo Hadoop File System EMC Isilon OneFS S3 Kudu Avro NB: due to authors lack of familiarity with many alternative tools listed here, we cannot guarantee the tools shown can provide the technical capability shown

Minimum projects/products needed for ingesting syndicated data into OMOP CDM for analysis Interactive Database (SQL) for running queries Impala Hive Storing data (files) Hadoop File System (HDFS) Batch data ingest for ingesting data into CDM Programming framework for ingesting data into CDM AND/OR Spark Sqoop NB: this is one approach, multiple tools exist as alternatives, and adding tools to this architecture can make your process faster, or more organized, or more secure, or have other benefits.