Modern data architecture at scale in the cloud : Best practices of Serverless, lambda and microservices architecture Prakriteswar Santikary, PhD Vice President and Global Chief Data Officer
ABOUT ERT Founded in 1977; privately held Supporting Pharmacos, Biotechs & CROs around the world Operations in 12 countries 2500+ employees MINIMIZING RISK & UNCERTAINTY, SO YOU CAN MOVE AHEAD QUICKLY WITH CONFIDENCE │ © Copyright ERT 2018
Agenda 01 Definition Characteristics Advantages Disadvantages Monolithic Architecture Microservices Architecture Serverless Architecture Definition Characteristics Advantages Disadvantages Definition Characteristics Advantages Disadvantages Definition Characteristics Advantages Disadvantages Here are a few key learnings we hope to provide with this presentation that may inspire these organizations to embrace this change as an opportunity to innovate. Paradigm Shift: Companies embarking on new innovation initiatives to determine feasibility of mHealth solutions for clinical research Audience Questions: How many of you currently employ mHealth technologies in your clinical trials? How may of you currently utilize Actigraphy and/or Continous Monitoring devices in your clinical trials? How many of you work in organizations that are now tasked with POCs or pilots using mHealth devices? Data Optimization: Enhancing clinical research with mHealth data requires new techniques against non-conventional sources. mHealth data alone does not enhance clinical research. Practical Applications: Applied use of mHealth in select therapeutic areas in combination with other clinical source data and/or ways where mHealth data may be combined for more visual insights | Copyright ERT 2018
Monolithic Architecture – a refresher TIPS ; Fresh business idea, needs quicker turn around, small engineering team – start with monolith – build simple and lightweight application
Monolithic Architecture – Weaknesses and challenges
Software Complexity has been growing steadily 01 World of Big Data 02 DevOps and Automation Small incremental release Continuous deployment Continuous delivery Quick time to market Competitive pressure Multiple data sources Multiple channels Multiple data types Scale requirements Real time insight Big Data (unstructured) Forces behind Microservices Architecture 04 Cloud/Serverless computing 03 Emerging Technology New Programming language New Programming framework NoSQL databases API-first design Cloud-first design Focus on code, not on infrastructure – less IT Serverless deployment Pay per use Cost reduction │ © Copyright ERT 2018
Monolith to Microservices Transition – loosely coupled, independent services communicating via lightweight protocol TIPS : Microservices architecture is not a silver bullet for a bad software design. Design discipline must be exercised as to when to use monolith Vs Microservices.
Monolith is hard to scale; Microservices enable y-axis scaling by breaking the application into smaller independent services. Each service is indepedently scaled based on need, enabling horizonal scaling.
Microservices Logical Architecture
Let’s take a n-tier e-commerce application to explain monolith and microservices
Break it down into microservices │ © Copyright ERT 2018
Microservices architecture – a process view │ © Copyright ERT 2018
Microservices Architecture- Advantages │ © Copyright ERT 2018
Microservices Architecture- DisAdvantages │ © Copyright ERT 2018
Microservices architecture Design patterns 01 Decomposition Pattern 02 Deployment Pattern Service per host/VM Multiple services per host/VM Serverless Deployment On-premise Hybrid (with Cloud) By Business Capability By business domain Single responsibility principle Independently deployable Small service/small team You build it, you own it Domain-driven design Microservices Best Practice 04 Monitoring/Tracing Pattern 03 Data Management Pattern Polyglot persistence Database per service Data consistency Event sourcing Eventual consistency CAP theorem Log Aggregation Audit logging Distributed Tracing Exception Tracking Health-check API │ © Copyright ERT 2018
Microservices architecture – best practices │ © Copyright ERT 2018
Microservices Architecture – a summary view Characteristics Benefits Challenges Decentralized Independent Do one thing well Polyglot API-first Design You build it; You own it Agility Innovation Quality Scalability Availability Distributed systems Monolith->Microservices transition not easy Organizational issues (DevOps) Skillsets TIPS : Microservices is not a silver bullet. You still need to follow sound software design principles. Here are a few key learnings we hope to provide with this presentation that may inspire these organizations to embrace this change as an opportunity to innovate. Paradigm Shift: Companies embarking on new innovation initiatives to determine feasibility of mHealth solutions for clinical research Audience Questions: How many of you currently employ mHealth technologies in your clinical trials? How may of you currently utilize Actigraphy and/or Continous Monitoring devices in your clinical trials? How many of you work in organizations that are now tasked with POCs or pilots using mHealth devices? Data Optimization: Enhancing clinical research with mHealth data requires new techniques against non-conventional sources. mHealth data alone does not enhance clinical research. Practical Applications: Applied use of mHealth in select therapeutic areas in combination with other clinical source data and/or ways where mHealth data may be combined for more visual insights | Copyright ERT 2018
Serverless Architecture Focus on your application, not on infrastructure Characteristics Benefits Challenges No server to maintain No server to provision Focus on code Ephemeral; Stateless Event-driven Horizontal scaling (auto) Pay per use; no idle time Faster innovation Lower operational cost Reduce operational management Speed to market Fosters innovation Skillsets Vendor lock-in concerns Lack of operational tools Architectural complexity Restriction on execution time Restriction on memory TIPS : Build scalable platform and applications using a combination of serverless and microservices in the cloud Here are a few key learnings we hope to provide with this presentation that may inspire these organizations to embrace this change as an opportunity to innovate. Paradigm Shift: Companies embarking on new innovation initiatives to determine feasibility of mHealth solutions for clinical research Audience Questions: How many of you currently employ mHealth technologies in your clinical trials? How may of you currently utilize Actigraphy and/or Continous Monitoring devices in your clinical trials? How many of you work in organizations that are now tasked with POCs or pilots using mHealth devices? Data Optimization: Enhancing clinical research with mHealth data requires new techniques against non-conventional sources. mHealth data alone does not enhance clinical research. Practical Applications: Applied use of mHealth in select therapeutic areas in combination with other clinical source data and/or ways where mHealth data may be combined for more visual insights | Copyright ERT 2018
SERVERLESS ARCHITECTURE – FEW EXAMPLES | Confidential – Internal Use Only
SERVERLESS ARCHITECTURE USING MICROSERVICES | Confidential – Internal Use Only
Lambda Architecture – execution of real time and batch together Characteristics Benefits Challenges Real time data processing Batch data processing Merging of real time and batch in single architecture Architectural complexity skillsets Serve both real and batch use cases simultaneously TIPS : One architecture supports both real time and batch Here are a few key learnings we hope to provide with this presentation that may inspire these organizations to embrace this change as an opportunity to innovate. Paradigm Shift: Companies embarking on new innovation initiatives to determine feasibility of mHealth solutions for clinical research Audience Questions: How many of you currently employ mHealth technologies in your clinical trials? How may of you currently utilize Actigraphy and/or Continous Monitoring devices in your clinical trials? How many of you work in organizations that are now tasked with POCs or pilots using mHealth devices? Data Optimization: Enhancing clinical research with mHealth data requires new techniques against non-conventional sources. mHealth data alone does not enhance clinical research. Practical Applications: Applied use of mHealth in select therapeutic areas in combination with other clinical source data and/or ways where mHealth data may be combined for more visual insights | Copyright ERT 2018
Lambda architecture- Physical view on aws (Serverless) │ © Copyright ERT 2018
KEY TAKEAWAYS FROM TODAY’S DISCUSSION Monolithic architecture Simple application; quick turn around – good for simple lightweight application – validate ideas quickly. Not ideal for complex and growing applications – scalability is the biggest issue. Microservices Complex application; scalability need is high; strong engineering skillsets available; distributed computing is complex; Microservices is no silver bullet – still depends on sound software design. Serverless Architecture Cloud-native; no HW to maintain; focus on code, not on infrastructure; works well with microservices; │ © Copyright ERT 2018
Questions? Prakriteswar Santikary, PhD prakriteswar.santi@ert.com ? …