Download presentation
Presentation is loading. Please wait.
1
Mastering Person Data:
Making The Data About You, Work For You #AIM4
2
#AIM4 >_whoami Jim K Application Architect, The Ohio State University Founder, Storm Labs Limited
3
#AIM4 Blast from the past
4
HighEdWeb 14’ HighEdWeb 15’ #AIM4
Let Your Data Run Free and Run Your University Plan and vision for Enterprise Integration Platform HighEdWeb 15’ Data “Silo” SMACdown : Enter the EIP Year 1 review of EIP Turning point-to-point connections into reusable enterprise web services 2014 – Everyone had their own connections, firewalls, and common data elements - Everyone needed data about people... A majority needing the same information 2015 – First to move in was the PersonBioDemo services with a quick adoption rate
5
#AIM4 - The 2nd year goal was to hit 100% coverage for all enterprise services, which was met. - Currently OSU is moving their Financial ERP to Workday and all integrations will be accessed via EIP
6
What types of people are there?
#AIM4 What types of people are there?
7
Students Employees Guests Sponsored Guests #AIM4 Prospective
Undergraduate Graduate Professional Employees Academic Administrative Research Medical Center Guests Spouse Dependents Sponsored Guests Consultants Traveling Professor Traveling MD Auditors
8
What data elements are important to you?
#AIM4 What data elements are important to you?
9
Student Employee #AIM4 Academic Information Appointments Program
Degree Plan Sub-Plan Location Affiliation of ”student” Address Home Knowledge Management Employee Appointments Department Department Level Title Job Code Salary Supervisor Affiliation of “employee” Address Home Business Knowledge Management KM Data is a system about research, dissertations, publish works, etc. Student or Employee (Professor/Researcher) may or may not have KM Data
10
#AIM4 Guest Appointments Department Job Code Address Business
11
Employee Student Guest #AIM4 Appointments Academic Information
Department Department Level Title Job Code Salary Supervisor Affiliation of “employee” Address Home Business Knowledge Management Student Academic Information Program Degree Plan Sub-Plan Location Affiliation of ”student” Address Home Knowledge Management Laying all of these types of people out leads us to the next question. Where do we find the data? Guest Appointments Department Job Code Address Business PS SIS PS HR IDM KM
12
What is an example use case?
#AIM4 What is an example use case?
13
#AIM4
14
What is an example use case?
#AIM4 What is an example use case?
15
Department of Education
#AIM4 Department of Education Creating a hub for companies to find faculty that had expertise in specific areas Company A: I need to find a weld engineering expert All state members currently use SFTP on a scheduled basis Ohio State is the only school with a live data feed via API PS SIS KM PS HR Ohio Department of Education
16
Appointment Look Up Service
#AIM4 Appointment Look Up Service Many projects and initiatives needed to look up appointments for a given person PeopleSoft service exists for single person lookup Requires PS Access per consumer Requires firewall request Later several project teams requested bulk look up of appointments by department PS HR IDM
17
Bulk Population Loads #AIM4
Increasing cloud service acquisition requires loading entire populations Company A: I need to find a weld engineering expert Some can do SFTP Some can do web services Primary route Create a single source for common approach to future loads Previous load times were approaching 13 hours with existing technologies processes PS SIS IDM PS HR
18
#AIM4 Geek time
19
#AIM4 What the….tech?
20
#AIM4
21
#AIM4 1. Data Assembly
22
Build elastic Bulk API Call
#AIM4 PS HR Select HR Core MV Build JSON Per Person Build elastic Bulk API Call Execute Bulk API Call Materialized Views EIP Attempt 1 at load elastic search. Example of a POC that makes its way to production. Testing NodeJS for many different jobs. The initial load was built with varying levels of EIP up and running.
23
Build elastic Bulk API Call
#AIM4 PS HR Select HR Core MV Build JSON Per Person Build elastic Bulk API Call Execute Bulk API Call Materialized Views GC Errors EIP Framework Misuse 1. Why ETL to this scale via web service when running LOCAL TO THE DB MACHINE 2. Holding all of the data in memory caused, what was thought hard to do, doable. Cause NodeJS to crap out on V8 garbage collection. 3. Node is meant to be non-blocking, which isn't optimal for ETL Inefficient Flow
24
#AIM4 PS HR Select Each MV Load to Local Table Build JSON Per Person
Build elastic Bulk API Call Execute Bulk API Call Build JSON Per Person Load JSON Docs to Local DB PS HR Materialized Views 1.Create local copies of MVs 2. Looping through people, look ups are instance as the source is local 3. Load assembled json to table (hot-back up) PM
25
#AIM4 2. Data Consumption
26
#AIM4 PS HR Select Each MV Load to Local Table Build JSON Per Person
Build elastic Bulk API Call Execute Bulk API Call Build JSON Per Person Load JSON Docs to Local DB PS HR Materialized Views Knowing this is the process for creating the data store, HOW IS IT CONSUMED? PM
27
#AIM4
28
HR Core, KM, Acad Info, Buss Addr
#AIM4 EIP Dept. of Edu HR Core, KM, Acad Info, Buss Addr 1:1 Appts By Emplid HR Core, Appts 1:Many Via ESB we have a set of APIs that proxy the elastic queries to a RESTful service and add security. Secured so that only the elements required are returned. Bulk Loads HR Core, Appts 1:Many
29
~13 hr ~1.5 min ~65k people #AIM4 Bulk Loads PS IDM SIS Load Process
KM ~65k people Here we have old busted and new hotness. 65k people and their complex data return. Sub-documents. ~1.5 min Bulk Loads HR Core, Appts
30
#AIM4 Going forward
31
Ad-Hoc Person Searches Reporting Opportunities
#AIM4 Functional Find People Change search interface and data source Ad-Hoc Person Searches Micro-services tailored to need Reporting Opportunities Analytic queries Dashboarding Via Elastic Kibana Find People – Looking into single field person search Reporting – You don’t know what you don’t know
32
Technical MS SQL Server 2016 Elasticsearch .Net Assembler #AIM4
JSON Data Types Pre-Assemble sub documents via scheduled job Elasticsearch Better leverage elastic APIs Alias, Queries, etc. .Net Assembler Try extensible data model Add/Remove elements to model via CLI
33
#AIM4 Questions?
34
#AIM4 Thank you! #AIM4 Jim K -
Similar presentations
