Implementing Elasticsearch - Lessons Learned Round-table discussion for those who have implemented the Elasticsearch and/or who are planning to implement it 101450 Kevin Antoff Senior Application Systems Engineer Wells Fargo & Company

Implementing Elasticsearch - Lessons Learned Who is the Moderator Get Started Questions About Wells Fargo Wells Fargo PS Search Environment Wells Fargo’s Cutover to Elasticsearch Wells Fargo’s Elasticsearch Implementation – things didn’t go as planned. Wells Fargo’s Success Reference Material for Elasticsearch Open floor for discussion

Who is the Moderator Kevin Antoff – Wells Fargo Employee - I’m an Senior Application Systems Engineer. I’ve been working in IT since 1990 and have been employed with Wells Fargo since 1996. I started working with Peoplesoft in 1999 with version HRMS 7 and PT 7.5x. I went away from Peoplesoft supporting applications in our HR Systems Linux environments for several years and I was re-introduced into the Peoplesoft space when we enabled the SES for Peoplesoft. More recently I’ve worked with Oracle in bringing the new Elasticsearch engine into Peoplesoft and have recently implemented this into Wells Fargo in June 2017. My current role is a PS Systems Admin. Personally: I come from a family of nine. I am an identical twin, I have sisters who are also identical twins. My wife’s mother is a twin and I am the proud parent of twin boys who are now 22. I enjoy everything about the outdoors; Golfing, fishing, hiking, scuba diving and I am an avid sports fan. The only sport I’m not a big fan of is Pro basketball although college basketball is great to follow.

Elasticsearch - Get Started Questions Who has implemented Elasticsearch? Who hasn’t? What other search engines do you currently use? Does anyone want to share about there experience with Elasticsearch? What issues have you had with implementing Elasticsearch? What if any apprehension do you have with moving to Elasticsearch? Has anyone extended the use of Elasticsearch to do monitoring? What else would you like to talk about in regards to search?

About Wells Fargo Wells Fargo & Company (NYSE: WFC) is a diversified, community-based financial services company with $1.9 trillion in assets. Founded in 1852, Wells Fargo provides banking, insurance, investments, mortgage, and consumer and commercial financial services through more than 8,600 locations, 13,000 ATMs, online (wellsfargo.com), and mobile devices. Employees: 269,000+ team members in 36 countries and territories across our more than 90 businesses Early Adopter: Initially installed PeopleSoft version 2.0 in 1993 Modules: HR, Payroll for NA, Benefits Administration, eRecruit, and related Self-Service Apps and HR Help Desk. Home-grown Manager Self-Service App. Stay Current Strategy: With the upgrade to HCM 9.2, Wells Fargo committed to staying current, by applying two PUM images per year. Current Installation: HCM: 9.2.19 CRM: 9.2.12 PT: 8.55.16

Wells Fargo PS Search Environment Wells Fargo has many Peoplesoft environments. In the HR space we have two that use the Elasticsearch PS search environment. There is a single index in our Helpdesk implementation that utilizes ES CR_CO_SOLUTIONS In HCM we have Recruiting indexes and indexes for Talent Management Job Data, Job Opening, and Applicant to name a few We have a 3 node cluster in production for Elasticsearch Our servers are all physical servers using Tier 2 San storage We have a Sandbox, DEV, SIT/UAT, BCP and PROD environments for Elasticsearch For monitoring we use Kibana and Marvel to view into Elasticsearch There is a separate license acquired outside of the PS stack, it must be acquired for Elasticsearch Elasticsearch will give you a free license capable of monitoring a single cluster.

Wells Fargo PS Search Environment DEV 1 Physical Server – 8 CPU, 64gb RAM, 2TB San Storage Tier 2 SIT/UAT 3 Physical Servers – 8 CPU, 128gb RAM, 2TB San Storage Tier 2 F5 Load Balancer – VIP/Pool – Pass-through ssl BCP PROD 2 Physical Servers – 8 CPU, 128gb RAM, 2TB San Storage Tier 2 1 Physical Server – 8 CPU 128gb RAM, 11 TB Physical Disk F5 Load Balancer – VIP/Pool – Pass through ssl

Wells Fargo Cutover to Elasticsearch On June 4 Wells Fargo switched one index in our HR Helpdesk module to use the Elasticsearch - CR_CO_SOLUTIONS Wells Fargo went into production on June 6 with multiple indexes using Elasticsearch in HCM. We were able to leverage the Oracle Implementation technique to leave SES running and cutover to Elasticsearch after the new indexes in Elasticsearch were built. By having two PS instances we were able to use the Helpdesk instance which wasn’t utilized as much for search as a test before switching all the indexes for HCM. Another way to test is to use the User/Search Instance page

WellsFargo ‘s Implementation - Things Didn’t Go As Planned We recommend having all the patches + most current dpk release for Elasticsearch installed. For us, having the em dash and reserved character patches was critical. Without them you have the potential for issues with searches and attachments processing. We began to look at other things associated with reserved characters too. Applicants who submitted a resume with reserved characters in the document name would get dropped because of errors in processing the resume. We found that when an attachment had brackets in document name that it would cause the GetAttachment to fail. With reserved characters in the document name Oracle would change those special characters to strings. This would increase the length of the bind variable significantly and would cause sql errors when it grew too large. We have workarounds to fix these issues in production and Oracle has already provided patches which we are testing in our lower environments.

Wells Fargo’s Success Since implementing Elasticsearch we haven’t looked back. Elasticsearch is everything it was promised to be On SES our Job Agent job had abended 21 times since the beginning of the year. This job has not failed since implementing Elasticsearch! The Job Agent job was running for 6+ hours with SES and is now completing in 3.5 hours Server CPU and memory on our Elasticsearch servers are running really low Disk storage for storing the index data is minimal compared to what it was with SES Overall performance of the Elasticsearch has been wonderful Using Kibana and Marvel to monitor our Elasticsearch stack has worked out well

Reference Material for Elasticsearch Reserved/Special characters can cause issues. Reserved characters according to Elasticsearch https://www.elastic.co/search?q=special+characters Oracle’s main Elastic Search page https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=85818723264344&id=2189508.1&_afrWindowMode=0&_adf.ctrl-state=1d9a5jhxvg_57 This page has the PeopleTools Elasticsearch Home Page, Elasticsearch on PT 8.56, Elasticsearch on PT 8.55 and Troubleshooting and Known Issues page. Collecting Information for Elasticsearch Troubleshooting https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=1d9a5jhxvg_98&_afrLoop=86658405921972

Reference Material for Elasticsearch On RH Release 7 – Oracle’s dpk fails - Installing the DPK Manually https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=1d9a5jhxvg_98&_afrLoop=86799880360285#FIX

Open floor for Round-table What issues/successes have you had with your Elasticsearch Implementation? What are your companies plans for implementing Elasticsearch? Question and Answer time What is on your mind?

101450