1. Implementing Elasticsearch - Lessons Learned
Round-table discussion for those who have implemented the Elasticsearch and/or who are planning to implement it
101450
Kevin Antoff
Senior Application Systems Engineer
Wells Fargo & Company

2. Implementing Elasticsearch - Lessons Learned
Who is the Moderator
Get Started Questions
About Wells Fargo
Wells Fargo PS Search Environment
Wells Fargo’s Cutover to Elasticsearch
Wells Fargo’s Elasticsearch Implementation – things didn’t go as planned.
Wells Fargo’s Success
Reference Material for Elasticsearch
Open floor for discussion

3. Who is the Moderator
Kevin Antoff – Wells Fargo Employee - I’m an Senior Application Systems Engineer. I’ve been working in IT since 1990 and have been employed with Wells Fargo since I started working with Peoplesoft in 1999 with version HRMS 7 and PT 7.5x. I went away from Peoplesoft supporting applications in our HR Systems Linux environments for several years and I was re-introduced into the Peoplesoft space when we enabled the SES for Peoplesoft. More recently I’ve worked with Oracle in bringing the new Elasticsearch engine into Peoplesoft and have recently implemented this into Wells Fargo in June My current role is a PS Systems Admin.
Personally: I come from a family of nine. I am an identical twin, I have sisters who are also identical twins. My wife’s mother is a twin and I am the proud parent of twin boys who are now 22. I enjoy everything about the outdoors; Golfing, fishing, hiking, scuba diving and I am an avid sports fan. The only sport I’m not a big fan of is Pro basketball although college basketball is great to follow.

4. Elasticsearch - Get Started Questions
Who has implemented Elasticsearch? Who hasn’t?
What other search engines do you currently use?
Does anyone want to share about there experience with Elasticsearch?
What issues have you had with implementing Elasticsearch?
What if any apprehension do you have with moving to Elasticsearch?
Has anyone extended the use of Elasticsearch to do monitoring?
What else would you like to talk about in regards to search?

5. About Wells Fargo
Wells Fargo & Company (NYSE: WFC) is a diversified, community-based financial services company with $1.9 trillion in assets. Founded in 1852, Wells Fargo provides banking, insurance, investments, mortgage, and consumer and commercial financial services through more than 8,600 locations, 13,000 ATMs, online (wellsfargo.com), and mobile devices.
Employees: 269,000+ team members in 36 countries and territories across our more than 90 businesses
Early Adopter: Initially installed PeopleSoft version 2.0 in 1993
Modules: HR, Payroll for NA, Benefits Administration, eRecruit, and related Self-Service Apps and HR Help Desk. Home-grown Manager Self-Service App.
Stay Current Strategy: With the upgrade to HCM 9.2, Wells Fargo committed to staying current, by applying two PUM images per year.
Current Installation: HCM: CRM: PT:

6. Wells Fargo PS Search Environment
Wells Fargo has many Peoplesoft environments. In the HR space we have two that use the Elasticsearch PS search environment.
There is a single index in our Helpdesk implementation that utilizes ES CR_CO_SOLUTIONS
In HCM we have Recruiting indexes and indexes for Talent Management
Job Data, Job Opening, and Applicant to name a few
We have a 3 node cluster in production for Elasticsearch
Our servers are all physical servers using Tier 2 San storage
We have a Sandbox, DEV, SIT/UAT, BCP and PROD environments for Elasticsearch
For monitoring we use Kibana and Marvel to view into Elasticsearch
There is a separate license acquired outside of the PS stack, it must be acquired for Elasticsearch
Elasticsearch will give you a free license capable of monitoring a single cluster.

7. Wells Fargo PS Search Environment
DEV
1 Physical Server – 8 CPU, 64gb RAM, 2TB San Storage Tier 2
SIT/UAT
3 Physical Servers – 8 CPU, 128gb RAM, 2TB San Storage Tier 2
F5 Load Balancer – VIP/Pool – Pass-through ssl
BCP
PROD
2 Physical Servers – 8 CPU, 128gb RAM, 2TB San Storage Tier 2
1 Physical Server – 8 CPU 128gb RAM, 11 TB Physical Disk
F5 Load Balancer – VIP/Pool – Pass through ssl

9. Wells Fargo Cutover to Elasticsearch
On June 4 Wells Fargo switched one index in our HR Helpdesk module to use the Elasticsearch - CR_CO_SOLUTIONS
Wells Fargo went into production on June 6 with multiple indexes using Elasticsearch in HCM.
We were able to leverage the Oracle Implementation technique to leave SES running and cutover to Elasticsearch after the new indexes in Elasticsearch were built.
By having two PS instances we were able to use the Helpdesk instance which wasn’t utilized as much for search as a test before switching all the indexes for HCM.
Another way to test is to use the User/Search Instance page

10. WellsFargo ‘s Implementation - Things Didn’t Go As Planned
We recommend having all the patches + most current dpk release for Elasticsearch installed. For us, having the em dash and reserved character patches was critical. Without them you have the potential for issues with searches and attachments processing.
We began to look at other things associated with reserved characters too.
Applicants who submitted a resume with reserved characters in the document name would get dropped because of errors in processing the resume.
We found that when an attachment had brackets in document name that it would cause the GetAttachment to fail.
With reserved characters in the document name Oracle would change those special characters to strings. This would increase the length of the bind variable significantly and would cause sql errors when it grew too large.
We have workarounds to fix these issues in production and Oracle has already provided patches which we are testing in our lower environments.

11. Wells Fargo’s Success
Since implementing Elasticsearch we haven’t looked back.
Elasticsearch is everything it was promised to be
On SES our Job Agent job had abended 21 times since the beginning of the year. This job has not failed since implementing Elasticsearch!
The Job Agent job was running for 6+ hours with SES and is now completing in 3.5 hours
Server CPU and memory on our Elasticsearch servers are running really low
Disk storage for storing the index data is minimal compared to what it was with SES
Overall performance of the Elasticsearch has been wonderful
Using Kibana and Marvel to monitor our Elasticsearch stack has worked out well

12. Reference Material for Elasticsearch
Reserved/Special characters can cause issues. Reserved characters according to Elasticsearch
Oracle’s main Elastic Search page
This page has the PeopleTools Elasticsearch Home Page, Elasticsearch on PT 8.56, Elasticsearch on PT 8.55 and Troubleshooting and Known Issues page.
Collecting Information for Elasticsearch Troubleshooting

13. Reference Material for Elasticsearch
On RH Release 7 – Oracle’s dpk fails - Installing the DPK Manually

14. Open floor for Round-table
What issues/successes have you had with your Elasticsearch Implementation?
What are your companies plans for implementing Elasticsearch?
Question and Answer time
What is on your mind?

15. 101450