1. © 2015 IBM Corporation IT Analytics for System z IT Analytics on z Systems – IBM zAware 2.0 (z13 orderable feature) Yuk (Patrick) Chan, IBM Senior Software Engineer chanyuk@us.ibm.comchanyuk@us.ibm.com Twitter: @AboutPatrick 6/25/2015

2. © 2015 IBM Corporation 2 IT Analytics for System z Trademarks The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both. The following are trademarks or registered trademarks of other companies. * All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography. Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce. DS8000 ECKD FICON* GDPS* GPFS HiperSockets IBM* IBM (logo)* InfiniBand* Parallel Sysplex* PR/SM Redbooks* System x* System z* System z9* System z10* Tivoli WebSphere* Z9* z10 z10 Business Class z10 EC z/OS* z/VM* zEnterprise

3. © 2015 IBM Corporation 3 IT Analytics for System z 3 Notice Regarding Specialty Engines (e.g., zIIPs, zAAPs and IFLs): Any information contained in this document regarding Specialty Engines ("SEs") and SE eligible workloads provides only general descriptions of the types and portions of workloads that are eligible for execution on Specialty Engines (e.g., zIIPs, zAAPs, and IFLs). IBM authorizes customers to use IBM SE only to execute the processing of Eligible Workloads of specific Programs expressly authorized by IBM as specified in the “Authorized Use Table for IBM Machines” provided at www.ibm.com/systems/support/machine_warranties/machine_code/aut.html (“AUT”). www.ibm.com/systems/support/machine_warranties/machine_code/aut.html No other workload processing is authorized for execution on an SE. IBM offers SEs at a lower price than General Processors/Central Processors because customers are authorized to use SEs only to process certain types and/or amounts of workloads as specified by IBM in the AUT.

4. © 2015 IBM Corporation 4 IT Analytics for System z Agenda  Background – Problem determine: existing approach and new appraoch  Why the new approach (zAware)?  Why should you care?  zAware History  Analytic on Linux  zAware UI  zAware High Level View, Operating Requirement  Setup and Configuration  zAware Use Cases

5. © 2015 IBM Corporation 5 IT Analytics for System z IT Analytics - IBM zAware IBM System z Advanced Workload Analysis Reporter

6. © 2015 IBM Corporation 6 IT Analytics for System z Background – Problem Determination, the existing and new approach Traditional Problem Avoidance and Determination  Scenario based, from known problems.  Message or Message ID based  e.g. Kernel Panic  Threshold based  e.g. % of paging space available A new and different approach – Analytics  What’s expected? Deduces what’s unexpected (i.e. anomaly)?  How it works: Highly unexpected Highly expected Normal Logs ……………… Model Realtime Logs (Tons of) ……………… IBM zAware Analytics Algorithm: IBM Research Deep System z expertise IBM zAware Analytics Algorithm: IBM Research Deep System z expertise Logs representing Normal System behaviors IBM zAware UI So that you don’t need to learn the deep as much System z knowledge

7. © 2015 IBM Corporation 7 IT Analytics for System z Background – Why the new approach (zAware)?  Complex use case, complex system, complex software, complex interaction, complex EVERYTHING! Difficult to detect using traditional method  Problems that was never seen before.  Problems that involves multiple software, firmware, systems and hardware components.  Small problems/signs that manifest into big problem. Difficult to diagnose and isolate problems  Failure involving multiple software, firmware, systems and hardware components. How to find the component, system in error?  Volume of diagnose data is not humanly consumable. One company has 18.6M msgs/day -> 215/second Another has 2.46M msgs/day -> 28/second

8. © 2015 IBM Corporation 8 IT Analytics for System z Background – Why should you care? Significant failure -Noticed by users -Detected by traditional tools Shorter Time to diagnose the problem Time to fix the problem Without IBM zAware With IBM zAware Time to diagnose the problem Earlier problem detection AVOID or shorten time to recover from an early detected problem Shorter MTTR Better met SLA Shorter MTTR Better met SLA Small problems show up as anomaly, could show up across components.

9. © 2015 IBM Corporation 9 IT Analytics for System z zAware History zEC 12 3Q2012 zAware V1 Pattern recognition for z/OS OPERLOG messages Browser based graphical UI APIs for vendor integration zAware V2 Enhanced pattern recognition algorithm Enhanced graphical UI Updated APIs with additional analytic details z13 1Q2015 zAware V2, with MCF Supported Linux on System z, syslog messages. Supported grouping of similar Linux systems z13 Jun 26, 2015

10. © 2015 IBM Corporation 10 IT Analytics for System z Analytic on Linux  Using a group of Linux Systems to build a model of “expected behavior”.  Analyzing each Linux System independently against the model.  System X didn’t contribute to the model, but available for analysis.  Allows systems that are dynamic (comes and goes) in nature to benefit from IBM zAware immediately. System A Normal Logs ……………… Model System A Realtime Logs ……………… IBM zAware IBM zAware UI System B Normal Logs ……………… System C Normal Logs ……………… Similar systems and workload (A Model Group) System X Realtime Logs ……………… What is unexpected from System A? What is unexpected from System X?

11. © 2015 IBM Corporation 11 IT Analytics for System z Log Stream How anomaly are reported?  Model Group -> System -> Date -> Interval  What is an interval?  Analysis result provided every 2 minutes incoming logs.  Analysis result hardened every 10 minutes. Harden Result Analyze using 60 minutes of logs Harden Result, every 10 minutes Currently incoming logs: temporary result every 2 minutes  60 minutes of sliding window? Relationship are found for log messages within the same windows. Different OS might have different window size. Linux – 60 minutes z/OS – 10 minutes NOW Harden Result, 10 minutes Future 10:00 PAST 10:10 10:20 …….. 10:30 10:22

12. © 2015 IBM Corporation 12 IT Analytics for System z IBM zAware GUI – Interval View Height shows number of unique messageIDs Clicking on a bar drills down to Interval Color shows anomaly score

13. © 2015 IBM Corporation 13 IT Analytics for System z IBM zAware GUI - Heatmap, Group Aggregated analysis score for group with ability to drill down Monitor multiple plexes

14. © 2015 IBM Corporation 14 IT Analytics for System z IBM zAware GUI - Systems in a group Score by the Hour Score by the Day

15. © 2015 IBM Corporation 15 IT Analytics for System z IBM zAware GUI – Interval View with details Ids are generated 1 6 8 3 2 4 5 7

16. © 2015 IBM Corporation 16 IT Analytics for System z zAware High Level View z13 IBM zAware host Linux on system z z/OS IBM zAware Host Partition zAware Server IBM zAware monitored client Linux on system z z/OS IBM zAware Web GUI to monitor results z/VM

17. © 2015 IBM Corporation 17 IT Analytics for System z Operating Requirements – IBM zAware IBM zAware Serverz13 z/OS and zLinux IFL or CP (recommend 2 partial IFL or CP) zEC12, zBC12 z/OS IBM zAware, Linux ClientLinux level SLES 10 or later RHEL 6 or later Native or as z/VM guest Linux syslog daemon (/var/log/messages) RFC5424 format Supports: rsyslog, syslog-ng Unsupported: syslog relay (direct connection to zAware)

18. © 2015 IBM Corporation 18 IT Analytics for System z zAware Setup  Purchase and install the IBM zAware Feature Code (firmware)  Loaded from the Support Element  Update firmware: SE, HMC, CDU (Concurrent Driver Upgrade), MCF/MCLs  Define I/O using HCD or HCM.  Defines zAware Partition (similar to other partitions)  Define Profile with “zAware Mode” Assign processors Assign storage size Assign network HiperSockets, shareable OSA ports or IEDN IP Address  Define storage on the zAware UI  Requires EDKD DASD  Configure security / user credential and roles on the zAware UI  Configure analytic options  Configure monitor clients

19. © 2015 IBM Corporation 19 IT Analytics for System z References  IBM System z Advanced Workload Analysis Reporter (IBM zAware) Guide SC27-2623-00  http://www.ibm.com/systems/z/os/zos/bkserv/r13pdf/#E0Z http://www.ibm.com/systems/z/os/zos/bkserv/r13pdf/#E0Z  Or IBMResourceLink Library → zEC12 → Publications  IBM System z Advanced Workload Analysis Reporter (IBM zAware) Guide V2.0 SC27-2632-00  https://ibm.biz/BdEW9J https://ibm.biz/BdEW9J  Redbook Web Doc: IBM zAware Migration from an IBM zEC12 to an IBM z13  http://www.redbooks.ibm.com/abstracts/tips1296.html?Open http://www.redbooks.ibm.com/abstracts/tips1296.html?Open  Redbook: Extending z/OS System Management Functions with IBM zAware  http://www.redbooks.ibm.com/abstracts/sg248070.html?Open http://www.redbooks.ibm.com/abstracts/sg248070.html?Open  IBM Mainframe Insights blogwww.ibm.com.systemzwww.ibm.com.systemz  The Journey to IBM zAware http://www.ibm.com/connections/blogs/systemz/entry/zaware?lang=en_us  zAware Installation and Startup http://www.ibm.com/connections/blogs/systemz/entry/zaware_installation?lang=en_us  Top 10 Most Frequently Asked Questions About IBM zAware http://www.ibm.com/connections/blogs/systemz/entry/zawarefaq?lang=en_us  IBM zAware Demo http://www.ibm.com/connections/blogs/systemz/entry/zawaredemo?lang=en_us

20. © 2015 IBM Corporation 20 IT Analytics for System z Sample User Cases zOS

21. © 2015 IBM Corporation 21 IT Analytics for System z Identify anomaly Which z/OS image is having unusual message patterns? Yellow and dark blue on CB88 When did the behavior start? Around 2:30

22. © 2015 IBM Corporation 22 IT Analytics for System z Drill down - configuration error What component is having the problem? Drill down indicates 900 IRRC131I and IRRC144I messages per interval. A review of SYSLOG showed that this was the result of work being performed in the LDAP address spaces. Further analysis showed that the LDAP PC Callable Interface was not enabled. At 6:40, the function was enabled, and the 131I and 144I messages are no longer generated. Impact Unnecessary messages blocking ability to see anything else. Impacts ability to look at the console When did the behavior start? Around 2:30

23. © 2015 IBM Corporation 23 IT Analytics for System z Identify unusual behavior – quickly Which z/OS image is having unusual message patterns? Recurring yellow and dark blue on CB8C When did the behavior start? After an IPL at 13:30

24. © 2015 IBM Corporation 24 IT Analytics for System z Identify unusual behavior – quickly Which subsystem or component is abnormal? Examine high-scoring messages When did the behavior start? When did the messages start to occur? Were similar messages issued previously? Easily examine prior intervals or dates Moving left and right by interval shows messages due to TNPROC being cancelled by TCP/IP

25. © 2015 IBM Corporation 25 IT Analytics for System z Identify unusual behavior after a change Are unusual messages being issued after a change? New / updated workload (OS, middleware, apps) was introduced Detected as yellow bars Once messages confirmed as ok, can rebuild your system model, and workload now understood as “normal.” A new model included several days of new workload

26. © 2015 IBM Corporation 26 IT Analytics for System z Sample User Cases zLinux

27. © 2015 IBM Corporation 27 IT Analytics for System z Demo – 3 mirrored Linux systems got similar authentication error After an incident, zAware helps narrow down problem that traditional method wouldn’t. 3 mirrored systems has anomaly at the same time?!?

28. © 2015 IBM Corporation 28 IT Analytics for System z Note: Detailed view is removed for security reason. Detailed view shows all 3 systems has similar messages caused by login from the same host. This is not necessary a problem. This could means a new employee trying to login to a system that doesn’t have the proper UserId setup, or this could be an cyber attack.

29. © 2015 IBM Corporation 29 IT Analytics for System z Demo – System Upgrade and Restart Logger offline Testing System or Logger offline System booting zAware paints a picture of “what happened” based on anomaly.

30. © 2015 IBM Corporation 30 IT Analytics for System z Demo – Kernel Point dereference error zAware didn’t receive log data System restarted Kernel Reference Pointer Error

31. © 2015 IBM Corporation 31 IT Analytics for System z Demo – Kernel Pointer dereference error

32. © 2015 IBM Corporation 32 IT Analytics for System z Demo – Repeating Error More Anomalous than normal

33. © 2015 IBM Corporation 33 IT Analytics for System z Backup

35. © 2015 IBM Corporation 35 IT Analytics for System z Legal Disclaimer © IBM Corporation 2015. All Rights Reserved. The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete: Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete: All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.http://www.ibm.com/legal/copytrade.shtml If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete: Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. If you reference Java™ in the text, please mark the first use and include the following; otherwise delete: Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete: Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete: Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete: UNIX is a registered trademark of The Open Group in the United States and other countries. If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete: Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.