1. Enterprise Best (Community) Practices Workshop
OGF 21 - Seattle
Nick Werstiuk
October 18, 2007

2. OGF IPR Policies Apply
“I acknowledge that participation in this meeting is subject to the OGF Intellectual Property Policy.”
Intellectual Property Notices Note Well: All statements related to the activities of the OGF and addressed to the OGF are subject to all provisions of Appendix B of GFD-C.1, which grants to the OGF and its participants certain licenses and rights in such statements. Such statements include verbal statements in OGF meetings, as well as written and electronic communications made at any time or place, which are addressed to:
the OGF plenary session,
any OGF working group or portion thereof,
the OGF Board of Directors, the GFSG, or any member thereof on behalf of the OGF,
the ADCOM, or any member thereof on behalf of the ADCOM,
any OGF mailing list, including any group list, or any other list functioning under OGF auspices,
the OGF Editor or the document authoring and review process
Statements made outside of a OGF meeting, mailing list or other function, that are clearly not intended to be input to an OGF activity, group or function, are not subject to these provisions.
Excerpt from Appendix B of GFD-C.1: ”Where the OGF knows of rights, or claimed rights, the OGF secretariat shall attempt to obtain from the claimant of such rights, a written assurance that upon approval by the GFSG of the relevant OGF document(s), any party will be able to obtain the right to implement, use and distribute the technology or works when implementing, using or distributing technology based upon the specific specification(s) under openly specified, reasonable, non-discriminatory terms. The working group or research group proposing the use of the technology with respect to which the proprietary rights are claimed may assist the OGF secretariat in this effort. The results of this procedure shall not affect advancement of document, except that the GFSG may defer approval where a delay may facilitate the obtaining of such assurances. The results will, however, be recorded by the OGF Secretariat, and made available. The GFSG may also direct that a summary of the results be included in any GFD published containing the specification.”
OGF Intellectual Property Policies are adapted from the IETF Intellectual Property Policies that support the Internet Standards Process.
IPR Notices Note Well for OGF meetings 2

3. Session Objective Update on Activity with Best Practices
Where do things sit on the first best practice
Call for Participation
Ask for input and feedback on the document
Ask for feedback on other best practice topics 3

4. Agenda Recap – how did we get here ? Context/Definitions/Scope
Community Practice for Enterprise Grid Deployment and Management
Problem #1 – Deployment and Configuration management
Problem #2 – Maximizing Utilization and Efficiency
Problem #3 – Monitoring, Troubleshooting and Reporting
Conclusions and Next Steps 4

5. Recap Best Practice session at OGF 19
Community generated list of potential topics
Surveyed Attendees to develop priorities
Defined 1st priority as “Best Practices to Deploy and Manage an Enterprise Grid
OGF 20 Session – feedback on what the practices should be, are we looking at the right questions
End User Research - ongoing
Documentation of the user information into the first document 5

6. Initial List of Best Practice Areas#2
Application Suitability for Grid
Application Porting to Grid
Grid in Financial Services
Increasing Infrastructure Utilization
Grid Deployment and Management
Grid Financial Justification
Grid Service Level Management
Vendor Selection Process
What type of Grid to use ?
License Optimization and Management
Use of External Grids integrated with Enterprise
Data Management and Integration, Security, Clean Up
Integration of Grid Security with Enterprise Environment #1 #3 6

7. Recap OGF 20 Workshop Recommendations
Use this best (or community) practice activity as a way to catalog and comment on the approaches being used today in the Enterprise environments
Focus on the approaches (tools) and limitations on the approaches, not which approach is the absolute ‘Best’ 7

8. Context – Intended Audience
Enterprises looking at deploying a grid infrastructure
Trying to understand some of the approaches and techniques currently being used in the community to deploy and manage grid infrastructure
Existing Enterprise grid users
Looking for additional information and insight on how other organizations are tackling problems and issues around deployment and management of their grid infrastructure 8

9. Context - Scope
Typical and current use cases and approaches for Grids within enterprise environments.
80/20 Rule Applies
The majority of these Enterprise grids are used for the execution of compute and data intensive workloads within the boundaries of a single enterprise
Have chosen to focus on these types of deployments. 9

10. Context – Enterprise Scope
Commercial Users of grids for compute and data intensive processing
Application, Infrastructure ISV’s used for compute, data intensive grids.
Organization Focus
‘Data Center’ High Performance
Commercial users of grids beyond high performance applications
Data Center automation, management, virtualization etc, etc
End User Ecosystem
Organization Type 10

11. Context – Scope Vertical Market Examples Horizontal Examples
Industrial Manufacturing - MDA
Electronics - EDA
Financial Services – Investment Banking Risk, Pricing
Life Sciences
Oil and Gas
Horizontal Examples
Analytics
Business Intelligence 11

12. Example - Telco Billing Environment
200xLinux servers
2-CPU
3.2GHz
8GB RAM
All Markets concurrently
3 x SMP, 32 CPU
SMP app
SAN >30 Target Markets
Billing CRM Database
Switches Billing Data
3 x SMP, 32 CPU
SMP with RDBMS
app – can process 2-3 Target Markets concurrently
SAN >30 Target Markets
Billing CRM Database
Switches Billing Data
Challenges
Current SLA 24-48
Desired SLA 4 hours
Additional SMP to meet SLA = $12M
Results
Desired SLA Met
Grid Resources = $1.2M 12

13. Context – Out of Scope
Enterprise or Data Center Grids that are used primarily for large scale transactional environments.
Emerging end user deployment types, not as common as the compute and data intensive use cases that are widely deployed across vertical industries.
Data management and access issues in compute intensive grid environments.
The focus of this document is primarily on the deployment and management of the server infrastructure and the applications that run on that infrastructure, not the associated data management issues. 13

14. Alignment and Discussion
Comments on the Focus Area? 14

15. Context – Drivers and Impact
Business Agility/Flexibility
Apply resources flexibly to meet business needs
Infrastructure Cost Challenges
Drive to scale out infrastructure to meet price/performance needs
End Result = Management Complexity
Large scale‑out architectures (100s to 1000s of servers) have introduced severe complexity and manageability challenges 15

16. Problem Area #1
Deployment and Configuration of the servers in the Grid:
Initial deployment and ongoing configuration management of large numbers of servers is a significant challenge.
Problem magnified in dynamic environments, where the ‘personality’ of the resources needs to change to meet workload demands and business policies.
Complexity of the environment grows in multiple dimensions based on key variables like number of applications, type of application pattern, number of resources, number of user groups.
Key Discussion Question:
What tools and approaches are organization using to manage the provisioning and update of 1000’s of servers at a time in their enterprise Grid?
What tools and approaches are people using to deploy their applications and required application and middleware configuration onto their Enterprise Grid?
What are the current benefits and limitations of these different approaches? (needs work) 16

17. Additional Issues Faced
Remote Install Requirements and scalability
Heterogeneous (and Multiple Layers of) Resources
Hardware
Operating Systems
Middleware
Applications
Configuration Management
Updates and Patching
Tool Costs
Pre Integrated ‘Stacks’ vs Best of Breed Integrations 17

18. Typical Configurations
Disk-full server nodes
The Operating System is installed onto a disk directly on the compute node.
In this type of configuration a standard Ethernet network is used and a boot image is loaded over the network
Disk-less server nodes
Network Attached Storage – Operating system is installed on network attached storage and server is booted over the network. (Boot over SAN, Infiniband)
In Memory OS Deployment – Operating system is installed into memory in the compute node
Desktop Compute nodes
Decentralized deployments 18

19. Typical Configurations
Package Based
Golden Image
VM Appliance 19

20. Approaches and Tools Used
Commercial Provisioning/Data Center Automation
Tools providing a comprehensive suite of operating system provisioning and application configuration management.
Typically used in enterprises for server environments beyond the grid infrastructure
Commercial Cluster Management
High performance computing cluster management tools.
Open Source Cluster Management/Provisioning
A variety of options from the academic and research oriented HPC environments, built by the users in these institutions to deploy their clusters and grid. 20

21. Approaches and Tools Used
Open Source Configuration Management Tools
Variety of tools have come from the research community grid deployments, and are a potential option for enterprise users.
These tools typically operate on the configuration of the nodes once an Operating System is installed on the server.
Storage and Networking Based Deployment
Storage and networking vendors providing tools to enable the deployment and configuration of the grid nodes. 21

22. Approaches and Tools Used
OS Specific Tools provided by the Operating system vendor
Operating system vendors provide the capability for their users to deploy the Operating system onto server hardware.
Hardware Provider Specific Solutions
Server vendors typically provide their customers with a provisioning and hardware management tool to enable them to manage their specific brand of hardware.
“Home grown” developed installation tools
Many organizations have built out their own provisioning tools to meet their specific enterprise needs. 22

23. Approaches and Tools Used
Grid Workload Managers
Workload managers used in the enterprise grid environments have some deployment management capabilities built into their offerings.
Enables file transfer, and compute service installation to be managed via services within the grid.
Once the workload manager is installed, then these remote execution and management facilities can be used to address many of the typical application deployment use cases on the grid.
Virtual Machine Based Environments
Emerging approach in the Enterprise Grid community is to leverage VM’s as the means to deploy and configure the operating system on the grid nodes. 23

24. Anything Missing ?
Are there other approaches being used ? 24

25. Problem Area #2 Maximizing Grid Resource Utilization and Efficiency:
Competing goals within the organization, the infrastructure owners typically want to maximize their resource utilization, while Application owners want the best Service Levels.
As diverse workloads are deployed into the grid infrastructure, the problem of efficient resource utilization mapped against business priorities becomes magnified.
Key Discussion Question:
What tools and approaches are people using to maximize the utilization of their enterprise Grid, while managing Service Levels for individual Application Users/Business Units 25

26. Additional Issues Faced
Unlimited (infinite) demand with limited Supply
Balance prioritization based on business policies with the ability to take maximum benefit of available resources
Complexity is introduced in mixed application and workload environments, where a variety of workloads may be deployed onto the same infrastructure.
Short running batch jobs
Long running batch jobs.
Sequential vs. Parallel Jobs
Workflows of Jobs.
Compute Service Oriented
Interactive User Sessions 26

27. Typical Configurations
Application Specific Resources
Pool of resources dedicated to a single application or business unit
Shared Resources – Single Workload Type
Resources pooled across similar application pattern and business units
Shared Resources – Multiple Workloads
Dedicated and Opportunistic Resources
Dedicated Servers
Opportunistic Servers
Opportunistic Desktops 27

28. Approaches and Tools Used
Demand Policy Dimensions
Priority
Response Time
Supply Policy Dimensions
Ownership
Lending
Borrowing
Sharing
Service Level is combination and manipulation of both 28

29. Approaches and Tools Used
Application Specific Workload Managers
Job/batch environments
Service Oriented Application Environments
Meta Schedulers
Layer on top of the clusters to distribute workload to different resources
Resource Managers
Application independent resource allocation and control. 29

30. Anything Missing ?
Are there other approaches being used ? 30

31. Problem Area #3 Monitoring, Troubleshooting and Reporting:
Different End user, application owner and infrastructure management needs
Coordinated across the layers – different groups (application and infrastructure) have their own views of what is happening to the applications and the underlying grid resources
Key Questions:
What tools and approaches are people using to monitor the infrastructure and the applications?
What tools and approaches are people using to integrate the different monitoring and configuration approaches to provide an integrated management infrastructure for their Enterprise Grid? 31

32. Additional Issues Faced
Real Time Resource Monitoring for Workload dispatch
Real Time Monitoring for Problem Diagnosis
Application Monitoring
Log collection for troubleshooting
Remote administrator access
Reporting on Application and Resources
Analysis of Historical Data 32

33. Additional Issues Faced
Correlation of information across different layers
Unified Presentation
Too Many Agents 33

34. Approaches and Tools Used
Workload Managers
Provide monitoring information on the grid resources.
Metrics on the physical resources are typically available in support of the workload allocation activities.
The workload managers also provide workload status and information, on job activity, performance and
Cluster Monitoring Tools
Tools available (usually included in a cluster management package) that enable monitoring of the physical resources in the cluster/grid
Reporting and Performance Analysis Tools
Leverage data collected from the workload manager and other sources into a data-mart/data warehouse enabling the creation of historical reports 34

35. Approaches and Tools Used
Hardware Monitoring and Management
Provided with the hardware by the server vendors
Intelligent Platform Management Interface (IPMI)
Network Management and Monitoring Frameworks
Extensible Application and infrastructure monitoring
Enterprise Management and Monitoring Tools
Leverage components or suites from the large Enterprise System Management vendors
Industry Specific Tools
Business process specific tools that monitor the grid and infrastructure/application outside the grid 35

36. Anything Missing
Are there other approaches being used ? 36

37. Conclusions
Diversity of tools and approaches indicates interoperability problems
Enterprises make choices between Stacks and Best of Breed Integration
Initiatives emerging to address complexity via configuration recipes and verification of everything from the HW up to the Application 37

38. Next Steps Write up the details on Monitoring approaches
Feedback, Review and Comments
Other Community Practice Areas
How can we scale this?
Volunteers and Participation 38

39. Initial List of Best Practice Areas#2
Application Suitability for Grid
Application Porting to Grid
Grid in Financial Services
Increasing Infrastructure Utilization
Grid Deployment and Management
Grid Financial Justification
Grid Service Level Management
Vendor Selection Process
What type of Grid to use ?
License Optimization and Management
Use of External Grids integrated with Enterprise
Data Management and Integration, Security, Clean Up
Integration of Grid Security with Enterprise Environment #1 #3
Still Valid / Anyone want to start one ? 39