1. Under the Covers of the Force.com data Architecture
Multi-Tenant Magic:
Under the Covers of the Force.com
data Architecture
Craig Weissman
Chief Software Architect
salesforce.com 1

2. Safe Harbor Statement
“Safe harbor” statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements including but not limited to statements concerning the potential market for our existing service offerings and future offerings. All of our forward looking statements involve risks, uncertainties and assumptions. If any such risks or uncertainties materialize or if any of the assumptions proves incorrect, our results could differ materially from the results expressed or implied by the forward-looking statements we make.
The risks and uncertainties referred to above include - but are not limited to - risks associated with possible fluctuations in our operating results and cash flows, rate of growth and anticipated revenue run rate, errors, interruptions or delays in our service or our Web hosting, our new business model, our history of operating losses, the possibility that we will not remain profitable, breach of our security measures, the emerging market in which we operate, our relatively limited operating history, our ability to hire, retain and motivate our employees and manage our growth, competition, our ability to continue to release and gain customer acceptance of new and improved versions of our service, customer and partner acceptance of the AppExchange, successful customer deployment and utilization of our services, unanticipated changes in our effective tax rate, fluctuations in the number of shares outstanding, the price of such shares, foreign currency exchange rates and interest rates.
Further information on these and other factors that could affect our financial results is included in the reports on Forms 10-K, 10-Q and 8-K and in other filings we make with the Securities and Exchange Commission from time to time. These documents are available on the SEC Filings section of the Investor Information section of our website at Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements, except as required by law.

3. True Multi-Tenancy is our Religion
Time
Technology
Advances
Major
Architectural
Shift
Agenda
Our Religion
Virtual Database
App Model
Business
Model
Changes
Agenda for this talk:
Shared-everything religion and runtime composition of applications
Database constructs: metadata, data, optimization
Development model: declarative and procedural

4. Single tenant applications: lots of waste          
App  
App        Db Db               
App  
App 
Dedicated software instances: under-utilized hardware stacks for each tenant
We prevent a great deal of hardware and software from being sold to our customers
We do with 1000 machines what would otherwise require 100,000+
Multiple staffs to hire, train, and pay
Database instances are expensive to maintain
Point-in-time backup strategies are expensive
Rolling upgrades are hard
Multiple versions to support
3rd party vendor versions
More importantly your own code base (patches)
Your data schema Db Db          

5. Multi-tenancy benefits are self-evident But isolation is much easier said than done…
App Db
Only one software instance and hardware stack for multiple tenants
Significant cost savings in hardware
Provisioning is free – this is critical (we add rows to existing tables – this is the sweet spot for RBDMS)
Actually we have about 10 such identical “POD’s” running 10,000’s of tenants each (avoid SPOF’s)
Only one staff needed to maintain the service
Surprisingly small number of humans run the entire universe of tenants
Monitoring and automation are key, but much easier in a shared environment
Very small number of versions to support
3rd party platform support matrix collapse
Your own code base and schema are the same everywhere
Critically: our developers are only working on 2 versions of our code at a time – much more innovation possible in the long term this way

6. Our religion: Not all “multi-tenant” designs are created equal
“Can’t we create a
separate stack for just this
one customer? I promise
it’s just this one…”
App
App Db Db
It’s a slippery slope, this is hard
Upgrade timing
Functionality for one tenant
Performance tuning for one tenant
UI for one tenant
Workflow for one tenant
Security concerns (real or perceived)
Has any other enterprise database application resisted this temptation?

7. Introducing the Force.com metadata-driven, multi-tenant, Internet application platform
Poly-
Morphic
Application
Separate concerns:
Tenant-specific metadata
Tenant business data
Shared runtime engine (kernel)‏
Metadata-driven architectures facilitate multi-tenant applications
Over time less and less functionally is statically built into our kernel
Instead, each tenant application is composed at runtime from rich metadata

8. Key Architectural Principles
Stateless AppServers
Database system of record
No DDL
All tables partitioned by OrgId
Smart PKs, Polymorphic FKs
Creative de-normalization and pivoting
Use every RDBMS feature/trick
All tenants are equal (AppServers scale horizontally)
We can run the site with just the database and no caching (just performs/scales worse)
Only cache metadata at our layer. RDBMS SGA caches data
No global locks ever
Primary key prefix = Object Type Id
Polymorphism allows pivot tables and flex schema to work

9. Metadata, data, and pivot table structures store data corresponding to virtual data structures
A simplified look at some key metadata and data tables
No DDL at runtime!
Our flex schema uses clever de-normalization – many patents pending
We add data across tables for reporting (the big grid-like CFDATA tables)
But we also have pivoted specialized tables

10. The Objects table stores metadata about custom objects (tables)‏
Naturally we have our own multi-tenant catalog tables – like Oracle’s own system catalog, but for our own bookkeeping
The definition of our custom objects is one of the most important metadata tables

11. The Fields table stores metadata about custom fields (columns)‏
And we have our own field catalog table – our field types are richer than standard RDBMS types.
For instance multi-select picklists are stored in a special compact representation

12. The Data heap table stores all structured data corresponding to custom objects
Massive, massive table – wide and deep, billions of rows, partitioned, shared
All flex data stored in varchar columns – canonical forms for data types
ObjId distinguishes different “tables” for the same tenant
The same physical column is used for different purposes across tenants and across objects within a tenant
Our query engine emits the proper SQL for all readers and writers of data (e.g. the Api and reporting)

13. A single slot can store various types of data that originate from different objects
Example of a single column
Note that the underlying RDBMS cannot know about this usage.
Small conversion penalty for native data type processing
So how do indexing and statistics work?

14. The Indexes pivot table manages tenant-specific selective indexes
An example of a critical de-normalized pivot table
Data from a single object’s slot of a VAL column is copied to this tall/skinny table as well
Used by the external Id feature for example…
Real RDBMS btree indexes are then used on the value columns
However, to use this table, SQL must lead with it
10’s of billions of rows are ok – tables are like super models

15. The UniqueFields pivot table facilitates uniqueness for custom fields
The uniqueness feature is backed by a real RDBMS unique index.
Same schema as previous slide, but with unique index
The platform detects the index violation and then provides additional information (the Id of the duplicating row)
Other pivot tables: NameNorm, SNL, etc.

16. The Relationships pivot table facilitates referential integrity and optimizes joins
Navigation from parent to children would not be possible in the heap table
This table has indexes in both directions
Again, the various users of this table must know to inject this table into the SQL

17. All data & metadata structures are partitioned to improve performance and manageability
Tables hash partitioned by OrgId
Separate conn pools point to physical hosts
App tier is also dynamically partitioned by OrgId
Distributed metadata cache w/transactional invalidation
List partitioning has been suggested, but questions about DDL
Local indexes all lead by OrgId
Stability in the event of failover (RAC and appservers)

18. Application Framework: a whole lot for free
Native Declarative features
Bulk Processing
The Recycle Bin
Full Text Search
Smart Bulk DML
Web Services APIs
Our “database” is really an application server too – many higher level features built-in and tightly integrated

19. Force.com’s native Application Framework provides declarative development, no coding
Example: activity tracking, field history tracking
Btw, the audit data is stored in another massive pivot table

20. Validation rules and simple formulas: Business analysts can “code” these
Fairly straight-forward, like check constraints in a RDBMS

21. Not so simple: Rollup-summary fields provide for easy cross-object summaries
RSF’s must be de-normalized for grid reporting
Very hard to get right all the time
Done automatically by the bulk save engine
Invalidated and bulk re-calculated when needed

22. Force.com’s bulk processing optimizations reduce overhead for data loads
We have spent years making the save loop bulk from beginning to end
Retry and partial save logic is novel
Well-defined blending of declarative (RSF, validation, auditing) and procedural (Apex) logic

23. Data definition processing is optimized to avoid performance hits or concurrency limits
Examples:
Sort all records by primary key before attempting DML
Operate on tables in deterministic order
Slot reallocation for field datatype change
Deferred calculation for new rollup-summary field
Background processing of mass changes
Again, years of experience and database expertise
Benefit from our best practices

24. The Recycle Bin: Smart Undeletes
Individual object instances (records)‏
Related object instances (parent/child records)‏
Entire fields and objects (dropped columns and tables)‏
Restore
Another high-level application feature not provided by standard RDBMS
Deep cascading operations are difficult to get right (deadlocks e.g.)
Restoration across schema changes is novel

25. Force.com’s full-text search engine
Asynchronously maintains indexes for all text fields
MRU caches contain recently updated objects
Optimizes ranking of search result records based on current user, modification history, and weighting preferences
Another hard platform area to get right
Years of iteration for massive multi-tenant performance
Various point solutions available on the market – but our is integrated for free

26. Multi-tenant Query Optimization Principles
Consistent SQL generation across the application
Deep awareness of pivot table structure
Flex schema does impose a cost
Tenant, user, object, fields statistics are crucial
No runaway queries allowed
Deep integration with the sharing model
Core competency of our platform
“Index” plans become extra joins plus indexes
DML maintains some statistics synchronously; others gathered on a scheduled basis
Sharing: Application layer blended into the DB layer (only possible with a platform)

27. Multi-tenant optimizer statistics
Force.com’s query optimizer writes optimal queries for internal data access operations
Multi-tenant optimizer statistics
Multi-tenant optimization based on our flex schema with knowledge of de-normalized pivot tables and sharing model
Data sharing is scalable and robust and tightly integrated into the optimizer – another high-level application feature pushed into the DB layer
Statistics kept automatically for custom indexes and foreign keys
Histograms for picklist values, special statistics for record owner field, etc.

28. Pre-Query Selectivity Measurements
The optimizer considers pre-query selectivity measurements when writing a query
Pre-Query Selectivity Measurements
… use of index related to filter.
High
… ordered hash join; drive using Data table.
Low
… nested loops join; drive using view of rows that the user can see.
Write final database access query, forcing …
Filter
User
Some statistics are cached – others require dynamic gathering
Query plans differ not only in hints but in the skeleton of the query itself (extra tables joined)
Ongoing research into parallelism, hashing vs nested loops, etc.

29. Apex: Force.com’s procedural frontier
Adding a procedural language to our service was not a simple decision – eventually we knew we had to do it, but safely
Functionality of pl/sql
Syntax and idioms of Java/C#
But direct access to salesforce.com features for SOQL, DML, cursors, etc.
Tightly integrated into our save logic
Years of refactoring of our core database interaction made this possible

30. Apex code is stored as metadata, interpreted at runtime, and cached for scalability
The code itself is stored in metadata like everything else
The actual compiled form is just a POJO (we’re not compiling any Java – we built our own parser (ANTLR) and runtime, written in Java)
The Apex VM has a dynamic class loader that can re-use the same compiled form for multiple concurrent requests
Managed code is shared across tenants

31. Apex is deeply integrated with platform features
Bulk DML
and messaging
Asynchronous processing (Futures)
XmlStream / HTTP (RESTful) services classes
Declarative exposure as new Web Services
The “ADK” provides access to powerful platform features

32. Force.com governs Apex code execution
Limits on:
CPU
Memory
# of DML statements
# calculations
# web service calls
… and more
Governance is a key concept for any multi-tenant shared system
Built for safety first – controlling the runtime completely made this much easier
Plus we force people to use good practices such as Bulk SQL/DML

33. Unit tests must accompany Apex code
Required 75% code coverage
Profiling is built into the platform
Run during application install
All tests are run before each platform release by us
Novel aspect of our development environment – the development and deployment model are combined
Yes, we are like big brother, but we’re not apologetic
The ability to run on upgrade hammer against tenant tests is invaluable for our platform server compatibility story

34. Force.com is a proven multi-tenant application platform that performs and scales
Quarterly Transactions (billions)‏
Page Response Time (ms)‏
As our workload continues to evolve with Apex and VisualForce, we continue to focus on performance and scalability
Emergent property of developing and running the site
We have access to all the logs and all the data and all the exceptions and all the performance information
Virtuous cycle of improvement – everyone benefits
2005
2006
2007
Fiscal Year

35. Concluding Remarks PaaS is a major architectural shifts
PaaS is Application focused, high level of abstraction
Force.com is the most mature, proven PaaS offering available today
Optimized for fast, secure, and reliable multi-tenant application development and deployment