Selecting and Implementing An Embedded Database System Presented by Jeff Webb March 2005 Article written by Michael Olson IEEE Software, 2000

Overview  Key Strategy – Focus on application requirements  Embedded DB products vary  Some do less than what you need  Some will do more  Choose the tool that most closely matches your needs

Overview  After choosing OS, HW platform and DB you must design for reliability  Design for performance up front  Evaluate performance once the application is built

Evaluate Database Services Available  High end RDM systems provide:  Concurrency  Transaction processing  Disaster recovery  Although these features may be needed, enterprise systems are seldom a good choice due to:  platform differences  packaging differences

Evaluate Database Services Available  Several embedded DB systems use the same techniques as enterprise systems but in smaller packages.  Often, full blown disaster recovery is not needed  Many embedded databases are configurable allowing inclusion or exclusion of services

Consider Services Required  Now… consider which services you need. For example:  Locking?  Will run faster without locking  Recovery from failures matter?  Lack of, or disabling will increase performance

Operating Systems For Embedded systems  Hundreds of OS for variety of processor hardware  DB developers have an enormous job porting their software to the variety of platforms  Few OS dominate the market  MS Win CE  Neutrino  VxWorks  Wind Rivers

Databases For Embedded systems  Classification  Client Server relational systems  Client Server OO systems  Embedded libraries

Classification  Client Server relational systems  + Programmers already know SQL  - Extra run-time cost of client server communications

Classification  Client Server OO systems  Appear to be a good choice however  Most are designed for Unix systems and their deeply engrained assumptions about memory management and interprocess communications are difficult to port to embedded OS.

Classification  Embedded libraries  Created specifically for embedded systems  Provide simple language API that does not require SQL  + Faster execution, increased reliability  - Require developers to master nonstandard programming interfaces

Platform Support  Consider combination of:  OS  Processor  Storage system  Exotic processor boards  Rare combinations can be difficult to fit  Often embedded OS vendors maintain lists of partner companies whose products run on their systems

Performance Considerations  High concurrency?  Size of database  Multiple control threads?  Can not rely on standard benchmark measuring systems  Evaluation of actual performance is a must

Designing the Application  Design for performance  Consider:  Speed  Predictability & Reliability

Designing the Application Speed Predictability & Reliability  Data representation  Access patterns  Configuration

Designing for Speed Data representation / Access patterns/Configuration  Most embedded DB tools operate on a fixed set of data types  Every fetch and store may require translation  A few systems, mostly library types allow storage in program-native format

Designing for Speed Data representation/ Access patterns /Configuration  Consider the queries that the application will need to make  Data should be laid out accordingly  Can keys be used that will allow related records to be physically stored together?  B+tree storage typically performs faster than hash table algorithms

Designing for Speed Data representation/Access patterns/ Configuration  Must understand the chosen systems configuration options  Memory use for secondary cache  Write data to disk or store in memory?  Locking system granularity  Entire locking system on/off  Vendors often choose the wrong defaults

Designing the Application  Design for performance  Consider:  Speed  Predictability & Reliability

Predictability & Reliability  May need to run with no humans present  Not easy  Fanatically check return values and error indicators  Resource exhaustion  More common in embedded systems  Track and release resources yourself

Predictability & Reliability  Are transactions required so that changes won’t be lost after a crash?  The recovery system must be callable from the application program on start up?

Performance tuning  Common causes for poor performance  Contention – 2 or more threads contending for same data  Disk-to-memory transfers  Deadlocks

Contention/ Disk-to-memory transfers/ Deadlocks  When several threads are waiting for the same resources  Use record level locking if possible  Use smaller pages if possible to make page level locks more like record level locks  Touch the Hot data last and hold it for short periods of time

Contention/ Disk-to-memory transfers / Deadlocks  Disk latency – mechanical  Flash RAM  More memory for buffer cache  Indexes

Contention/Disk-to-memory transfers/ Deadlocks Obj O1 locked T1 Obj O2 locked T2 T1 waits for a lock on O2 T2 waits for a lock on O1

Deadlocks  Turn off locking if it is not needed and if the application permits  Break large transactions into several smaller transactions  Write transactions so that they all acquire the same resources in the same order  Consider Optimistic Concurrency Control

Price  Helps make final decision  Some are available at no cost  Licensing methods vary  Per developer  Per application using it  Per deployment platform  Per user is less common in embedded systems  During development / deployment

Questions