The Duality of Memory and Communication in the Implementation of a Multiprocessor Operating System Michael Young, Avadis Tevanian, Richard Rashid, David Golub, Jeffrey Eppinger, Jonathan Chew, William Bolosky, David Black, and Robert Baron ACM Symposium on Operating System Principles, 1987 Presented By Rajesh Sudarsan October 21, 2005

Agenda Introduction Key Ideas Monolithic vs Microkernel Primitive abstractions Implementation Details Issues with External Memory Management Benefits of Duality Conclusion CS 5204

Introduction Mach OS project started in 1985. Continued till 1994. Successor to Accent OS developed at CMU MACH NeXTSTEP OPENSTEP Mac OS X Mach OS kernel – mainly designed to support multiprocessors. Microkernel - a small, efficient kernel providing basic services such as process control and communication CS 5204

Design goals Object oriented interface with small number of basic system objects Support for distributed and multiprocessing Portability to different multiprocessor and uniprocessor architectures Compatibility with BSD UNIX Performance comparable to commercial UNIX distributions CS 5204

Key ideas Communication and virtual memory can play complementary roles in OS kernel Increased flexibility in memory management Support for multiprocessors Improved performance Easier task migration Memory represented as abstract objects called memory objects Single level store implementation CS 5204

Key ideas (contd.) Virtual memory implementation using memory objects External memory management – Structure for secondary storage management CS 5204

Microkernel vs Monolithic Monolithic kernel Kernel interacts directly with the hardware Kernel can be optimized for a particular hardware architecture Kernel is not very portable Microkernel Kernel is very small Most OS services are not part of the kernel and run at a layer above it Very easily portable to other systems CS 5204

Examples Microkernel Monolithic kernel Amoeba, Minix, Chorus, Mach, GNU Hurd, NeXTSTEP, Mac OS X, Windows NT Monolithic kernel Traditional UNIX kernels, such as BSD, Linux, Solaris, Agnix CS 5204

Architecture No direct data exchange between modules User application Memory module Process module File module Microkernel Hardware User mode OS interface Kernel mode System Call CS 5204 *source Distributed systems – Principles and Paradigms, Andrew Tannembaum, Maarten van Steen

Primitive abstractions in Mach OS Four basic abstractions from Accent Task, Threads, Ports, Messages Port set Fifth abstraction introduced in Mach Memory Objects Tasks and Threads – Execution control primitives Ports and Messages - Interprocess communication CS 5204

Interprocess communication Two components of Mach IPC – ports and messages Ports Communication channel Provides finite length queue Protected bounded queue within the kernel Messages Fixed length header and variable size collection of typed data objects CS 5204

IPC (contd.) One receiver, multiple sender Tasks allocate ports to perform communication Task can deallocate rights to a port destination port reply port size/operation pure typed data port rights out-of-line-data … Message control Memory cache object Port Format of Mach messages* CS 5204 *source Operating System Concepts, Sixth Edition by Avi Silberschatz, Peter Baer, Galvin Greg Gagne

Memory Management Virtual memory – level of abstraction between process memory requests and physical memory Continuous address space Demand Paging Transparent relocation of running programs in memory Page and Page frame VM -> RAM – page global directory, page table, offset CS 5204

Virtual Memory Management in microkernel Each task has its own virtual address space Restriction – virtual address space must be aligned with the system page boundaries Supports read/write sharing of memory among tasks of common ancestry through inheritance API provided for operation on VM CS 5204

External Memory Management (EMM) External memory management interface based on memory object Memory object – abstract collection of data bytes with operations defined on them Memory object represented by a port Secondary storage objects available using message passing (data managers) Mach kernel as cache manager for contents of memory object CS 5204

External Memory Management (contd.) Interface between kernel and data manager consists of 3 parts: Calls made by application program to cause object to be mapped into its address space Calls made by the kernel on the data manager Calls made by the data manager on the Mach kernel to control use of its memory object CS 5204

Fault Handling … Validate Hardware Map Victim Task External Pager Task Thread ----- ---- Thread ----- ---- Thread ----- ---- Thread Receive Request Find Data Send Reply (data) … Kernel Context Check Validity Check Protection Page Lookup Do Copy-on-write Call pmap module Resume thread Pmap Module Validate Hardware Map CS 5204

Minimal Filesystem Call from the application Char * file_data; int i, file_size; extern float rand(); ….. fs_write_file(“filename”, file_data, file_size/2); vm_deallocate(task_self(), file data, file_size); return_t fs_read_file( name, data, size) { //Allocate memory object (a port) and accept request port_allocate(t…); port_enable(…); ….. //perform file lookup. Find current file size ….. //Map the memory object into client address space vm_allocate_with_pager(…); return (success); } fs_read_file(“filename”, &file_data, file_size); Call from the application CS 5204

Minimal Filesystem (contd.) void pager_data_request( memory_object, pager_request, offset, size, access) { //Allocate disk buffer vm_allocate (…); //Lookup memory object and read disk data disk_read(…); //Return the data without any lock pager_data_povided(…); //Deallocate disk buffer vm_deallocate(…); } void port_death (request_port) { //find associated memory object with the port lookup_by_request_port(…); //Release resources port_deallocate(…); vm_deallocate(…); } Release filesystem resources after application deallocates its resources File retrieval for the application CS 5204

Consistent Network Shared Memory (Initialization) Client A Client B A Request X B Request X Shared Memory Server vm_allocate_with_pager Client A resumed Client B resumed vm_allocate_with_pager pager_init(X, request_A, name_A) pager_init(X, request_B, name_B) Mach Kernel B Mach Kernel A CS 5204

Consistent Network Shared Memory (Read) Client A Client B Shared Memory Server Client A resumed Client A faults pager_data_request(X, request_A, offset, page_size,VM_PROT_READ) pager_data_request(X, request_B, offset, page_size,VM_PROT_READ) Client B resumed Client B faults pager_data_provided( request_A, offset, page_size, VM_PROT_WRITE) pager_data_provided( request_B, offset, page_size, VM_PROT_WRITE) Mach Kernel B Mach Kernel A CS 5204

Consistent Network Shared Memory (Write) Client A Client B Shared Memory Server Client A resumed Client A write faults pager_data_unlock( X, request_A, offset, page_size, VM_PROT_WRITE) pager_data_lock( request_A, offset, page_size, VM_PROT_NONE) pager_flush_request(request_B, offset, page_size) Mach Kernel B Mach Kernel A CS 5204

Implementation Details Four basic data structures used to implement EMM Address Map -Two level map Top level – protection and inheritance information, link to second level Second level – Map to memory object structures Virtual Memory Object structures Resident Memory Structures Page replacement queues Address map – Task address map is a directory that map each of many valid address ranges to a memory object and the offset within than object. Other information stored are inheritance and protection information. Two level is used to account for sharing thru inheritance VM object structures – An internal memory object structure is kept for each memory object used in the address map. Components of this structure include the ports used to refer to the memory object, its size, number of address map references to the object, permission to cache the memory object Resident Memory structure - corresponds to the page of the physical memory. Records the memory object and offset into the object along with the access permissions by data manager Pageout queues – active queue – contains pages currently in use passive queue- pages ready to be pageout. free queue – pages not caching any data. CS 5204

External Memory Management Issues Types of Memory failure – Data manager doesn’t return data fails to free flushed data floods the cache changes its own data backs up its own data Handling Memory failure Timeout, notification, wait, abort Default pager Reserved memory pool CS 5204

Benefits of duality Multiprocessor support for UMA, NUMA, and NORMA architectures The programmer has the option to choose between shared memory and message-based communication Emulation of operating system environment such as UNIX achieved on Mach Generic UNIX system calls can be implemented outside Mach kernel Other features supported are transaction and database facilities, task migration, and AI knowledge bases CS 5204

Conclusion Significant contribution to the operating system research No experimental to support performance claim More information could be presented in the implementation Drawbacks Frequent message passing may cause degradation in performance Continuous monitoring of external services CS 5204

Current Trend Pure microkernel architecture not common Most OS kernels are hybrid models of monolithic and microkernel, e.g., Windows XP, Windows 2000 CS 5204

Questions? CS 5204