Kernel Structure and Infrastructure David Ferry, Chris Gill, Brian Kocoloski CSE 422S - Operating Systems Organization Washington University in St. Louis St. Louis, MO 63130

Kernel vs. Application Coding What are some of the differences? CSE 422S – Operating Systems Organization

Kernel vs. Application Coding What are some of the differences? What happens when a user space program executes: printf(“Hello world\n”); int * array = malloc(sizeof(int) * 10); CSE 422S – Operating Systems Organization

Kernel vs. Application Coding What are some of the differences? What happens when a user space program executes: printf(“Hello world\n”); int * array = malloc(sizeof(int) * 10); About how large would you expect a hello world program to be? CSE 422S – Operating Systems Organization

Kernel vs. Application Coding Two major differences: The core kernel has no standard libraries The core kernel must be a monolithic, statically linked library No standard libraries: No libc (malloc, pthreads, string handling, etc.) Partly because of chicken/egg situation Also, standard libraries can be too slow No dynamic loading of shared libraries for the core kernel CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Two Big Problems A totally static kernel would be enormous About 20 million lines of code in 2015 Most of this is hardware drivers that are never used on any given platform Programmers rely on “library” functions for efficiency and correctness E.g., things like data structures, sleeping routines, etc. What libraries to use? (hint: not glibc) Kernel code has to be entirely self-contained Also, chicken+egg problem, as glibc functions may in turn invoke the kernel via system calls CSE 422S – Operating Systems Organization

First Solution: Loadable Kernel Modules Kernel modules are kernel code that can be loaded dynamically: Can be loaded/unloaded whenever Runs in kernel mode Can access exported kernel variables and functions Can export variables and functions to kernel or other modules CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Why Kernel Modules Linux is a monolithic kernel. All functionality is compiled into the same static binary that is loaded into memory on boot. Without modules, the entire kernel would need to be loaded into memory to boot a node. Problems? Waste of memory (embedded systems) Slower boot time Larger trusted computing base (TCB), more room for bugs CSE 422S – Operating Systems Organization

What are Kernel Modules used for? What pieces of code do we think might not be needed on every system that the kernel boots on? Device Drivers Architecture-specific code Lines of code (just .c files) for: Device Drivers: 1,583,159 Everything else: 1,003,777 (Includes all of the architectures that we’re not using!) CSE 422S – Operating Systems Organization

What are Kernel Modules used for? Beyond space savings, what else might modules be useful for? “Out-of-tree” functionality that is not accepted into “mainline” kernel Allowing users to load custom functionality at runtime Configuring/patching a running system without requiring a reboot Your lab assignments in this class  CSE 422S – Operating Systems Organization

Second Solution: Kernel “Libraries” Kernel libraries re-implement a lot of the functionality programmers expect in user space Are statically compiled into the kernel Automatically available just by including relevant header Built to be kernel-safe (sleeping, waiting, locking, etc. is done properly) Features: Utilities: kmalloc, kthreads, string parsing, etc. Containers: hash tables, binary trees etc. Algorithms: sorting, compression Mostly found under /lib and /include/linux CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Kernel “Libraries” Mostly found under /lib and /include/linux Many kernel “libraries” have clear analogues to user space libraries: - malloc/free vs kmalloc/kfree - pthread_create vs kthread_create - sleep/usleep/nanosleep (user) vs msleep (kernel) Some, however, are a bit different: - e.g., linked list implementation CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example: Linked Lists (LKD pg. 88) struct fox { unsigned long tail_length; unsigned long weight; bool is_fantastic; }; If you wanted a list of structs of type struct_fox: struct fox { unsigned long tail_length; unsigned long weight; bool is_fantastic; struct fox * prev; struct fox * next; }; struct fox { unsigned long tail_length; unsigned long weight; bool is_fantastic; struct list_head list_node; }; CSE 422S – Operating Systems Organization

Case 1: Manually Linked Together struct fox { unsigned long tail_length; unsigned long weight; bool is_fantastic; struct fox * prev; struct fox * next; }; struct fox { unsigned long tail_length; unsigned long weight; bool is_fantastic; struct fox * prev; struct fox * next; }; struct fox { unsigned long tail_length; unsigned long weight; bool is_fantastic; struct fox * prev; struct fox * next; }; struct fox { unsigned long tail_length; unsigned long weight; bool is_fantastic; struct fox * prev; struct fox * next; }; Disadvantage: need to “roll your own code” for each list you create Duplicate code throughout the kernel Introduce bugs Lose optimizations (placement within cache lines, etc.) CSE 422S – Operating Systems Organization

Case 2: Embedding Fox in list_head A list-able struct must contain struct list_head struct list_head { struct list_head *next; struct list_head *prev; }; If you wanted a list of structs of type data: struct fox{ unsigned long tail_length; unsigned long weight; bool is_fantastic; struct list_head list_node; CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Source: https://notes.shichao.io/lkd/ch3/ CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example: Linked Lists Initializing a list dynamically: struct fox *new_fox; new_fox = kmalloc(sizeof(struct fox), GFP_KERNEL); new_fox->tail_length = 40; new_fox->weight = 6; new_fox->is_fantastic = false; INIT_LIST_HEAD(&(new_fox->list_node)); Or statically at compile time: static LIST_HEAD(fox_list_head); CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example: Linked Lists Q: list_node.next, list_node.prev point to other list_head. So, how do I get access to the struct fox * embedded within? struct fox { unsigned long tail_length; unsigned long weight; bool is_fantastic; struct list_head list_node; }; list_node.prev list_node.next struct list_head { struct list_head * prev; struct list_head * next; }; struct list_head { struct list_head * prev; struct list_head * next; }; CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example: Linked Lists Casting out from list_node to containing structure: struct fox * fox; struct list_head * node; fox = list_entry(node, struct fox, list_node); /* list_entry( arg 1: Pointer to list_head structure, arg 2: Type of outer “containing structure”, arg 3: Struct member of list_head within containing structure ) */ CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example: Linked Lists Functions may take pointers to list nodes or pointers to the list head: Adding: struct data *new_data; list_add(&new_data->list, &data_list_head); Deleting: list_del(&new_data->list); kfree(new_data); //if dynamic CSE 422S – Operating Systems Organization

CSE 422S – Operating Systems Organization Example: Linked Lists Iterating: struct list_head ptr; list_for_each(ptr, data_list_head){ //ptr points to each list structure } struct data *data_ptr; list_for_each_entry(d_ptr, data_list_head, list){ //d_ptr points to each data structure Also: list_for_each_entry_reverse() list_for_each_entry_safe() //for modifying list elements CSE 422S – Operating Systems Organization

Example: Linked Lists See also: Moral of the story: /include/linux/list.h /include/linux/types.h Moral of the story: Always search for functionality before writing it yourself. CSE 422S – Operating Systems Organization

Module Implementation Must define: An initialization function called on load An exit function called on unload The init function must be self contained! Must unwind actions if initialization cannot complete successfully E.g. if you kmalloc() space but don’t free it, that physical memory is now lost (until system restart) You can also pass parameters to modules at load time. CSE 422S – Operating Systems Organization

Loading and Unloading Modules Modern approach uses modprobe utility Checks dependences, other important features Optional enrichment exercise today uses it Today we will use insmod and rmmod Must use sudo to invoke these on your Rpi ERROR: could not insert module jiffies_module.ko: Operation not permitted Cannot unload a module that’s not loaded ERROR: Module jiffies_module is not currently loaded Cannot load a module that’s already there ERROR: could not insert module simple_module.ko: File exists CSE 422S – Operating Systems Organization