1. Kthreads, Mutexes, and Debugging
Sarah Diesburg
CS 3430
Operating Systems

2. Refresher: procfs Kernel Module
procfs “hello world” example
Creates a read/write procfs entry
Steps
Create entry in module_init function
Registers reading/writing function using file_operations structure
Delete entry in module_cleanup function

3. Testing Procfs
#> insmod hello_proc.ko #> tail /var/log/syslog #> cat /proc/helloworld #> echo “hi” > /proc/helloworld #> rmmod hello_proc

4. Project 3: Remember
You will write your own proc module called remember that
Allows the user to write a string to /proc/remember (max length 80)
Allows the user to read /proc/remember and get back the string that was just added
If no string was added, the string EMPTY should be read instead

5. Remember Example #> echo “Hello there” > /proc/remember
#> cat /proc/remember
Hello there
#>

6. Run the main logic of your module in a kthread!
Kthreads
Run the main logic of your module in a kthread!

7. Refresher: hello.c
#include <linux/init.h> #include <linux/module.h> MODULE_LICENSE(“Dual BSD/GPL”); static int hello_init(void) { printk(KERN_ALERT “Hello, world!\n”); return 0; } static void hello_exit(void) printk(KERN_ALERT “Goodbye, sleepy world.\n”); module_init(hello_init); module_exit(hello_exit);

8. Kernel Modules Remember, kernel modules are very event-based
How can we start an independent thread of execution to run in the background?

9. kthread_run kthread_run(threadfn, data, namefmt, ...)
Creates a new thread and tells it to run
threadfn – the name of the function the thread should run
data – data pointer for threadfn (can be NULL if the function does not take any args)
namefmt – name of the thread (displayed during “ps” command)
Returns a task_struct

10. kthread_run example
struct task_struct *t; t = kthread_run(run, NULL, “my_thread");

11. kthread_stop int kthread_stop(struct task_struct * k);
Sets kthread_should_stop for k to return true, wakes the thread, and waits for the thread to exit
Returns the result of the thread function

12. kthread_stop_example
ret=kthread_stop(t);
if(ret != -EINTR)
printk("Main logic tread stopped.\n“);

13. Thread Function Example
static int run(void *arg) { /* Lock here */ while(!kthread_should_stop()) { /* Do stuff */ } /* Unlock here */ printk("%s: kernel thread exits.\n", __FUNCTION__); return 0;

14. Demo kthread module

15. Part 3: Kitchen Scheduling

16. Part 3: Kitchen Scheduling
Implement a /proc kernel module that simulates an busy kitchen at a restaurant
Your /proc/kitchen file must accept writes of different dishes
Each dish takes some time to process
Your /proc/kitchen file must return status information when read

17. Why Scheduling? Classic producer/consumer analogy
Similar to disk schedulers
File system produces read/write requests
Disk consumes requests, optimized for disk head position, rotational delays, etc.

18. Your Kitchen One kitchen 20 order slots Four types of dishes
Use a static array of ints?
Four types of dishes
1 - Caesar Salad
2 - Hamburger
3 - Personal Pizza
4 - Beef Wellington

19. Food orders
A write of 1-4 to /proc/kitchen will fill an order slot with the dish corresponding to the number
You decide how the order slots get filled (e.g. round robin or other way)
Kitchen takes 1 second to look in an order slot
Regardless if empty or containing an order

20. Food Orders The kitchen can only process one order at a time
Each order takes a different amount of time to process
2 seconds for a Caesar Salad
3 seconds for a Hamburger
4 seconds for a Personal Pizza
8 seconds for a Beef Wellington

21. Kernel Time Constraints
#include <linux/delay.h>
void ssleep(unsigned int seconds);
A call to ssleep will have the program cease to the task scheduler for seconds number of seconds

22. Food Order
Once an order is processed, the kitchen can mark that order slot as empty and should look at other order slots to find more orders to process.

23. Processing the orders?
As soon as the kitchen module loads, it should start a persistent background thread that looks at each slot in the array and processes orders
The actual order processing logic will be run in a kthread
This is where your module should ssleep to look in the queue and “process” an order

24. Status Information
Performing a read on /proc/kitchen should return some status information
Current spot being looked at in the queue
Current order being processed
Number of orders processed
If the kitchen is not running, the last two pieces of information do not need to be displayed

25. Additional Design Considerations
How to place orders in order slots?
How to check order slots from the kitchen?
What scheduling algorithm to use?

26. Demo
Project 3 Specification
Working binary
Kitchen starter code

27. Concurrency Aspects of Project 3
Synchronizing access to request queue
Multiple producers may access request queue(s) at the same time
Multiple consumers may access request queue(s) at the same time
Synchronizing access to other global data

28. Kitchen Queue Concurrency
Jobs may appear on the queue at the same time the kitchen module checks the queue
The status may be read at the same time that you're updating
Number of jobs that you've serviced
Which job is currently being processed
Which slot in the queue kitchen is looking at
How do you guarantee correctness?

29. Global Data vs. Local Data
Global data is declared at global scope, e.g. outside of any function body
Often necessary for kernel programming
Particularly sensitive to concurrency issues
Be extra careful when handling globals

30. Global Data vs. Local Data
Local data is declared within a function
Local data is sensitive to concurrency issues when it depends on global data or when parallel access is possible
Think carefully about whether it needs to be synchronized

31. Synchronization Primitives
Semaphores
User space
Kernel space
Mutexes
Spin locks
Atomic Functions

32. Synchronization Primitives (We'll Only Cover These)
Mutexes
User space
Kernel space
Does anyone remember the differences between a mutex and semaphore?

33. The Mutex
Caught up in the mutex?

34. Mutexes Mutex – A construct to provide MUTual EXclusion
Based on the concept of a semaphore
Found at <source_dir>/include/linux/mutex.h
Can be locked or unlocked
Only one thread of execution may hold the lock at a time

35. Kernel-Space Mutex - Initialization
mutex_init(&mutex)
Declare and initialize a mutex
Only initialize it once

36. Kernel-Space Mutex - Locking
void mutex_lock(struct mutex *);
int mutex_lock_interruptible(struct mutex *);
mutex_lock() can wait indefinitely
mutex_lock_interruptible() locks a mutex as long as it is not interrupted
returns 0 if locking succeeded, < 0 if interrupted
Use of the interruptible version is typically preferred

37. Kernel-Space Mutex – Unlocking
void mutex_unlock(struct mutex *);
Guarantees that the mutex is unlocked
Why is there no interruptible version of this function?

38. Kernel-Space Mutex Example
/* Declare your mutex */ struct mutex my_mutex; /* Initialize your mutex */ mutex_init(&my_mutex); /* Lock */ if(mutex_lock_interruptible(&my_mutex)) return -ERESTARTSYS; /* Do stuff to protected global variables */ /* Unlock */ mutex_unlock(&my_mutex);

39. User-Space Mutex Also used with pthreads in regular user applications
pthreads operate very similar to kthreads
Might be useful if you are prototyping your code in user-space before porting to kernel

40. User-Space Mutex - Initialization
int pthread_mutex_init(pthread_mutex_t *, NULL);
int pthread_mutex_destroy(pthread_mutex_t *);
Pthread_mutex_init() dynamically allocates a mutex
Pthread_mutex_destroy() frees a mutex

41. User-Space Mutex - Locking
int pthread_mutex_lock(pthread_mutex_t *);
Returns 0 on locking, < 0 otherwise

42. User-Space Mutex - Unlocking
int pthread_mutex_unlock(pthread_mutex_t *);
Returns 0 on unlocking, < 0 otherwise

43. Kitchen Scheduling Advice

44. General Advice Just make kitche work first Optimize if there is time
Use a very simple algorithm
My kitchen searches the queue in round-robin fashion and processes every maintenance request
Optimize if there is time

45. Round Robin Method: Pros/Cons?
Service requests in round-robin fashion (e.g. queue slot 0, 1, 2, 3, etc.)
Pros/Cons?

46. Shortest Job First
Method:
Service fastest orders first
Pros/Cons?

47. Hybrid Combine methods, come up with something new
Up to your creativity

48. Getting Help
Class time to answer questions
Regular office hours

49. Other Hints This is not a simple project Setup is different
You need to use different files and methods of compilation/running
Do NOT wait until 2 days before it is due to start
Too late
The Internet will likely NOT be very helpful

50. Other Hints Set milestones Ask questions early
If it’s a good question, I’ll share it with the class

51. Debugging

52. Kernel Debugging Configurations
Timing info on printks
Detection of hung tasks
Kernel memory leak detector
Mutex/lock debugging
Kmemcheck
Check for stack overflow

53. Select Kernel Hacking

54. Enable Debugging Options

55. Debugging through reads to /proc/penguin
Necessary to help you “see” what’s going on!
General process
Identify data to monitor in your module
Run your module
Query (read from) /proc/penguin for that information at any time

56. Kernel Oops and Other Errors
Kernel errors often only appear on first tty (terminal interface)
Why?

57. Oops!

58. Reason for failure

59. Current drivers

60. Call Trace

61. Call Trace

62. Failed command