Matt Weir

Our Original Goal To create a data logging system across the kernel with accurate timing that will monitor data as it moves up and down the data path.

The Feasibility of that Goal

The Current Goal Produce a framework that will assist in tracing the control flow of read/write operations in the Linux kernel using kernel markers

A Brief History of the Project

Week of May 26th 3rd Week of Class Created our group Decided upon our basic goals Did research on previous efforts into this field

Question: Has this been done before? Answer: Yes Understanding and Visualizing Full Systems with Data Flow Tomography

Week of June 2nd 4th Week of Class Talked to Dr. Wang and various graduate students to try and figure out how file IO works in Linux This is a generalization from my own imperfect ability to fully follow the conversations but... – There’s a lot of mystery about how the current version of Linux really works. Started playing around with printk

First Experience using prink Jun 2 12:20:30 device85 kernel: DATATAGGING: Someone called kmalloc (message repeated times) Not so bad

Second Experience using prink Decided to add a timestamp Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at

Size of the Log File

In Defense of printk When I added them to multiple functions, it does show the control flow Can grep through the log file to get a smaller snapshot of what is going on No noticeable performance issues from the user standpoint They work

That being said… It is hard to manage a large number of them Adding/removing printks is time consuming They require an external structure to turn them “on/off” during run time – Didn’t even think of this option until I was using markers When inserting them make sure you don’t add one right after an “if” statement that doesn’t use {}

Week of June 9th 5th Week of Class Decided to move from printk to markers Upgraded my kernel version to so that we would be using the same code Dr. Baker walked through the control flow of read statements with us Figured out how to implement markers and designed some basic test cases

Markers Added very recently to the Linux kernel In the creator’s own words – “It makes sense to offer an instrumentation set of the most relevant events occurring in the Linux kernel that can have the smallest performance cost possible when not active while not requiring a reboot of a production system to activate”

Adding Marker Support In menuconfig – General->Activate markers

The Marker Structure The Marker – A hook in the code to call a function in an attached probe The Probe – A function that can be attached to markers The Manager – A kernel module that manages/arms and disarms probes

Friday Night Started to worry since all we had was glorified prink’s Decided to have a few drinks…

Came up with an Idea Focus on the marker management kernel module Modify the marker code to support finer grained logging Try to trace the control flow in read/write statements

PirateAcorn The management kernel module that I wrote

PirateAcorn (continued) Acorn – Counter Intelligence term: Slang for someone who is performing traffic analysis Pirate – Because they are way cooler than ninjas

PirateAcorn (continued) Manages all the probes via ioctl commands – Breaks up the probes into read and write groups – Can enable them individually or at the same time – Supports the ability to have additional groups added to it – Can turn off monitoring for certain threads, such as other logging programs – Can be set to monitor a specific thread or all threads

Registering Probes

Arming the Probes

The Probe Function

Modification to marker.c Needed to add support so it would only fire if the marker was called by a thread that is being logged Didn’t want to put the check in the probe function since that was called only after the marker fires Instead made a quick function that checks to see if a marker should fire

The Marker Check Code

The Marker Code

Adding Markers For the most part I concentrated on mapping the VFS layer and the File system

Finding the current PID Most of the time it was easy – current->pid But in some cases I wasn’t allowed to reference current – seq_read() in linux/fs/seq_file.c – Called by vfs_read() as file->f_op->read

Possible Solutions Just print out all calls to seq_read(), and filter them out when processing the log file Don’t bother to log seq_read() at all Implement a binary value in marker.c that is set true when a previous marker is allowed to fire, and false when a marker is denied Create a wrapper function Include the PID value in a structure that is already being passed to it

Well what values are passed to it? File->f_op->read(file, buf, count, pos) – Count and pos are integer values – Buf is the buffer from the user, really don’t want to mess with that – What about file?

The File Structure

Yes it’s a Bad Idea But what would happen if I added a PID field?

Answer: PIRATEACORN: sys_read_start: Pid=3055 Time= PIRATEACORN: vfs_read_start: Pid=3055 Time= PIRATEACORN: vfs_read_fop_read: Pid=3055 Time= PIRATEACORN: seq_read_start: Pid=3055 Time= PIRATEACORN: sys_read_end: Pid=3055 Time= Yes, though I can see possible issues with this implementation, the biggest being multiple threads accessing the same file

The Mystery Call This one still has me stumped For several threads, such as metacity I hit a roadblock when I trace their read control flow

The Mystery Call (continued) The current control flow goes – sys_read_start – vfs_read_start – vfs_read_fop_aioread – do_sync_read_start – do_sync_read_forloop – do_sync_read_end – sys_read_end

The Mystery Call (continued) I loose the trace in the following call in do_sync_read – Filp->f_op->aio_read(&kiocb, &iov, 1, kiocb.ki_pos) This means there is a aio_read function associated with an f_op that I don’t know about

It Should be Pretty Easy to Track that Down…

My Current Guess There is an non standard kernel module that is installed and has its own aio_read handler Metacity is the gnome window manger – I can see it doing some funky stuff When adding markers I found lots of similar examples where the control flow didn’t go as I thought it would

QUESTIONS / COMMENTS?