Lecture 5 proc file system(procfs) (Project 2 included) Kyu Ho Park Sept. 27, 2016 Lecture 5 proc file system(procfs) (Project 2 included)

Why do we need the procfs? To access the various information of the Linux kernel easily, procfs has been provided(firstly by Tom J. Killan, 1984 for UNIX on VAX machine). Why we do not use the conventional file system using open, close, read, write system calls ? It is always available and is accessible to all users, of course,with appropriate permissions only.

ls /proc

/proc/cd 1

cat cpuinfo

meminfo

Size of file at /proc directory

Cat /proc/cpuinfo The contents of files under /proc directory are created dynamically when you give commands such as ‘cat /proc/meminfo’.

proc file system(procfs) functions proc_mkdir( ); //make a directory under /proc. create_proc_entry( ); //make a file under /proc. create_proc_read_entry( ); //make a readonly file. remove_proc_entry( ); //remove dirs or files. --procfs uses [struct proc_dir_entry] defined as follows; struct proc_dir_entry{ ..... struct inode_operations *proc_iops; struct file_operations *proc_fops; .. struct proc_dir_entry *next, *parent, *subdir; void *data; read_proc_t *read_proc; write_proc_t *write_proc; atomic_t count; int deleted;//in case of delete, this flag should be 0, if in use this flag is 1. }; //mostly used

Making a directory under /proc struct proc_dir_entry *proc_mkdir(const char *name, struct proc_dir_entry *parent); // name: the directory name to create, //parent: the parent of the directory to create. parent struct proc_dir_entry *son; struct proc_dir_entry *grand_son; son=proc_mkdir(“son”,NULL); grand_son=proc_mkdir(“grand_son”, son); Ex: son grand_son

File creation under /proc struct proc_dir_entry *create_proc_entry(const char *name, mode_t mode, struct proc_dir_entry *parent); //name: the RW file name to create. mode:file permission. To create read_only file, struct proc_dir_entry *create_proc_read_entry(const char *name,mode_t mode,struct proc_dir_entry *parent); Ex:[create a ‘test’ file at /proc/test] struct proc_dir_entry *test_proc; test_proc=create_proc_entry(“test”, 0, NULL); // if mode is set to 0, the default permission is 0444.

Delete proc_files and directories void remove_proc_entry(const char *name, struct proc_dir_entry *parent); test remove_proc_entry(“test”,NULL); remove_proc_entry(“grand_son”, son); remove_proc_entry(“son”,NULL);

read static int my_read(char *page, char **start, off_t off, int count, int *eof, void *data) { ……//program codes *eof =1 ; when this function is called only one time. return len; //len: length of data read. } -The memory space used in procfs is a page whose size is typically 4KBytes. The location to save the data. -eof : end-of-file, if eof =1, no read anymore. -start: it is used rarely. -off: current location of the file. -data: not used in read.

write static int my_write(struct file *file, const char *buffer, unsigned long count, void *data) { char *proc_data; proc_data=(char *)data; copy_from_user(kernel_data, buffer, count); … return count; } -buffer: user space data, -count: written data size, -*data: kernel address to store the data of buffer.

for_each_process( ) at <linux/sched.h> #include<linux/kernel.h> #include <linux/proc_fs.h> #include<linux/sched.h> #include<linux/module.h> int read_process(char *buf, char **start, off_t offset,int count,int *eof,void *data ) { int len=0; struct task_struct *task_list; for_each_process(task_list) { len += sprintf(buf+len, "\n %s %d\n", task_list->comm,task_list->pid); } return len; void create_new_proc_entry() { create_proc_read_entry("proc_list",0,NULL,read_process,NULL); } int ps_init (void) { int ret; create_new_proc_entry(); return 0; void ps_exit(void) { remove_proc_entry("proc_list",NULL); module_init(ps_init); module_exit(ps_exit); MODULE_LICENSE("GPL");

Macros defined at sched.h for_each_process(), next_task() for_each_process() is a macro which iterates over the entire task list. It is defined as follows in linux/sched.h: #define for_each_process(p) \ for (p = &init_task ; (p = next_task(p)) != &init_task ; ) next_task() is a macro defined in linux/sched.h which returns the next task in the list: #define next_task(p) list_entry((p)->tasks.next, struct task_struct, tasks)

cat /proc/proc_list

seq_file The "seq_file" interface to the /proc filesystem was introduced in Linux 2.4.15-pre3 and Linux 2.4.13-ac8. It provides a safer interface to the /proc filesystem than previous procfs methods because it protects against overflow of the output buffer and easily handles procfs files that are larger than one page. It also provides methods for traversing a list of kernel items and iterating on that list. It provides procfs output facilities that are less error-prone than the previous procfs interfaces.

proc_create( ) using seq_file

proc_create() or create_proc_entry() #include <linux/version.h> If (LINUX_VERSION_CODE )< VERSION(3,11,0)&&defined(CONFIG_PROC_FS) create_proc_entry(); else proc_create( );

Project 2. “Linux Fundamental” procfs Dong-Jae Shin, PhD Student 2016. 09. 27.

Introduction to Projects Project 1: Setting up Environment for Project Project 2: Linux Fundamental Project 3: System Call and Synchronization Project 4: File System Project 5: Device Driver for Embedded H/W Especially, including “BeagleBoard Development”

Contents Follow-on Tasks Project Tasks 1. Linux System Administration 2. File System & Software Management 3. Build your own Linux Kernel Project Tasks Write a System Monitoring Tools 1. Analyze the given skeleton program (10%) 2. Traverse Process - tasklist (10%) 3. per Process Memory Usage (20%) 4. per Process I/O Usage (20%) 5. Sorting Features (20%) Report (20%) ------------------------------------------------------ Max. 100%

Account Management Add user Delete user Settings per user Linux System Administration Account Management Add user # useradd <user_name> Delete user # userdel <user_name> Settings per user /etc/passwd Change or create user password # passwd # passwd username

File System Management File System & Software Management File System Management Add a Virtual Disk Virtual Machine Settings -> Add -> Hard Disk -> SCSI -> Create a new virtual disk -> 20GB You need 20GB of free space in your HDD(or SSD)

File System Management File System & Software Management File System Management Disk initialization Format : Create a empty disk with underlying file system File system How to store and retrieve data in the storage as a logical unit(file and directory) File system manages your files in their own structure We’ll learn file system in our lectures later ex) ext2, ext3, ext4, FAT, NTFS Commands mkfs.XXX : format with XXX file system We will use ext4

File System Management File System & Software Management File System Management Format disk we added # ls /dev/sd* /dev/sdb is our new disk # mkfs.ext4 /dev/sdb Mount new disk # cd /root && mkdir mnt # mount /dev/sdb ./mnt Check mounted disk partition # df -hT Unmount disk # umount /dev/sdb we can also use directory name # umount /root/mnt # gnome-disks

Software Install and Uninstall File System & Software Management Software Install and Uninstall Ubuntu use “Advanced Packaging Tool” Install : apt-get install package-name ex) # apt-get install vim Uninstall : apt-get remove package-name Search : apt-cache search search-string ex) # apt-cache search gcc Redhat distribution family use “Yellowdog Updater Modified” # yum install vim # yum erase vim # yum search gcc

Kernel Hardware Kernel User Program User Interface System Calls Build Linux Kernel Kernel Kernel Core of Operating System User Program User Interface System Calls Memory Management Process Management I/O File System Network Security Device Drivers User Space Kernel Space Hardware

Application Framework Build Linux Kernel Linux Distributions Linux Distributions Built on the Linux kernel Linux Kernel System Software Compiler Libraries User Interface (GUI, CLI) User Program Office Database Browser Server Program Libraries & Android Runtime Application Framework Apps Camera Browser SMS

Linux Kernel Download Download Linux Kernel Source Code Build Linux Kernel Linux Kernel Download Download Linux Kernel Source Code https://www.kernel.org/ or http://core.kaist.ac.kr/~EE516/resources/linux-3.18.21.tar.xz

Compile Linux Kernel change to root account Build Linux Kernel Compile Linux Kernel change to root account # su - root Install libraries for compile # sudo apt-get install build-essential libncurses5-dev Unzip # cd /root/mnt # tar -xvf linux-3.18.21.tar.gz # cd linux-3.18.21 Copy original configuration file # cp /boot/config-3.19.0-25-generic .config ./ Kernel compile needs large disk space about 11GB. We can use new disk we already mounted.

Compile Linux Kernel Kernel build configuration # make menuconfig Build Linux Kernel Compile Linux Kernel Kernel build configuration # make menuconfig just save & exit (we already copy .config file)

Compile Linux Kernel Linux Kernel Build Check your # of cores Build Linux Kernel Compile Linux Kernel Linux Kernel Build Check your # of cores # cat /proc/cpuinfo |grep processor Build with multiple processors if you have 4 processors # make -j4 It takes long time Copy compiled kernel & modules # make modules_install # make install Check /boot directory

Booting with new Kernel Build Linux Kernel Booting with new Kernel # reboot Keep “shift key down” while rebooting! Boot menu -> Advanced options -> Select 3.18.21 Check your new kernel # uname -a Compile Time of Kernel

(1)Kernel Debugging Print Kernel-level Message Kernel Log Message printk function Kernel version of printf usage) printk(“%s %d %lu”, str, i, j ); Kernel Log Message tail -f /var/log/kern.log See also (optional) for advanced debugging http://lwn.net/images/pdf/LDD3/ch04.pdf

(2)/proc File System /proc Various information provided by proc a special file system which displays the present of the system pseudo and virtual file system which resides in the RAM provides a method of communication between kernel space and user space Various information provided by proc https://www.kernel.org/doc/Documentation/filesystems/proc.txt

(3)/proc File System Hardware Kernel  User communication using /proc Get In-Kernel Information Change Kernel Parameters User Program User Interface System Calls Memory Management Process Management I/O File System Network Security Device Drivers User Space /proc File System Kernel Space Hardware

(4)Create a proc entry Download Skeleton Files Compile Useful ref) Proc File System (4)Create a proc entry Download Skeleton Files # wget http://core.kaist.ac.kr/~EE516/Projects/Project2/Makefile # wget http://core.kaist.ac.kr/~EE516/Projects/Project2/proc_sample.c Compile # make Useful ref) http://linux.die.net/lkmpg/c708.html

(5)Test proc entry Kernel Log Insert Module Write Proc File Proc File System (5)Test proc entry Kernel Log # tail -f /var/log/kern.log Insert Module insmod proc_sample.ko Write Proc File # echo blahblah > /proc/proc_sample Read Proc File # cat /proc/proc_sample Remove Module # rmmod proc_sample Kernel Message (printk) Proc Write and Read

Project Tasks Make a System Monitoring Tools Tasks Report System monitoring is an important job especially on the embedded system Low computing power, Less memory, Limited resources Monitoring tools top, ps, netstat, gnome-system-monitor Tasks 1. Analyze the given skeleton procmon.c (10%) 2. Traverse Process - tasklist (10%) 3. per Process Memory Usage (20%) 4. per Process I/O Usage (20%) 5. Sorting Features (20%) Report (20%)

Task 1 Analyze the given skeleton of proc_sample.c Project Tasks Task 1 Analyze the given skeleton of proc_sample.c Find the functions that you do not know and explain the operations of those functions referring to references and Google.

Task2. Traverse Task List Project Tasks Task2. Traverse Task List Every Linux Tasks are managed by doubly linked list Print every task’s PID and Process Name in your proc file system Textbook Chapter 3. Understanding the Linux Kernel, 3rd free e-book version : http://gauss.ececs.uc.edu/Courses/c4029/code/memory/understanding.pdf

task_struct task_struct (=Process Descriptor) Project Tasks task_struct task_struct (=Process Descriptor) KERNEL/include/linux/sched.h Useful materials : Textbook Chapter 3. of ULK

Project Tasks task_struct ………

Task3. Memory Usage Memory Mapping Project Tasks Task3. Memory Usage Memory Mapping RSS : Resident Set Size VIRT : Virtual Memory Size Print every task’s VIRT and RSS Memory Size VIRT RSS Hint) KERNEL/include/linux/mm_types.h struct mm_struct { … unsigned long total_vm; struct mm_rss_stat rss_stat; } VIRT RSS

Task3. Memory Usage Project Tasks include/linux/mm_types.h include/linux/sched.h mm_struct VIRT Memory task_struct … total_vm rss_stat … *active_mm rss_stat.count[MM_FILEPAGES] rss_stat.count[MM_ANONPAGES] RSS Memory MM_FILEPAGES : Type of File Mapped Page MM_ANONPAGES : Type of Anonymous Page (Stack, Heap …)

Project Tasks Task4. Process I/O Stat I/O Stat is stored in struct task_io_accounting include/linux/task_io_accounting.h Hint) # cat /proc/PID/io rchar : read bytes by process wchar : written bytes by process syscr : # of read system calls syscw : # of write system calls read_bytes : read bytes from disk write_bytes : written bytes to disk Process rchar wchar Disk Cache in DRAM read_bytes write_bytes

Task4. Process I/O Stat Hint 1) Hint 2) Hint 3) fs/proc/base.c Project Tasks Task4. Process I/O Stat Hint 1) fs/proc/base.c print function of /proc/pid/io Hint 2) include/linux/task_io_accounting_ops.h task_io_accounting_add() function Hint 3) task_struct->ioac task_struct->signal->ioac for each thread Process rchar wchar Disk Cache in DRAM read_bytes write_bytes

Project Tasks Task5. Sorting Entries Make sort-control proc entry “/proc/procmon_sorting” Implement controllable sorting entry (PID, VIRT, RSS, I/O) Display current sorting order # cat /proc/procmon_sorting Setup sorting order # echo pid > /proc/procmon_sorting # echo virt > /proc/procmon_sorting # echo rss > /proc/procmon_sorting # echo io > /proc/procmon_sorting

Example of Output1 (PID) Project Tasks Example of Output1 (PID)

Example of Output2 (VIRT) Project Tasks Example of Output2 (VIRT)

Example of Output3 (RSS) Project Tasks Example of Output3 (RSS)

Example of Output4 (I/O) Project Tasks Example of Output4 (I/O)

Check your results Check the correctness of your results Task 2. Project Tasks Check your results Check the correctness of your results Use multi-threaded tasks (ex firefox browser) Task 2. # ps -ef Task 3. # top -b -n 1 Task 4. Get pid ps -ef |grep firefox # cat /proc/PID/io Task 5. Correct sorting order

Caution for your score Send your final source code (not every tasks) Project Tasks Caution for your score Send your final source code (not every tasks) Unit (KB, MB, …) should be correct Write comments on your source code /* This code ~~~~~ blah ~~~~ */ Compiling errors and warnings will deduct your points Free your memory when it is not necessary It can cause memory leak Every results should be printed using proc file system Don’t use just printk() function

Problems Problem1. Problem2. Project Tasks Problems Problem1. Describe similarity and difference Process(Thread Group) and Thread in the Linux Answer should include following task_struct, PID management, data sharing and scheduling Problem2. 2.1 Describe why we should use copy_to_user() 2.2 Describe why we should use copy_from_user() functions in the proc file system

Submission Contents Submission Source Code Report Project Tasks1~5 Key point of your Source Code Final Screenshots Answers for the problem 1, 2 Page Limit : about 5 pages for Project Tasks1~5. (except figures) Submission Due Date : Oct. 10 , 23:59 Delay Penalty 10%/day (AM 00:00) E-mail : eu8198@kaist.ac.kr [EE516 Project2] student_number.zip (various compression format is allowed)