Recent Changes - Search:

Research

Notes

Architecture

Faults

System

Planning

Background

OS

Misc

edit SideBar

KernelDataStructures

Investigating Kernel Data Structures


January 06, 2015

The Problem

For some reason the fork() call is taking longer and longer for my restarted replicas (best documented in my SO question). First I tried to find the cause of the problem using kmemleak, but that revealed nothing. Which is good in a sense: it probably means that a data structure in the kernel is growing but that there is a way to access it.

So my next attempt will be to take a closer look at those structures, starting with the rather monstrous task_struct. The task_struct for a process is only available in kernel space, so I will need to insert code in the kernel, and then have a process invoke said code when I want to take a peek. Which sounds to me like I'll be adding a syscall and a kernel module.

Preliminary Setup

I am going to use QEMU to run a virtual machine of Ubuntu 12.04(instructions). I prefer the server edition for this since I don't need a gui. I had to use Ubuntu 10.04 for the guest OS since later versions always hang on startup.

Create a drive (10G was not sufficient):

  host> qemu-img create drive-name 24G
  host> qemu-img create -f qcow2 drive-name.img 24G

Install OS on drive:

  host> qemu-system-x86_64 -enable-kvm -hda drive-name.img -cdrom ~/ubuntu-10.04.4-server-amd64.iso -m 1024 -boot d

Follow the normal installation path. So bloody easy.

To run without the "cdrom" in (-enable-kvm for significantly better performance, but guest must be same arch as host):

  host> qemu-system-x86_64 -enable-kvm -hda drive-name.img -m 1024

Take a snapshot if you would like:

  qemu-img create -f qcow2 -b drive-name.img snapshot.img

From now on you should use snapshot.img as the drive.

The rest is being run from inside the VM. To do anything useful, I'll need to install some software packages:

  guest> sudo apt-get update
  guest> sudo apt-get upgrade
  guest> sudo apt-get install build-essential emacs git-core libncurses5-dev kernel-package

Update grub so that the resolution is better, and so that you see different kernel options:

  guest> sudo emacs /etc/default/grub

Change the following lines:

    GRUB_HIDDEN_TIMEOUT=
    GRUB_CMDLINE_LINUX_DEFAULT="splash vga=792"
  guest> sudo update-grub

Adding a Syscall

While you are in there, you may want to increase the size of dmesg's buffer: kernel/printk.c: #define LOG_BUF_LEN

Log buffer length is now a config option. General Setup -> Kernel log buffer size

Fortunately for me, adding a syscall to the kernel is a OS course staple. Unfortunately, I want the syscall code itself to actually be handled by a kernel module, which is less standard. This is usually a bad idea, since the module may or not be loaded.

First, grab your kernel's source:

  guest> wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.2.55.tar.gz
  guest> gunzip linux-3.2.55.tar.gz
  guest> tar -xvf linux-3.2.55.tar

More recent kernels have a more streamlined way to deal with x86_32 and x86_64 having similar system calls. For information on the 3.14 kernel, see here: https://shanetully.com/2014/04/adding-a-syscall-to-linux-3-14/.

I am using the 3.2.55 kernel, so the process is a bit more redundant.

Edit arch/x86/kernel/syscall_table_32.S to have a new entry for the new system call:

Update the number of Syscalls extern the table?

Building / Testing

From the linux source directory. I really need to figure out what I can deselect in make menuconfig to speed things up here.

  guest> fakeroot make-kpkg --initrd --revision=3.2.55.task kernel_image
  guest> sudo rm -rf /lib/modules/3.2.55/*
  guest> sudo dpkg -i ../linux-image-3.2.55_3.2.55.task_amd64.deb
  guest> sudo update-initramfs -c -k all
  guest> sudo update-grub
  guest> sudo update-grub2

I need to add all of the code here. The kernel module / syscall intercept is interesting enough I think.


Unfortunately investigating task_struct did not reveal the problem. Had to do a binary printk search which took days. but worked. Fill in details here.

Edit - History - Print - Recent Changes - Search
Page last modified on January 18, 2015, at 02:44 PM