DUSK: Develop at User level, inStall in Kernel

1
DUSK Develop at User level, inStall in Kernel

• Alexey Smirnov and Tzi-cker Chiueh
• ECSL Research Seminar
• 09/13/05

2
Outline

• Introduction
• Related Work
• Netfilter Overview
• System Architecture
• Evaluation
• Conclusion

3
Introduction

• Kernel is often extended using kernel modules.
• Netfilter hooks in network component.
• Development at user-level is convenient gdb,
  ddd, etc.
• Kernel modules have lower overhead.
• DUSK achieves best of both worlds.

4
User-level Development of Kernel Extensions

• Provide kernel API at user-level. A lot of
  functions, changes from one version to another.
• Implement a new API that resembles kernel API.
  Not convenient for the programmer, requires code
  rewriting.

5
DUSK Overview

• DUSK is an extension to GCC that compiles the
  source code of a kernel module into a user-level
  program.
• It links several helper functions to the
  user-level module.
• Same helper functions for different versions of
  Linux kernel.

6
DUSK Framework

• User-level component the kernel module compiled
  as a user level program.
• Kernel-level component handles user-level
  modules requests.
• The two components are connected using UDP
  sockets.
• Can run on different architectures user-level
  module on x86 running in gdb and kernel-level
  module on MIPS wireless router.

7
Related Work

• VFS layer UFS uses NFS loopback API. UpcallFS
  a stackable file system.
• Device drivers Userdev provides a user-level
  library.
• Network stack icTCP exposes TCP stack parameters
  to user programs. Netfilter simulator runs
  Netfilter hooks at user level.

8
Netfilter Architecture

• Netfilter adds a set of five hooks to kernel
  network stack

• A Netfilter module verdict NF_ACCEPT, NF_DROP,
  NF_STOLEN, NF_QUEUE, NF_REPEAT.

9
Netfilter User Mode Modules

• NF_QUEUE verdict sends sk_buff to user space
  using a netlink socket. The queue handler is
  either standard or programmer-defined.
• A user module can only read sk_buff but cannot
  change kernel memory.

10
iptables

• iptables controls parameters of Linux firewall.
• iptables -A PREROUTING p tcp --destination-port
  80 j QUEUE
• iptables A POSTROUTING p udp --source-port 5678
  j DROP
• Has string matching capabilities. Stateless IDS.

11
Kernel-level Programming at User Level

• User-level kernel modules need to
• Read/write kernel memory
• Call kernel functions
• Access global kernel variables, e. g. current
• Programmers can modify source code of kernel
  module using these blocks to run it at user level.

12
Three Basic Blocks

• Kernel memory access /dev/kmem
• Kernel function calls new system call
  sys_exec(). System.map for address lookup.
• A new system call sys_kvar() for variable access.
• sys_exec() and sys_kvar() are implemented in DUSK
  kernel module.
• Kernel module and user module communicate using
  UDP sockets.

13
Using Basic Blocks to Run Kernel Module at User
Level

• Kernel module source code
• sk_buff-gtlen
• User-level code
• Define a shadow variable u_sk_buff
• malloc() memory for it
• Use kmem_read() to copy sk_buff from kernel
  memory to shadow variable
• Commit changes to u_sk_buff using kmem_write().
• Our goal automatic transformation.

14
DUSK Framework
15
DUSK Run-Time Support

• Kernel extension registration/de-registration.
  The address of a function is sent to the kernel.
• Processing kernel requests. An appropriate
  extension handles each request.
• Module initialization/cleanup. init_module() and
  cleanup_module() are called when user-level
  module starts and terminates.

16
Extension Management

• nfho_post_routing.hookkhook_ip_input
• nfho_post_routing.hooknum NF_IP_POST_ROUTING
• nf_register_hook(nfho_post_routing)
• Kernel-level DUSK module replaces khook_ip_input
  with a proxy kernel function and returns
  extension type and sequence number to the user
  program.
• DUSK clones a new process for each extension.
  They share same address space.

17
Blocking and Non-blocking Extensions

• Extensions that block the process that called
  the extension blocks until the user-level module
  returns a verdict. Exampe procfs, timer.
• Extensions that dont block the kernel callback
  returns as soon as a message is sent to the user
  space. Example Netfilter.

18
Why Netfilter Is Non-blocking?

• Hardware interrupt handler receives network
  packets.
• Runs with interrupts disabled. It saves packets
  in a queue and invokes software interrupt, then
  returns.
• One software interrupt handler per processor.
  Hardware interrupts enabled.
• A packet is re-injected into the network stack
  when user-level processing is finished.

19
Extension Types

• DUSK supports Netfilter only.

• Timers, threads, procfs, ioctl, etc. are required
  for advanced modules.

20
DUSK Run-Time Configuration
21
Compile-time Support

• DUSK adds two functions check() and commit().
• check() ensures that every buffer accessed using
  a pointer exists in kernel memory and in
  user-space.
• commit() updates the kernel version when a
  function is called it refreshes the user version
  when the function returns.

22
Function check()

• check() is called for every array access,
  de-referenced pointer, or .
• It maintains the kernel-to-user address mapping
  buffer.
• For a new user address DUSK calls kmalloc() and
  kmem_write()
• For a new kernel address DUSK calls malloc() and
  kmem_read().
• Addresses under PAGE_OFFSET are user addresses,
  above are kernel addresses.

23
Type Information

• Type information is gathered at compile-time.
  Type size, offsets of pointer fields are stored.
• Type size is used in check().
• commit() uses field information.

24
Kernel Function Calls

• A kernel function is replaced with
• void_kernel_func() for void functions
• int_kernel_func() for int functions
• Function arguments are written into kernel memory
  when the function is called.
• Each pointer-type argument is wrapped into
  commit() call.

25
Function commit()

• commit() traverses data types recursively.
• Issue array elements accessed using a pointer
  (a1), (a2). If a is a function parameter then
  commit() ensures that all elements get
  synchronized.
• Issue a unions elements of different type have
  same offset. Types of the argument and that of
  the address mapping table are compared.

26
Function commit()
27
Function Prolog and Epilog

• All changes are committed when user-level module
  returns a verdict.
• DUSK adds a function epilog that commits the
  changes.
• depth variable counts the current nesting level.
• Incremented in the function prolog, decremented
  in the epilog.

28
check() and commit() Example

• printk(test)
• int_kernel_func(addr_of(printk),
  commit(check(test)))

29
Cross-Platform Debugging

• Linksys WRT54gs wireless router 32 MB RAM, 200
  MHz MIPS CPU. Runs Linux 2.4.20.
• Cannot run gcc, gdb. Supports Netfilter.
• printk() debugging possible. Logs are not stored
  after reboot.

30
X86 and MIPS Differences

• Data structures are different

• PAGE_OFFSETs are different.
• DUSK compiles module twice with the
  cross-compiler and with x86 compiler.
• Addresses in data structures are changed when
  data from kernel is received.

31
Evaluation

• nfsiff FTP password sniffer
• POSTROUTING finds USER and PASS commands
• PREROUTING replies to special ICMP packets
• lwfw light-weight firewall
• PREROUTING drops certain packets (port,
  interface).
• ipp2p P2P traffic classifier
• POSTROUTING scans for P2P signatures.

32
nfsiff

• Sniffer on wireless router

33
Compile-time Overhead
34
Saved Work
35
Conclusion

• DUSK allows to debug kernel modules at user level
  without changing the source code.
• Future work add better debugger support, i.e.
  integrate DUSK with gdb and ddd.
• Support device driver development.