Multithreading the SunOS Kernel

1
Multithreading the SunOS Kernel

• Eykholt et al.

2
Introduction

• Context enhance SunOS kernel to support
  multiprocessors
• SunOS 5.0 kernel - main OS component of Solaris
  2.0
• Process model changed from heaveyweight to
  lightweight processes (LWPs)
• Example of a real implementation
• Goals
• Capable of high degree of concurrency
• Support multithreading within one process
• Threads that can execute system calls, page
  faults (many to one model wont work)
• Real-time application support
• Bounded dispatch latency
• User control over scheduling
• Preemtion at almost any point within kernel
• Elimination of unbounded priority inversion
  whenever possible

3
Why a Threaded OS?

• Kernel threads used for asynchronous kernel
  activity
• Natural to support asynchronous operations, e.g.,
  writes to disk
• Increases potential concurrency (multiprocessors)
• Allows control of asynchronous activity
  scheduling through priorities

4
SunOS Thread Model
Process 1
Process 2
Process 3

• User thread
• Fast context switch w/o entering kernel
• Allows large number of threads

• Light weight process (LWP)
• Each LWP mapped one-to-one to a kernel thread
• Some kernel threads not assigned an LWP

LWP
LWP
LWP
LWP
LWP
LWP

• Kernel thread
• fundamental entity scheduled and dispatched
• Includes kernel data structure and stack
• Fully preemptable

CPU
CPU
CPU
CPU
CPU

• Implements many-to-many model
• Entire kernel is multi-threaded even interrupts
  handled by kernel threads

5
Data Structures
Virtual Memory Address Space

• Per process data
• List of kernel threads associated with process
• Pointer to process address space
• User credentials
• List of signal handlers

Process
User

• Thread data (not swapped)
• Kernel registers
• Scheduling class
• Dispatch queue links
• Pointers to stack, lwp, process, cpu structures

• Per LWP data structure (swapable)
• User-level registers
• System call arguments
• Signal handling masks
• Resource usage information
• Profiling pointers
• Pointer to kernel thread and process structure

• CPU structure
• Pointer to currently executing thread, idle
  thread,
• Current dispatching and interrupt handler
  information

6
Kernel Thread Scheduling

• Scheduling classes
• System
• Timesharing
• Real-time
• Priority scheduling
• Each thread given a global priority
• Highest priority thread gets CPU (preemption)
• Round-robin within a priority
• Example 1
• Thread A (high priority), B (low priority)
• B releases lock that A is waiting (sleeping) on
• B gives up CPU to allow A to run
• Example 2 - multiprocessor
• Thread A (high priority), B (medium priority), C
  (low priority)
• A and C running on CPUs, B waiting on lock owned
  by A
• A releases lock
• Signal CPU for C to give up CPU so B can run
• Preemption essential for real-time, interrupt
  threads

7
Priority Inheritance

• Recall priority inversion problem a high
  priority thread is blocked on a mutex held by a
  low priority thread, which cannot get CPU cycles
  to run!
• Example
• Thread C (low priority) locks mutex M
• Thread B (medium priority) takes CPU
• Thread A (high priority) runs, blocks on M
  execution returns to B
• B runs for a long time!
• A is locked out of CPU for a long time, even
  though it is the highest priority thread!
• Solution Priority inheritance
• Once A blocks on M (waiting on C), C gets
  (inherits) As priority

8
Recursive Mutex

• If a thread locks a mutex, then tries to lock it
  again, it will deadlock!
• Recursive mutex can be locked multiple times by
  the thread owning the lock w/o blocking
• Must be unlocked the same number of times (once
  for each lock operation) to allow others to
  access the critical section

9
SunOS Synchronization Architecture

• Kernel implements the same synchronization
  mechanisms for internal use that are provided by
  user-level libraries
• Mutual exclusion, condition variables,
  semaphores, multiple readers, single writer
  (readers/writer) locks
• Mutex, writer locks support priority inheritance
• Does not support recursive mutexes
• Argue poor programming practice

10
Lock Implementation

• To spin or not to spin Adaptive mutex
• Motivation If locks are held for a short time,
  probably OK to spin
• Lock causes a thread to spin, so long as the
  thread holding the lock runs on the CPU
• If thread holding lock goes to sleep, end spin
  and go to sleep
• What happens on a uniprocessor?
• Optimize storage used by synchronization object
  (turnstyle)
• Motivation many synch objects, each needs a
  queue of threads waiting on that object but most
  of the queues are empty
• Do not store queue header (for sleeping threads)
  in object
• Rather, when first thread goes to sleep on lock,
  dynamically allocate a queue (turnstyle), and
  store 2 byte index in synch object that acts a
  pointer to queue header and priority inheritance
  information return turnstyle to free list when
  last thread waiting on synch object wakes up

11
Interrupts as Threads

• Suppose (traditional OS)
• A thread locks a mutex, enters critical section
• An interrupt occurs, and the thread handling the
  interrupt tries to lock the same mutex
• Deadlock!
• A way out
• Raise interrupt level before locking mutex, lower
  interrupt level after releasing lock
• Bad (1) raising/lowering interrrupt level may be
  expensive, (2) interrupt level may be high for
  too long, delaying interrupt handlers
• SunOS solution
• Interrupt handlers are implemented as
  asynchronously created high priority threads
• Use standard synchronization primitives to
  control access to critical sections interrupt
  handlers can sleep if necessary

12
Implementing Interrupts as Threads

• Traditional approach
• Interrupted process stopped (pinned) until the
  interrupt returns
• SunOS 5.0 implementation
• Interrupt asynchronously create new thread
• But, full thread creation for each interrupt is
  too expensive!
• Solution
• Initially, interrupt handled much like the
  traditional approach in that the interrupted
  thread is pinned interrupt is not a separate
  thread
• If interrupt thread blocks on synchronization
  variable (mutex or condition variable), a full
  fledged thread is created that goes to sleep, can
  run on another CPU
• Interrupt level adjusted so lower level
  interrupts are ignored
• Pre-allocate interrupt threads to speed up
  creation

13
Performance

• 40 SPARC instructions per interrupt
• Savings of 12 instructions per mutex (avoid
  raising interrupt level, etc.)
• Space per pre-allocated interrupt thread - 8K per
  thread, 9 interrupt levels for SPARC