Temporal Inventory and realtime synchronization in RTLinuxPro

1
Temporal Inventory and real-time synchronization
in RTLinuxPro

• Victor Yodaiken

2
Points of discussion

• What is RTLinux?
• Intro to Synchronization in RTLinux
• Basics/Rules of Real-Time Programming
• Goals
• Priorities
• Too many levels
• Too much pre-emption
• Priority Inversion

3
Points of discussion..

• 4. Just-in-Time Scheduling
• i) Scheduling Just-in-Time
• ii) Asymmetric just-in-time and non-blocking I/O
• iii) Event driven scheduling
• iv) Interrupt handlers
• - interrupts and semaphores
• v) Lock-free structures

4
Points of discussion..

• 5. Unavoidable synchronization
• i) Atomic Code block
• ii) Guarded Critical sections
• - Priority and semaphores
• iii) Avoiding priority inversion

5
What is RTLinux ?

• RTCore is a POSIX real-time kernel
• multi-threaded POSIX process with its own
  internal scheduler.
• RTLinux is RTCore with Linux as the secondary
  kernel.
• Secondary operating system is the idle thread for
  the real-time operating system

6
What is RTLinux ? ..

• Real-time applications run as real-time threads
  and signal handlers either within the address
  space of RTCore or within the address spaces of
  processes belonging to the secondary kernel.
• Real-time threads are scheduled by the RTCore
  scheduler without reference to the process
  scheduler in the secondary operating system.

7
Introduction to Synchronization in RTLinux

• Temporal Order on Software systems
• Use of more powerful hardware
• Temporal Inventory

8
Basics of Real Time Programming.Goals

• Goals of Real-Time Programming
• Synchronization operations must be
• Fast
• Take up a small percentage of total computation
  time
• Do not deadlock and
• Do not defeat modularity
• Worst-case delays are bounded and small
• Interactions among logically distinct components
  do not cause unacceptable timing changes.

9
Basics of Real Time ProgrammingPriorities

• Real-time programming is generally based on a
  priority ranking of tasks
• Too many levels
• - Rationale behind different levels
• - Rate Monotonic Scheduling

Rule 1 Nothing should delay the highest
priority runnable task
Rule 2 Do not add priority levels without
rationale
10
Basics of Real Time ProgrammingPriorities

• Worst-case timing analysis
• Uncertainties of cache misses, DMA, memory access
  etc.
• Testing and approximate analysis

• Rule 3 Design so that you can reliably test
  worst case timing.
• Keep real-time tasks and primary control path
  simple
• Run tests for a long periods under varying
  worst-case loads.

11
Basics of Real Time ProgrammingPriorities

• Priority Inversion
• Violation of Rule 1 due to synchronization
• Task A uses resource R (exclusively)
• Higher priority task B becomes runnable
• Task B also requires resource R
• Task B has to wait for Task A completion

12
Just-in-time (JIT) Scheduling

• Best Synchonization is no synchronization.
• Task scheduled to start at time t
• Delays in waiting and negotiating for resource
  takes time w.
• Task starts at tw.
• Relax deadlines and build slack time.

• Rule 4 Schedule just-in-time
• No 2 real-time tasks to be scheduled to run
  during a common interval
• No scheduled tasks are blocked waiting for a
  resource

13
JIT Scheduling..Example

• Example 1 Stagger data acquisition task and a
  control output task.
• Alternating threads input
• task_input
• clock_gettime(CLOCK_REALTIME,t0)
• // read the time
• while(!stop)
• clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME,
  t0, NULL)
• /DEBUG TEST/if(sem_trylock(overlap_sem))fail(
  )
• collect_data_from_a2d_device(data)
• queue_data(data)
• timespec_add_ns(t0,INPUT_DELAY_NS)

14
JIT Scheduling..Example

• Example 2 Stagger data acquisition task and a
  control output task.
• Alternating threads output
• task_output
• clock_gettime(CLOCK_REALTIME,t1)
• // read the time
• timespec_add_ns(t1,OFFSET_NS) // compute start
  time
• while(!stop)
• clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME,
  t1, NULL)
• /DEBUG TEST/if(sem_trylock(overlap_sem))fail()
  
• dequeue_data(data)
• compute_output()
• output_control()
• timespec_add_ns(t1,OUTPUT_DELAY_NS)
• /DEBUG TEST/sem_post(overlap_sem)

15
JIT Scheduling..Asymmetric JIT and non-blocking
I/O

• Use of non JIT techniques for non-real time
  tasks.
• Application contains a real-time component and
  non real-time component.
• Eg. Collection video frames from camera and
  writing to file.

• Rule 5 Asymmetric just-in-time
• Pre-empting and blocking non real-time tasks is
  not a violation of just-in-time scheduling rule.

16
JIT Scheduling..Asymmetric JIT and non-blocking
I/O

• Example 3 Using asymmetric FIFO
• Task 1 Reading from a2d device and writing to
  rt-fifo
• clock_gettime(CLOCK_REALTIME,t)
• while(!stop)
• collect_data_from_a2d_device(data)
• write(fd,D,sizeof(D)) / async write to rt-fifo
  /
• timespec_add_ns(t,DELAY_NS)
• clock_nanosleep(CLOCK_REALTIME,TIMER_ABSTIME, t,
  NULL)
• Task 2 Reading from rt-fifo and writing to file
• cat lt /dev/rtfifo0 gt logfile

17
JIT Scheduling..Event driven scheduling

• Blocking operations cause a task to be suspended
  until data is available or lock is released.
• Reason Re-use of non-real-time programming
  techniques in the wrong place.
• Example 4 Blocking producer (non-event driven)
• Thread A
• do
• read data by a blocking operation
• process the data
• if space on output queue
• enqueue data
• else
• discard data
• while(not done)

18
JIT Scheduling..Event driven scheduling

• Example 5 Blocking consumer (non-event driven)
• Thread A
• do
• semaphore decrement. / the interrupt routine
  does post/
• do something
• Yield
• while(not done)
• Example 6 Semaphore powered event driven
• High frequency thread checks temperature, does
  a simple operation and wakes up a thread when the
  temperature rises above a limit.

19
JIT Scheduling..Event driven scheduling

• Example 6 Semaphore powered event driven.
• check_temp_thread()
• if(temp gt 100) sem_put(sem_help_me)
• timespec_add_ns(t0,SMALL_PERIOD_NS)
• clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME,
  t0, NULL)
• ...
• help_thread(void)
• sem_get(sem_help_me)
• / sound the alarm /
• ...

20
JIT Scheduling..Interrupt handlers

• Interrupt handlers or ISRs are functions that are
  activated asynchronously by hardware events
• Run in the context of the thread that was
  executing
• While being started, the handler disables further
  interrupts and the current interrupt is masked
• While it is ready, the handler will unmask
  current interrupt but never re-enable interrupts

• Rule 6 Interrupt service overhead minimization
• Least possible context should be saved and
  restored during an interrupt service operation

21
JIT Scheduling..Interrupt handlers

• Example 9 Interrupt Handler
• rtl_request_irq(A2DDEVICEIRQ,handler)
• fd fd_normal
• ..
• handler()
• collect_data_from_a2d_device(data)
• reenable_the_device() // but interrupts are
  still disabled on this cpu
• if (write(fd,data,DATA_SIZE) ! DATA_SIZE)
• if (!overflow)
• write(fd_error,DATA_SIZE))
• overflow 1
• fd fd_overflow
• sem_post(failure_sem)
• // else drop the data
• return

22
JIT Scheduling..Interrupts and Semaphores

• Semantics of sem_post operation
• sem_post(s)
• atomically
• increment s
• if there are waiters wake the highest priority
  one
• if we woke a higher priority thread, on our CPU,
  switch.
• Example

23
JIT Scheduling..Lock Free Structures

• Test and Set Instructions (rtl_test_and_set )
• Atomically sets the bit passed as argument and
  returns the previous value.
• Works on microprocessors and uniprocessors
  equally well
• Interrupt handler safe
• Cache Ping Pong problem

• Rule 7 Lock Free data structures
• Where there are fast algorithms, use lock-free
  structures to avoid synchronization.

24
JIT Scheduling..Lock Free Structures

• Example Set of concurrent tasks sharing a
  collection of buffers
• struct mybuffer unsigned int flags char
  dataDATASIZE
• struct mybuffer BNUMBER_OF_BUFFERS 0,
  //zero it
• struct mybuffer allocate_mybuffer(void)
• int i
• for(i0 ilt NUMBER_OF_BUFFERS i)
• if(!rtl_test_bit_and_set(0,Bi.flags))
• return Bi
• return (struct mybuffer )0
• void free_mybuffer (struct mybuffer b)
• b-gtflags 0 // store is atomic
• return

25
JIT Scheduling..Lock Free Structures

• RT Core one-way queue
• Circular array based Queue
• Head pointer and tail pointer are not shared
  (consumer needs head, producer -gt tail)
• Do not need locks or test n set operations
• Works when there is a single reader and single
  writer
• Example 11
• include ltonewayq.hgt
• typedef struct int flags unsigned char dSZ
  mypacket_t
• BUILD_OWQ(pq_t,128,mypacket_t,-1)
• pq_t txq,modifyq
• rx_handler()
• if(norxerror())
• if(-1 pq_enq(modifyq,newpacket())) fail()

26
JIT Scheduling..Lock Free Structures

• tx_handler()
• / one transmit done. throw away sent packet /
• m pq_deq(txq)
• free_message(m)
• ...
• modify_thread()
• while(!stop)
• while( (m pq_deq(modifyq))
• process(m)
• pq_enq(txq,m)
• timespec_add_ns(t0,DELAY_NS)
• clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME,
  t0, NULL)

27
Unavoidable Synchronization

• Types of Synchronization
• Atomic Code block on Uniprocessors
• Disabling interrupts
• Eg.
• rtl_stop_interrupts()
• do some work // no preemption, no interrupts
• ...
• rtl_allow_interrupts()
• Problems
• Already disabled interrupts
• Length of do some work
• Multiprocessor systems??

28
Unavoidable Synchronization

• Atomic Code block on Multiprocessors
• Spin locks
• pthread_spin_lock(myspin)
• critical section // no preemption, no interrupts
• ...
• pthread_spin_unlock(myspin) // in a
  uniprocessor myspin is not locked
• Spin lock implementation
• void pthread_spin_lock(rtl_spin_t s)
• while(rtl_test_bit_and_set(0,s-gtlock))
• // dont use the expensive test and set,
• while( (volatile)s-gtlock 1)

29
Unavoidable Synchronization

• Semaphores can be used to guard critical sections
  of code
• A binary semaphore (mutex) may be locked as a
  thread enters a critical section and unlocked as
  the thread leaves the CS
• Critical section must be as small as possible

Rule 8 To put a critical section within
semaphores in order to allow pre-emption during
the section, shorten the critical section.
30
Unavoidable Synchronization Avoiding Priority
Inversion

• If rule 9 is violated, then
• 1. Make sure that there are no just-in-time
  solutions, the slow operations cannot be
  delegated to the non-real-time side, and there
  are no lock-free solutions. If this fails, go to
  the next step.
• 2. Apply rule 8 and make the critical section so
  fast that spin locks work better.

Rule 9 Semaphores guarding critical regions
should never be shared by tasks of different
priorities,
31
Unavoidable Synchronization

• 3. If it is absolutely required for a slow
  atomic operation on the data structure, design
  the system to make sure only tasks with the equal
  priority access this data structure.
• 4. Priority ceiling is an indirect method of
  building a server.