StarTrace: e2e PathBased Tracing

1
(Star)Trace e2e Path-Based Tracing

• James Zhang, Jesse Trutna, Armando Fox, Dave
  Patterson

2
RAD Datacenter / Node
(datacenter cluster or RAMP)
Policy Maker
Per node SW stack
Load- Balancer (IDLB)
Intrusion- Detection (IDID)
Service (IDS)
Web 2.0 Applications
Firewall (IDF)
Ruby on Rails Interpreter
Web Svc APIs
Trace, X-trace
Local OS functions
trace, X-trace, Lib log, D-trigger,
Identity-based Routing Layer,
Virtual Machine Monitor
Actuator Commands
Sensor Data
3
Overview

• (Star)Trace is a framework for end-to-end
  path-based request tracing in distributed web
  applications
• Trace leverages existing tracing tools
• X-Trace (Berkeley in development)
• D-Trace (Sun Microsystems production code)
• Trace aims for low overhead (12) to make
  tracing feasible in production
• We have build a preliminary implementation in
  Ruby-on-Rails and we are in the process of
  improving and evaluating the system

4
Motivation

• Past work demonstrated that path information can
  be extremely useful for
• Detecting and diagnosing failures
• Identifying problematic interactions and
  bottlenecks
• Obtaining a global view of distributed systems
• Most current production web apps cannot track the
  path of a request
• Path information can be used as input to machine
  learning algorithms to optimize applications

5
A Simple Example
XML-RPC
FCGI Request
HTTP POST
lthtmlgt ltbodygt .
SQL Insert
Frag. Cache Static Cont.
6
A Simple Example
HTTP GET
Hits page cache
lthtmlgt ltbodygt .
FCGI request
7
A Simple Example
FCGI Request
HTTP POST
SQL Select
lthtmlgt ltbodygt .
8
Problem

• Of course, all requests are interspersed and
  affect each other.
• As applications evolve and expand, things get
  ugly quickly.
• Bottlenecks and errors becomes very hard to
  diagnose.
• Path-based tracing is a good idea.

9
Design Considerations

• Instrumentation in the middleware to avoid source
  changes in business logic
• J2EE Containers
• Ruby interpreter Rails library/framework
• Minimum overhead (12) to allow always on
  tracing in production
• Architecture should be easily implemented in
  different middleware frameworks

10
Trace Overview
11
X-Trace Trace

• Trace uses X-Trace identifiers (id/xid)
• Requests are assigned probabilistically unique
  ids at each component (web server, app server,
  database, etc.)
• Ids parent-child information used for path
  reconstruction

12
Logging Trace

• Most web application components already performs
  logging
• Apache, Ruby on Rails, Tomcat, MySQL
• Modify logging framework to insert id in the
  logs
• ids are delimited differently for fast pattern
  matching
• Production logging overhead is low and carrying
  Trace information adds no measurable overhead

13
Logging Overhead

• Different logging levels exists
• Example RoR has production or development level
  logging, in addition to webserver logs and/or
  database logs.
• On a Linux VM
• No logging --gt production logging 2.5
• Production logging --gt development logging 5.5
• On a Solaris VM
• No measurable effect

14
Prototype Implementation

• Preliminary implementation of Trace exists for
  Ruby-on-Rails
• Apache and Lighttpd webserver
• Propagate and log Trace id
• Rails framework
• Stores Trace in thread-local variable
• Propagate and log Trace id
• Database
• SQL queries are tagged with Trace id as comments
• Deployed as a plug-in, require no modifications
  to the application code
• Path-Analysis can be done online or offline

15
Dtrace Add-on

• Dtrace is a ubiquitous tracing tool
• Currently in Solaris, will be in FreeBSD and OS X
• Trace anything as long as theres a provider
• Created a custom Trace provider
• Pass id into dtrace with a dummy function call
• Loupe Coordinate dtrace with RoR dispatchers
• Runs dtrace scripts from a central location
• Can use arbitrary dtrace scripts
• More detailed tracing on demand

16
Dtrace Trace
17
Path Reconstruction

• Implemented a Version 0 tool for path
  reconstruction/visualization
• Lumberjack extract data from logs
• Apache Log, Rails Log, MySQL Query Log
• SQLite save extracted data
• id, timestamp, host, and other data (url, sql,
  errors)
• Display paths using an AJAX website

18
Trace Demo Version 0
19
Experience So Far

• Even in this small example trace was useful
• Found some requests generated over 1000 SQL
  requests
• Some SQL requests repeated over and over
• Example show fields from XX

20
Future Work

• Instrumentation BackPlane (IBP)
• Need a better infrastructure to gather path
  information and reconstruct paths.
• See Poster
• Implement Trace for other web frameworks
• Java Servlets
• Python (Django), Perl, PHP?
• Evaluation with real websites with real users