Dan Gunter

1
NetLogger Lite

• Dan Gunter
• 14 May 2004

2
What Is NetLogger Lite?

• A simple, easy-to-use debugging, instrumentation,
  and logging toolkit
  
• Based on the NetLogger Toolkit

3
Motivation

• NetLogger has grown
• C API 50,000 lines
• All APIs require the C API!
• Overkill for simple logging
• Makes it hard to incorporate into other packages

4
Enter NetLogger Lite

• Single .c/.h for the C API
• Separate Python, Java API
• Use of C API is optional
• Much smaller, simpler
• Enhanced data model
• New data format (no more
• binary format (whew!))

5
Rest of this talk

• Event data model
• Event format
• C API
• Dynamic log levels
• Python API (v0.1)
• Discussion
• Feedback on above(?)
• DSD Projects that could benefit?

6
Data Model Prologue

• Why bother with a data model?
• Makes it easier to write generic consumers of the
  data for analysis, visualization
  
• Relational databases need as much structure as
  they can get
  
• Elegance if we can describe a general principle,
  we should!

7
Event Data Model
Logged event fields
Standard target fields
Namestring
Timestamptime
Log levelenum
Target-dependent
User-definedstring, number, time
8
Event Record Format
Type code (time, string, int, long, or double)
Single space
Field value
Newline (\n)
t DATE 2004-04-15T213601.425059
Field name
Colon and space
field
File or stream
...
field

9
Sample event records

• t DATE 2004-04-15T213601.422069
• s LVL Debug
• s HOST 131.243.2.143
• s TGT dbg
• s EVNT loop.start
• s FILE ../../samples/sample_logger.c
• i LINE 41
• s FN main
• i start 0
• i stop 10

t DATE 2004-04-15T213601.425059
s LVL Info s HOST 131.243.2.143 s TGT node s
EVNT cpu.utilization.user i VAL 50 s UNITS
t DATE 2004-04-15T213601.425059 s LVL I
nfo s SRC 123.45.67.8 s DST 87.65.43.21 s TGT
path s EVNT delay.roundtrip d VAL 93.4
10
NL-Lite C API Prologue

• There are actually multiple C APIs
• Format an object into a string
• Format a stream of objects
• Format an event belonging to a module to a
  stream using write() with variable arguments
  
• Format a modules event at a given logging level
  for a given target type
  

11
NL-Lite C API

• Create logger
• NL_logger_module()
• Write messages
• NL_write() friends
• Cleanup
• NL_logger_del()

Note The entire C ApI is now in one .c and one
.h file
Less than 3,000 lines of code
12
NL_logger_module overview

• Creates a new (global) module
• Each time you write an event, you write it as
  part of a logger module
  
• An events module determines its..
• Log destination (URL)
• Logging level
• Target type

13
NL_logger_module syntax

• NL_result_t NL_logger_module(
• const char module,
• const char url,
• NL_level_t level,
• NL_tgttype_t ttype,
• ... )

Target types app, debug, and node can pass the
host IP address, or for automatic lookup (by
reverse DNS of hostname)
14
NL_write
Module name
Event name
NL write(level, module, event, fmt, args...)
name pairs, separated by spaces
Log level NL_LVL_NOLOG, FATAL, ERROR, WARN, INF
O, DEBUG, DEBUG1, DEBUG2, DEBUG3
Syntactic sugar for writing these log levels
directly NL_fatal(module, etc.)
Values for each pair in fmt
15
Example

• include nl_log.h
• main()
• int i
• NL_logger_module(foo",NULL, NL_LVL_DEBUG,
  NL_TYPE_DBG, "")
  
• for ( i0 i
• NL_debug1(foo","doit.start","indexi", i)
• / doit() /
• NL_debug1(foo","doit.end","indexi", i)

16
Dynamic Log Levels

• Changing logging on the fly is cool
• NL-Lite has a really simple way of doing this
• NLCONFIGLEVEL/tmp/level myprog
• echo 3 /tmp/level
• You can even do per-module levels
• NLCONFIGLEVEL/tmp/level myprog
• echo 5 foo\n3 /tmp/level

17
Python API

• Two separate APIs
• Homegrown
• Compatible with Python logging API
• Both can work with/without the C library
• Right now, the homegrown API is much faster than
  the Python logging one
  
• Im working on this!
• Cant say for sure the Python API is done

18
Python logging API

• log nllogging.Logger('default')
• handler logging.StreamHandler(strmsys.stderr)
• handler.setFormatter(nllogging.Formatter())
• log.addHandler(handler)
• log.setLevel(logging.DEBUG)
• log.info("loop.start")
• for i in range(0,10)
• log.log(DEBUG,"loop.iter", ("index",i) )
• log.info("loop.end")

Standard Python Logging call
Instead of a string, can also add any number of
(name,value) pairs
19
Java API

• Still in the early stages. Just formats a
  string
  
• NL nl new NL()
• Formatter fmt nl.getFormatter()
• for ( int i0 i
• log.info(nl.setEvent(foo").add("index",i).toStri
  ng())
  
• log.info(Non-netlogger message\n")

20
Compatibility Tools

• There is a (Python) netlogd that can read in the
  nllite format and spit out the ULM (ASCII)
  one
  
• Also a fast, C program to convert ULM logs to
  the nllite, or vice-versa
  
• Some new gnuplot-producing scripts

21
DSD Projects?

• C is the most mature API right now
• NLV ? done!
• Akenti?
• Python is close but still needs some
  requirements feedback and the 6M-man treatment
  
• py,swigGlobus?
• Jabber code
• And of course theres Java, which basically I
  need to finish..
  
• Karlo no longer believes Ill ever finish it

22
URL

• http//www-didc.lbl.gov/nllite/