Semantic Web based Log Analysis (for Distributed Systems and Applications)

1
Semantic Web based Log Analysis(for Distributed
Systems and Applications)

• ACET Seminar, University of Reading24th May
  2007
• Richard BoakesDistributed Systems Group,
  University of Portsmouth

2
Introduction

• Distributed Systems are
• a superset of clusters, Grids and parallel
  computing systems.
• composed of heterogeneous computers,
  applications, middleware and networks.
• This talk
• describes why post mortem analysis of Distributed
  Systems is difficult.
• describes how Semantic Web technologies can
  provide a basis for unifying the heterogeneous
  log data.
• presents system called Slogger that was used to
  prove the concept.

3
A Distributed System

• In a distributed system, nodes provide and
  consume services, by cooperating with other
  nodes.

4
Service Aggregation

• Nodes may provide services by aggregating
  services of other nodes.

5
Node Log Sources

• Commonly, nodes may contain servers which run
  applications that require libraries to function.
• Each element may record its operation in a
  logfile.

6
Multiple Node Logs
7
Other Logs
8
Everything Logs
9
More Complex System Composition
10
Multiple Logs
11
Heterogeneous Logs
12
Additional Complexity

• So on top of the fact that there are
• different applications and processes,
• recording data about about different things,
• for different purposes,
• potentially in multiple global locations.
• There are issues of
• Time Synchronisation
• Dynamic Runtime Discovery
• Security, Trust and Log Retrieval
• Data Size

13
Slogger Project Aims

• The aim
• Unify disjoint and loosely defined, heterogeneous
  data in order to simplify the task of analysing
  and understanding the status, failures, or errors
  that may occur in distributed systems and their
  applications.
• The technique
• the definition of vocabularies describing the
  contents of log files, including
• A general log vocabulary enables the combination
  of previously incompatible log records into a
  single storage and retrieval system.
• Subject specific vocabularies allow more precise
  descriptions data within each type of log.
• The benefit
• Concepts in subject specific vocabularies may
  overlap (e.g. CPU utilisation). Through
  inference, similar concepts can be treated as the
  same data type.
• The data store is not restricted to a single
  database schema and can evolve over time. As
  additional relevant log files are made available,
  they can be integrated.
• Open standards are used throughout, and the base
  of the system is compatible with logs in any
  language.

14
Slogger Project

• Review of existing parallel performance analysis
  post mortem tools.
• Investigated Semantic Web standards and
  technologies.
• Adopted an iterative approach to design and
  development of a proof of concept.
• Began with an Outline framework based on the
  logical process of unification.
• Slogger Framework emerged.
• Used open-source free technologies throughout.
• Released under the GNU Public License, so it too
  is open source and free.

15
The Semantic Web Stack
16
Graph Data Model

• Graph data model
• Comprising Nodes and Arcs
• Representing Subjects, Properties and Values

17
Represent Any Structure

• For Example, a database table

18
The Outline Framework
19
The Slogger Framework
20
GULF Schema
21
Example Log

• Thu 17th Feb 2005 113001 bd /tmp/eer
• Thu 17th Feb 2005 113001 gm /tmp/rjb
• Thu 17th Feb 2005 113001 wd /tmp/eer /tmp/rjb
• Thu 17th Feb 2005 113002 lv hea /tmp/cpl

22
Example Log
23
CLF Example
24
CLF Example
25
Visualisation Influences
26
Visualisation Goal
27
Visualisation Prototype
28
Visualisation Component
29
Time Problems

• Log Granularity
• Quantum Size
• Clock Drift

30
ClockSync
31
Message Annotation
32
Adjustment Annotations
33
Adjusting Data
34
Property Equivalence
35
Property Equivalence
36
Demo

• Data has been gathered using the Slogger
  Framework and will be shown in a browser, using
  the SVG Visualisation Component.

37
The Slogger Framework
38
Testing Evaluation

• Over 500 end-to-end tests performed (logging
  through to visualisation).
• Progressively more complex.
• Proved that Semantic Web technologies are useful,
  and highlighted problem areas.

39
Test Harness
40
Test Logs

• System profile data was recorded using background
  scripts
• /proc/memfree
• /proc/vmstat
• /proc/loadavg
• Environment data
• Network Ping Time
• Programme trace data
• Middleware communications gathered from
  MPJ-Express (MPJE)

41
Tests

• Baseline tests
• establish how Slogger records and represents data
  from an unloaded, and artificially loaded,
  sequential system.
• Linpack tests
• establish how the operation of a sequential
  computational program appears within Slogger.
  Various additional loads were applied to the test
  node and the variances from the basic Linpack
  performance was analysed.
• MPJ-Tests
• introduced more nodes to test Sloggers
  capabilities for handling larger amounts of data,
  and analysing the operation of more complex
  situations.
• Ping-Pong tests
• introduced a second node so program operation was
  dependent on communication between nodes. Basic
  operation was measured and described, then
  further tests were run to apply different types
  of load so the programs response could be
  analysed.

42
Baseline Test
43
Baseline Stress Test
44
Baseline Stress Zoom
45
Trace Topography
46
Trace plus Profile
47
Trace plus Profile Zoom
48
Linpack Profile
49
Linpack with VM Stress
50
Linpack with VM Stress Zoom
51
Deeper Zoom
52
Message Passing
53
MPJ Express Profile
54
Adjusted MPJ Express
55
MPJ Express Messages
56
Eight node Example
57
Ping Pong Test
58
External Fault
59
External Fault
60
Semantic Web Experiences

• Transformation Performance
• Log filtering
• Query Performance
• Graphs are flexible, but slow
• Computational Performance
• Data collapsing necessary
• Schema Design
• No (realistic) best practice

61
Query Performance
62
Collapsing Data
63
Schema Design
64
Conclusion

• This talk
• described why post mortem analysis of Distributed
  Systems is difficult.
• described how Semantic Web technologies can
  provide a basis for unifying and analysing
  heterogeneous log data.
• Demonstrated a system called Slogger that was
  used to prove the concept.
• Discussed some of the issues highlighted by the
  Slogger project.

65
Future Work

• Two key areas
• Visualisation
• Improved generation and interaction
• Alternative views
• Unification
• Transformer speed
• Transformer diversity
• Community involvement in schema and code
  development

66
Thanks

• Questions?

67
(No Transcript)
68

• Supplemental Slides

69
Quantum Size
70
Anchoring Hi-Res Logs

• A solution in some cases

71
ClockDrift
Machine A
Tx
Tx
Rx
Rx
1
3
2
4
Events
a
Machine B
Rx
Rx
Tx
Tx
1
3
2
4
Events
Rx
Tx
Tx
Rx
2
1
3
4
Machine A Log
b
Rx
Tx
Rx
Tx
3
2
1
4
Machine B Log
Rx
Tx
Tx
Rx
Rx
Tx
Rx
Tx
2
1
3
3
4
2
1
4
Unified Log
Rx
Tx
Tx
Rx
2
1
3
4
Machine A Log
Rx
Tx
Tx
Rx
3
4
2
1
Machine B Log
c
Tx
3
Rx
Tx
Rx
Tx
Rx
Tx
2
1
4
2
1
4
Unified Log
Rx
3
72
Adjustment Annotations
Period of
Operation
Machine A
x
x
x
x
x
1
3
4
5
2
Events
Machine A
ax
ax
ax
ax
ax
1
4
5
2
3
Adjusted
Events
a
a
time