COST 285 Short term scientific mission

1
COST 285 Short term scientific mission

• Francesco Potortì
• Matev Pustiek
• Dragan Savic

2
Objective

• To design and implement software that can
• import simulation or measurement results in
  form of trace files into a common data
  structure
• starting with ns2 and tcpdump
• launch post-processing (i.e. filtering and
  calculations) applications on the imported data,
  then
• complement the raw data with the results of
  post-processing,
• store the results of post-processing separately,
• feed them to other applications (e.g. for
  presentation).
• export data into various output formats (ascii,
  xml, )

3
Design issues

• Modular approach required
• Only a part of possible functionality implemented
  at the beginning
• HDF5 is selected as base for the common data
  format
• powerful, complete, efficient, well supported,
  widely used
• libraries available in C, C, Fortran, Python
• we plan to use the Python libraries (PyTables)
• A Python application for the core
• calls the different modules
• flexible structure does not need to know about
  the modules in advance, but examines and loads
  them dynamically
• the interface between the modules and the
  application is done through XML descriptions
• command line interface available
• the interface between the interface and the GUI
  renderer is done through HTML or XML

4
Design issues (2)

• Start by support importing from ns2 and tcpdump
  traces
• both are brought to a common format, which allows
• using common post processing tools
• combining simulation and measurement data
• supporting network emulation done by ns2 and
  nsclick
• Post-processing can be done with (any) HDF5
  compliant application
• Matlab and Octave functions
• Python based functions
• User interface
• command line
• web browser-based interface (both graphical and
  text-based, both local and remote)
• standalone renderer for XML
• PHP-GTK
• Flash or SVG player
• ad hoc Java application

5
Software architecture

• Building blocks
• Functional modules (.py, .m)
• import
• export
• process
• CoreGlue (.py)
• Reads module descriptions
• Executes modules, passes parameters, reads module
  execution results
• Handles HDF5
• Provides a CLI
• Provides a HTML interface
• Generates XML for additional UI
• UI
• GUI based (standalone)
• Tkinter, wxPython
• GTKPHP
• SVG, Flash
• Java
• HTML based thin HTTP server

php-gtk svg, flash
GUI1
GUI2
xml
html
CLI
CoreGlue
xml
6
Module description

• Module (self)description
• Check for possible standards or examples
• Create DTD for module description
• Structure (option)
• name
• description
• author
• group
• execution environment
• Python, Octave
• parameters
• name
• type
• default value
• data range
• optional parameters
• name
• type
• default value

7
HDF5 related issues

• Possible arrangements
• separate tables populating a tree structure
• multidimensional structure
• there is a problem for HDF5 to store efficiently
  sparse matrices
• check efficiency of adding a new dimension
• Does HDF5 support symbolic indexes
• If not need to map index to symbol (e.g. VOQ -
  1), maybe using attributes
• What to do with post-processed data
• store them together with raw data
• store them in separate HDF5 tables
• pass them to an external application
• Does HDF5 have a standard time format
• Conversion of time formats in various traces
  (ns2, tcpdump)

8
Tasks - Core

• Defining software architecture
• Core logic
• Database structure
• UI (GUI, command line)
• Design details of the CoreGlue
• overview of available XML libraries and features
• conversion from XML to Python data structures
• conversion from XML to HTML for rendering
• labeling of data in HDF5
• Writing the XML DTD for the module description
• using an already available DTD for command
  options
• starting from the gengetopt structure
• Consider architecture changes for supporting
  independent replications
• need to repeat simulation runs till some criteria
  is met
• the raw data is not precious, as you can
  regenerate it again

9
Tasks - modules

• Creating IP packet identifiers from tcpdump
  traces
• Algorithm definition and testing
• Creating ns2 import module
• format auto-detect
• Creating tcpdump import module
• Create IP packet identifiers
• Manage time-stamp formats auto-detect
• Packet delay and jitter calculation module
• .m scripts (Matlab and Octave)
• Python and shell
• Packet loss calculation module
• .m scripts
• Python and shell

10
Resources (1)

• HDF5 Home Page
  http//hdf.ncsa.uiuc.edu/HDF5/
• The nsclick simulation environment
  http//systems.cs.colorado.edu/Networking/nsclick
  /
• NS2 network emulation
  http//www.isi.edu/nsnam/ns/ns-emulation.html
• RFC 791 - IP protocol
  http//www.ietf.org/rfc/rfc0791.txt
• Flash and SVG http//www.eprg.org/projects/SVG/f
  lash2svg/
• Free software and research
  http//fly.isti.cnr.it/sl/
• Gengetopt - program for parsing input parameters
  http//www.gnu.org/software/gengetopt/gengetopt.h
  tml
• Creating GUI applications
  http//gtk.php.net/

11
Resources (2)

• Convert Flash to SVG http//www.eprg.org/project
  s/SVG/flash2svg/
• Convert SVG to Flash http//www.eskimo.c
  om/robla/svg2swf/
• wxPython
  http//www.wxpython.org/
• Introduction to Tkinter http//www.pythonware.co
  m/library/tkinter/introduction/
• Extensible Markup Language (XML)
  http//www.w3.org/XML/

12
Recent achievements

• Detailed software architecture was specified
• A prototype of Python application
• Components
• CoreGlue
• Functional modules
• User interface
• Simulation dataresults
• Raw ns2 and tcpdump traces
• Statistical results derived from raw data
• e.g. packet-loss probability, delay, jitter,
• Workshop paper proposal CoreGlue

13
Detailed software architecture
14
Open questions further steps

• HDF5 database structure
• data arrangement tables, sparse matrices,
• keep raw data
• what meta-data is needed
• Development and building Python modules for
• accessing, filtering and processing of the raw
  simulation data
• e.g. common time formats, identification of IP
  traffic flows,
• for export of data from HDF5 files to other
  formats
• preprocessing of ns2 and tcptrace traces to a
  common format.
• Design of an HTML graphical user interface.