WDB%20Weather%20and%20Water%20Database

1
WDBWeather and Water Database

• 19th EGOWS Meeting,Ljubljana, June 9-12, 2008.

2
Outline

• Introduction
• Background of WDB
• Overview of WDB
• WDB Components
• Future of WDB

3
Who are we?

• Michael Akinde
• Ph.D., Database Architect, IT, Met.no
• Systems architect on ROAD system at
  SMHI(2003-2006)
• Vegard Bønes
• M.Sc., Senior Engineer, IT, Met.no
• Børge Moe
• M.Sc., Senior Engineer, IT, Met.no

4
Who are you?

• Survey please raise your hands...
• Who has coded programs?
• Who has written SQL queries?
• Who has worked with database APIs
  (JDBC,ODBC,etc)?
• Who knows what database normalization is?
• Who knows what a database index is?
• Who has written database extensions?

5
What is this about?

• We need to store weather/water data
• We need indexed (fast) access to time series
  retrieved from multiple files or sources
• We need access from a variety of applications
• We need products to be consistent
• We want to be able to track changes
• We want programs to be independent of the data
  format
• We have limited ressources

6
What is WDB?

• WDB is a storage system for meteorological,
  hydrological, and oceanographic data
• Developed by the PROFF program at met.no
• Based on the ROAD (SMHI) architecture

7
History of ROAD

• Operational meteorological database, developed in
  the late 90s at SMHI operational since 2001
• Responsible for 90 of SMHIs automatic
  production
• One of the worlds fastest meteorological
  databases
• Mature and well-established production system
  3rd generation in development 2006-2008

8
Problems of ROAD

• Like most met systems, ROAD is a legacy system
• Prior to ROAD3G, the loading chain of the ROAD
  system included Fortran code developed in the
  1970s
• SMHI is in the process of a major rehaul of the
  production systems in terms of migrating software
  from OpenVMS to Linux
• IBM Informix platform
• Geodetic datablade (geospatial extension)
• Licenses are expensive

9
WDB is

• Open source collaboration between met.no and SMHI
• Can be considered the 4th generation of the ROAD
  system
• Built from the ground up without legacy code
• Developed at met.no
• Released under the GNU GPL2 license
• Fast, Free, Flexible

10
WDB Open Source Alternative

• Should be robust enough to handle high volume,
  high availability production
• e.g., as the backend engine of yr.no
• Should be flexible
• e.g., to easily handle new types of data
• Should support quality and consistency of data
• Should be simple to use
• Should be easy to maintain and operate

11
WDB provides...

• Quick and easy to deploy data storage system
• Modular loading programs
• Easy to add more
• Data-independent application programmers
  interface (API)
• Usable, in theory, from any environment
• Fast performance on queries

12
WDB Overview
13
Installing WDB

• Download the tar package, and...
• Or request the debian package

serverwdb-x.x ./configure --prefix/home/myuser/
local serverwdb-x.x make serverwdb-x.x make
install
14
Putting WDB through its paces

• WDB implements all the standard GNU targets
• make check runs the WDB unit tests
• tests that can be run without being installed
• make installcheck runs the WDB install tests
• tests that require an installed database
• 300 tests and counting
• CPPunit and TAP protocol
• Fully automated test framework on CruiseControl

15
Enabling Technology

• Linux (Debian, Fedora Core 5)
• PostgreSQL version 8.1.x or later
• PostGIS version 1.1.x or later
• Proj.4 version 4.6.0
• ECMWF GRIB API
• GNU Autotools
• C open source libraries
• Boost Std Libraries, log4cpp, cppunit, etc.

16
WDB Overview
17
Loading forecast data

• Loading data into the database, is as simple as

servermydata gribLoad myforecast.grib
servermydata servermydata gribLoad
/opdata/hirlam10/.grib
servermydata
18
Loading Programs

• GRIBLoad is the design pattern for the WDB
  loading programs
• Loads one or more GRIB1 files into the database
• Uses ECMWF GRIB API, so theoretically, GRIB2
  works as well
• Stand-alone client program that connects to any
  WDB database
• Loaders for other grid formats are easily built
  in the same way by reusing code
• Loading of BUFR will be ported from ROAD

19
WDB Overview
20
WDB Call Interface (WCI)

• WCI is the data retrieval API for WDB
• Based on a function interface in SQL

Why not stick to simple SQL? - SQL isn't really
that simple - Requires knowledge of the
underlying data model (tables/views) - Lacks
useful functionality e.g., interpolation,
point-in-field, etc.
21
WCI versus pure SQL

• SQL results in tight coupling between application
  and database
• Table names
• Table structure (attributes)
• Really strange database queries

22
WDB Call Interface (WCI)

• WCI is the data retrieval API for WDB
• Based on a function interface in SQL

Why not a lib file? - Changes in a library
require that the applications using the library
are recompiled - New libraries need to be pushed
to the user (application) - Each operating
system, machine and language its own library
23
WCI versus compiled software library

• How many combinations do we need?

Python
C
Perl
C
Java
Fortran
Debian
Fedora
Ubuntu
Sun Solaris
Windows XP
Windows Vista
Don't forget 32 vs 64 bit architectures...
24
WDB Call Interface Features

• Stable Interface
• Performance
• 100-1000 values/s in single point retrieval (to
  be improved)
• 10 million floating points/s in field retrieval
  (about 100 fields/s)
• Functionality
• Pointwise, Timeseries, 2d-volumes, Quality
  information
• Coming Paths, Interpolation in space and time,
  3D volumes?
• Easy to program
• 1/2 page of code to retrieve MHO-data
• Platforms C/C, Java, Perl, Python, Fortran,
  ODBC
• Operating systems Pretty much anything...

25
WCI functions

• wci.begin
• wci.end
• wci.read
• wci.write
• wci.browse
• wci.info
• Programmer doesnt need any knowledge of the data
  format or the database model

26
The WCI Data Model
Time (Valid)
Parameter
Time (Ref)
Level
Place
Version
Data Value
Data Provider
Quality
Value ID
??
27
WDB Overview
28
WDB2TS

• TS2XML for WDB
• Web service to retrieve time series of data from
  a WDB database
• delivers data as csv or xml
• REST-like interface
• http//host/path?lat10lon10parameterspec
• (parameterspec data specification)

29
The Road ahead for WDB

• WDB version 0.6.0
• First production version of WDB
• Hindcast archive (12 TBs of grids)
• WDB version 0.7.4
• To be released next week
• We currently release a new version about once a
  month
• Implements a new loading program
• FELT met.no field format
• Approximate time to develop 2 weeks x 2 persons

30
WDB Roadmap

• Version 0.8.0 is scheduled for July 2008
• Official beta version of WDB
• Limited scale real-time production
• Version 0.9.0 is scheduled for October 2008
• Release candidate version (full-scale testing)
• Version 1.0 scheduled for December 2008
• Full-scale production version
• WDB is a part of the PROFF framework minimum
  expected life-time is at least15 years

31
Future thoughts?

• Complete integration with Diana
• More data types
• Fields (NetCDF, GRIB2, etc.)
• Observations (BUFR, XML)
• Better administration tools
• Verified Stability
• 24x7 hands-free operation (have this in ROAD)
• Improved Performance
• We have not started optimizations

32
Future thoughts?

• Scalability and fail-over
• Want more load? Add another server
• Want to guarantee up-time? Add more servers
• Consistency
• Easy on one server we want multiple servers
• Distributed data/replication is a well-known
  problem in the database community
• Problems huge data, geographically distant
  regions
• Tracking of changes
• Metadata can be used to track history

33
Recommended Reading

• WDB website (wdb.met.no)
• WDB User Manual
• WCI User Manual
• Man Pages for all WDB utilities
• Source code browser (SVN repository and doxygen)
• Developer documentation (design docs)
• Eclipse setup instructions
• Subscribe to WDB project on Freshmeat.net

34
Summary

• WDB is designed as a generic weather/water
  database system
• Perspective one core system that can be
  leveraged for many different uses
• Functionality that isn't implemented can probably
  be added
• For users fast access to data
• For developers a simple, generic interface to
  all kinds of different data
• For operators a system that is easy to maintain

35
Concluding thoughts

• Many, many more WDB systems in the future
• WDB is intended as the backbone of the production
  systems at met.no and SMHI
• Open source is a different model of collaboration
  which can be of great benefit to meteorological
  institutions
• We all require broadly the same functionality
• Good communications better systems
• Free as in Freedom

36
THANK YOU !Questions?E-mail
michael.akinde_at_met.no