Data Operations Division

1
Climate Database Modernization Program
Program Manager- Tom Ross
2
Climate Database Modernization Program

• The Climate Database Modernization Program
  (CDMP) goal is to preserve and make major climate
  and environmental data available via the World
  Wide Web
• Supports tasks to make historical data available
  from all five NOAA line offices and NESDIS data
  centers

3
Climate Data Base ModernizationOutcomes
Benefits to the Nation

• Outcomes
• Improved access and digital conversion of data
  stored on aging paper, film obsolete digital
  media
• Provides partnerships and jobs through private
  sector companies in 4 states
• Benefits to the Nation
• Human Dimension Reduced loss of life impacts
  due to environmental events
• Business Dimension Enhanced opportunities and
  socio-economic gains by considering climate
  information in decisions
• Policy Dimension More effective government
  policy by appropriately using climate information
  in decisions

Ninth year of the program (began 2000)
4
Climate Database Modernization Program

• NOAAs CDMP partners with four private sector
  contractors and has placed online over 52 million
  weather and environmental images
• CDMP supports over 58 climate and environmental
  data modernization projects
• Most of this data and information was
  underutilized and in some cases in danger of
  being lost
• Keying and Imaging the data increases data
  accessibility and data integration

5
Major CDMP Data Recovery Tasks
NMFS Imaging and keying data on cetaceans ,
fish eggs, coral reefs, lightship records, and
magnetic tape recovery.
NESDIS The following data were either scanned
or keyed or both by CDMP contractors surface,
marine, upper air, solar, ionospheric data, DMSP
film , arctic sea ice charts, and glacier
photographs
OAR image and key historic European ship
logbooks, hurricane reconnaissance information,
and various publications
NWS Keying African Upper air Observations, and
surface observations from Uruguay and Chile
NOS Image nautical charts historical coast
pilots, vectorize and geo reference shoreline
charts, and image and key water level gauge
records
6
CDMP Proposal Process has NOAA Working Together
on Data Rescue and Recovery

• The proposals judged against the following
  criteria
• 1. Supports NOAAs Strategic Goals
• 2. Contribution to improved data access rescue
• 3. Value to climate community
• 4 General merit to overall program
• 5. Cost effectiveness
• 6. Ease of digitization by the contractors
• Call for papers- Issued Each July/August
• Data Access Workshop Held November each Year
• 53 Proposals were submitted for 2007 support-but
  delayed until FY 2008

7
Climate Data Base Modernization 2000-2007
Over 7 terabytes of climate and environmental
data now only a mouse click away

• 52 million weather and environmental images
  online
• Hundreds of million of records digitized and
  online
• The Nations environmental record properly
  preserved and saved from being lost forever

Amount of records available online
(in millions)
8
Climate Database Modernization Program
9
Climate Database Modernization Program
One of the earliest forms with detailed hourly
data
10
CDMP- NCDC Data Integration
GOAL CDMP funds integration projects include
SAO, Forts, and Climate Record Book. NOAAs
Regional Climate (NERCC,MRCC) centers and NWS
Offices play a vital role in QA/QC and data
integration for NCDC.
195 stations for the 1928-1948 period, and about
145 stations for the 1893-1928 period with over
40 million observations NERCC and MRCC are
working with NCDC on QA/QC of these data (FORTS,
SAOs and Climate Records Books) thru CDMP funded
contract at 210K. Keying costs for various
surface and upper air datasets run an additional
4120K per year under CDMP
Keyed stations integrated into NCDC CDO system by
the end of 2007 and merged into ISD. More paper
observations in the NCDC basement, including
Military and Foreign sites.
11
Individual State Data Rescue Projects
72,000 monthly cooperative observations are
available for AK on-line for the period
1893-current Another 80,000 records for the
1950-1980 period remain to be imaged in FY
2008 Missing period 1950-1980 will be available
later in 2008 as funding permits. Images will
then be available on-line
12
Climate Database Modernization ProgramPartnership
with Colorado State Climatologist Office
13
Data Received From Many Sources
Forecast Warning Analysis NWS Coop Observers
Global Synoptic Reports NCEP Charts Ship,
Buoy Reports Rocketsonde Radiosonde Storm
Data Doppler Radar (GOES, POES, NPOESS, many
other) Satellites Aircraft Profiler ASOS
USCRN
14
Selected NOAA ProjectsClimate Database
Modernization Program
Monthly Weather Review (MWR) Article from 1909
used in snowfall research for Mt. Rose, NV
Article is on-line at NOAA central library
15
Selected NOAA Projects
Climate Database Modernization Program

• On the left is a photograph of Muir Glacier in
  Alaska from on August 13, 1941, by glaciologist
  William O. Field on the right, a photograph from
  the same vantage on August 31, 2004, by geologist
  Bruce F. Molnia of the United States Geological
  Survey (USGS).
• According to Molnia, between 1941 and 2004 the
  glacier retreated more than twelve kilometers
  (seven miles) and thinned by more than 800 meters
  (875 yards).
• Ocean water has filled the valley, replacing the
  ice of Muir Glacier the end of the glacier has
  retreated out of the field of view. The glaciers
  absence reveals scars where glacier ice once
  scraped high up against the hillside. In 2004,
  trees and shrubs grow thickly in the foreground,
  where in 1941 there was only bare rock.

16
Selected NOAA ProjectsClimate Database
Modernization Program

• Supports NOAAs Ecosystem and Climate goals
• Digitize 7,200 negatives of killer, minke whales
  and other mammals
• Scan 15,000 pages of notes
• Keypunch 3,000,000 characters
• May expand into capturing 750 hours of audio
  tapes collected over 30 years.
• Supports research into the decline of the
  southern killer whale population- classified as
  depleted under the MMPA and could become listed
  under the ESA

17
Selected NOAA ProjectsClimate Database
Modernization Program
Hurricane Celia 1962- Hurricane Wallet Project
18
Climate Database Modernization Program Selected
NOAA Projects
Northern Hemisphere Synoptic Map Project
A quote from the Monthly Weather Review of
February 1899 remarks on the Great Cold Wave,
The most striking of the latter perhaps was the
flow of ice down the Mississippi River on the
17th, past New Orleans and into the Gulf of
Mexico, an event never before witnessed within
the memory of man. Ice an inch thick formed at
the mouth of the Mississippi in East and Garden
Island bays, and the temperature fell to 10
degrees F. on the 13th. The loss of human life,
from January 29th to February 13th, by freezing
and avalanches (in Colorado) as near as can be
ascertained was 105 persons".
19

NOAA the
Uruguayan Meteorological Service enter
cooperative project on data rescue. The U.S.
National Oceanic and Atmospheric Administration
(NOAA) donated digital equipment used for the
project.

Images received from the Uruguayan Meteorological
Service and the Uruguayan Navy Include 42,000
images This includes 6 synoptic stations with
observations early as the 1908 and as recent as
early 2005. NCDC developed multiple keying
format and keying began in 2006 and ended in mid
2007 Total imaging/keying for this multi-year
project involves data from over 30 stations
totaling nearly 1,000,000 records
20
Camera and Imaging Setup In Uruguay
High tech (35 mm) cameras and camera stands are
set up in a well lit environment Technicians are
trained to image the records using the camera
equipment Technicians check for quality,
exposure and completeness during the imaging
process of each station Digital jpeg images are
stored in the cameras hard drive then
transferred to the computer Images on the
computer are then written to CD-ROM or DVD and
sent to NCDC for storage and keying
These same procedures are generally followed at
the other international locations. A technical
leader responsible at each international location
is critical
21
Uruguay and Chile Civilian and Military Projects

• Data keyed for six stations with variable periods
  of records- Earliest data was 1908 with 148
  station years keyed.
• In Chile, data taken at a monastery (Punta
  Arenas) for the period 12/1896-1954 was keyed

22
Mexican Surface Project
401,280 synoptic and daily records imaged by
camera by Meteorological Staff in Mexico
Period of record covers 1877-1981. Data is keyed
from the imaged records and added to NOAAs
database. Data useful in water supply and
drought
monitoring
activities in the
Americas
23
North American Drought Monitor

• The North America Drought Monitor (NA-DM) is a
  cooperative effort between drought experts in
  Canada, Mexico and the United States to monitor
  drought across the continent on an ongoing basis
• Costs and effects of drought often extend beyond
  international borders. Concerns over
  international water rights and agricultural
  productivity affect all three countries
• CDMP is keying historical Mexican climate data
  that currently exists as paper records to help
  extend and fill in gaps of the climate record
  back into the 1800s. This will also be useful in
  historical drought analysis and future
  predictions.

24
European Data Rescue Activities

• European Ship logs and documentation were
  imaged by camera at the British Museum. Roughly
  250,000 observations from the 1938-1947 were
  imaged and loaded to the NCDC WSSRD system (53
  GB).
• These images are accessible to researchers
  without charge with a user name and password.
  These data have been keyed- 31 million records
  digitized and are in process to be added to the
  ICOADS database.
• East India Company logbooks will be rescued in FY
  2008

25
African Upper Air Project

• Total of 196,500 images of pibal records received
  to date from Kenya, Malawi and Senegal
• Pibal data contain upper air wind observations
  for various stations from these countries from
  the 1960s to 2003
• These data are being keyed and after passing
  quality control checks will be entered in NOAAs
  global upper air database
• Digital files of these data were provided to the
  partner countries that imaged the data. Keyed
  data files were also were hyperlinked to the
  actual images providing an easy access to the
  original record

Kenya
Senegal
Malawi
26
Additional International Projects

• Imaging and keying international lightship
  logbook data from Finland and Sweden covering the
  period from the 1860s to the 1930s will be
  added to NOAAs global marine database

• Processing historical and current incoming
  Canadian marine data via NCDCs incoming records
  system. These marine data from the period are
  added NOAAs marine database

27
Additional International Projects

• Foreign Climate Data Imaging by NOAAs Central
  Library- Project goal of the project is to
  preserve and disseminate unique climatological
  data from historical sources.
• Period of record covers 1830s through the 1970s
  with most data from the period prior to 1960.
  Each series typically includes observations for a
  number of meteorological and other geophysical
  parameters.
• Involves documents from over 14 countries in
  Africa, 8 in Asia, 6 in Europe, 10 in North
  America, 1 Pacific Island and 8 in South America.