Connecting the Silos

1
Connecting the Silos

• Strategies and Technologies for Sharing and
  Integrating Geospatial Information

2
Overview

• California Resources Agencys approach to sharing
  environmental data and information
• Strategies and technologies that address our
  needs
• Plans for the future

3
In the beginning
California Environmental Resources Evaluation
System
4
http//www.ceres.ca.gov/

• A program in the California Resources Agency
• Started in 1995 when there was very little use
  of the Internet by state agencies
• To address loss of Agency data/information
• Resources Agency conservancy web sites
• California Environmental Information Catalog
• California Spatial Information Library
• Interagency GIS coordination

Working with public and private partners as part
of the California Environmental Digital Library
Network (www.caledln.org)
5
- California Environmental Information
Clearinghouse Architecture - Web Services for
Discovery, Integration and Sharing of
Environmental Information
Environmental Guest Web Sites, Imagery Project
Registry
Client Web Sites
CEIC API
California Environmental Information Clearinghouse
Atlas
State Library Archives
Framework Geospatial Data Library
Web Map Services
Library for gray literature (e.g., plans,
studies, reports, papers, findings, surveys, etc.)
File Exchange
Geospatial One Stop
Discovery Tools
Most content remains at user sites catalog just
contains pointers
Producers and Consumers of Environmental Data and
Information
Studies, articles, books, reports, findings,
surveys, databases, etc.
Web Map Services
User Catalogs
Items in GRAY are in development
jpe 1/12/07
6
http//gis.ca.gov/catalog/
The CEIC Catalog is

• Standards based metadata catalog (FGDC Dublin
  Core)
• Spatially enabled via bounding box and spatial
  gazetteer
• Use of keyword thesaurus for more effective
  discovery
• 228 catalogs containing 11,116 records (5,298
  from NRPI) from 201 organizations
• Easy entry
• Online form
• Automated harvest
• Content published to Geodata.Gov portal

7
http//gis.ca.gov/

• Started in 2001
• Goal to Democratize geospatial holdings
• Broaden use of GIS data
• Seek stewardship (maintenance) of data
• Make data easy to find and understand
  appropriate use
• Expand GIS services
• Encourage partnerships for data development and
  enhancement

8
Accomplishments

• More than 75 Terabytes Served Since 2002
• High water mark to date was more than 225
  GB/day in November 2006
• Majority (85) of recent demand is for NAIP CCMs
• Im sure well soon receive a call from the UC
  presidents office about bandwidth congestion. .
  .
• More than 20 TB delivered via sneaker-net
• Downloads and sneaker-net deliveries will likely
  explode with the distribution of the 2005 DOQQs
  (NAIP Imagery).

9
Accomplishments

• Nearly 160,000 files comprising more than 4
  terabytes of geospatial data online
• Served via HTTP and FTP
• Now offered in an easy to navigate thematic
  folder structure
• CaSIL Mirror at UC Davis
• Partial Mirror at CalEPA
• Working on finding other partners (e.g. San Diego
  Super Computer Center?)

10
Spatially enabled tools for information discovery
and navigation
Watershed Browser
GeoFinder
http//cwp.resources.ca.gov/browser/
http//casil.ucdavis.edu/cgi-bin/gb/geofinder
11
Working on a new search interface using new
technologies
As you zoom out
More data is found
A simple interface that can make use of 3rd party
web services (e.g., Google Maps)
12
http//atlas.resources.ca.gov/atlas/
ArcIMS client from California Department of Fish
and Game
Data from many agencies
13
Its time to move on. Data information need to
be

• Made accessiblenot just discoverable
• Optimized and leveraged (customized to support
  local processes)
• Stewarded (updated of good quality)
• Completed (comprehensive for the time, region and
  subject of interest)
• Standardized (made consistent for combining
  across regions and the state)
• Liberated (effectively licensed, not locked away
  in proprietary systems)
• Prioritized (what do we need first/most)
• Funded (for the long term, not just snapshots)

14
The Governor wanted to blow up the boxes but
how many boxes should there be?
Appropriate technologies wisely applied can help
us transcend institutional barriers
www
Great institutional political resistance to
taking apart and rebuilding the boxes
15
The Most Challenging Barriers are Organizational
not Technical

• Organizational behavior makes data sharing
  difficult
• Obstreperous
• Cantankerous
• Uncooperative
• Internally focused
• Competitive
• Special purpose and limited funding
• Best practices seen as unfunded mandate

16
What is Service Oriented Architecture (SOA)?
SOA Platform
Event Module
Report Module
Routing Module
Geocoding Module
Map Module
Application A
Application B
A software design that uses programs and data
distributed over a network.
17
SOA plumbing can be used to connect
organizational silos
Environmental Hazards Portal
Internet
www.hazards.ca.gov
SOA also applies to sharing and integrating data
18
3/16/07
Bond Projects Accountability Database
CERES Area of Responsibility
DPR Area of Responsibility
Prop 84 1E Web Site
Query Report
Project Seeker
Map Display Client
Full Project Report
Bond project data entered directly into DPR
system by some agencies
Data Entry Forms
Thick Map Client
Web Services
Map Editor Client
Full Project Report
XML Exchange Services
Address Matching Services
All tabular attributes stored here
All spatial attributes stored here
Project ID used to link data
DPR Web Enabled Tabular Database
CERES Web Enabled Geo-Warehouse
Internal Bonds databases from other agencies
Extract, Transform, Load Error Check
Geospatial Data Error Check
Exception Reports
Map Validation
Map Editor Client
Validation Requests
Data Contacts
19
We need a better way to share and integrate
geospatial data
Traditionally, Internet GIS applications take
copies of data from others and house them at a
single location. For example, the California
Digital Conservation Atlas
Typically done as one off applications that can
only offer snap shots in time as copies of data
quickly become out of date.
20
Organizations can publish and integrate their
geospatial data services
Rather than making copies of data, using SOA
organizations can continue to maintain their own
data bases while offering access to these as web
services that can be used by any number of
desktop GIS clients and Internet map viewers.
Integrated Map
Base Map
Fire History
Stressors
Public Lands
21
Technologies for mobilizing geospatial data from
across the Internet
22
INSIGHT makes it safe to share GIS data over the
Internet
ArcGIS, Geomedia, MapInfo Pro, open source, etc.
DMZ
Your Organization
User
Internet
ArcGIS, Geomedia, MapInfo Pro, open source, etc.
Web Server
Spatial Server
Can be configured so that users do not directly
interact with data service providers
Internal process for building and maintaining GIS
data
Map Client
Registry
CERES
23
Organizations remain in firm control, deciding
managing

• Appropriate use
• Intended use of the data
• Licensing and Intellectual Property Rights
• Control who can access the data
• Metadata
• Information on all of the above to accompany each
  data set
• Data maintenance
• Keeping the data up-to-date and accurate
• Service levels
• Level of availability that a user/subscriber can
  expect

Give a copy of your data away and there is no
telling where it will end up or how it will be
used. It is better to control it at the source.
24
What does the future hold?
25
Marshaling Local Data
Federal
The National Map
Extract, Transform, Load
State
Emergency Response, Resource Conservation, Health
Services, Transportation, etc.
1
n
2

,
Extract, Transform, Load
Services Funds
Regional
1
2
n

,
Extract, Transform, Load
Local
1
2
n
1
2
n

,

,
1
2
n
Parcels, Roads, Critical Infrastructure, etc.

,
26
California GeoCommons

• Members should be able to
• See all layers including other proposed changes
• Select objects from Master to change
• Import shapefile into Edit layer
• Import GPS coordinates into Edit layer
• Do heads up digitizing in Edit layer
• Edit copy of attributes

Suggested changes communicated back to data
owners and they decide whether to accept or
reject. Changes made to source databases and
propagated back to GeoCommons.
Information on action taken on suggested edits
communicated back to the GeoCommons.
Edit Layer (sandbox)
Org 1
Objects copied into Edit layer to change
Org 2
Data Extraction, Standardization and Loading
Master (read only)
Provide option to expose any layer as a web map
service
. . .
Once acted on, suggested changes are removed from
Edit layer and moved into the Journal layer
Journal (change history, read only)
Org n
Base Map Layers (read only)
DATA COLLBORATION
GeoCommons
Consideration may be given to enabling the
borrowing of line work from base layers for
work in the Edit layer
7-25-06
27
California GeoCommons

• A collective view is expressed in the GeoCommons
  Master layer.
• Source systems continue to maintain their own
  unique structures and designs.
• A common view or core record standard must be
  negotiated or determined prior to starting.
• Accounts within the GeoCommons are created and
  administered by collaboration leads.

28
Questions?
John Ellison Agency Technology Officer CERES
Director john.ellison_at_resources.ca.gov (916)
653-2238