From STORET to WQX New Jersey

1
From STORET to WQXNew Jerseys Solution for
Integrating Ambient Water Quality, Facility and
GIS Information from Federal, State, County
Volunteer Organizations

• Harry Chen
• New Jersey Department of Environmental Protection
• November 28, 2007
• Austin, TX

2
Project Purpose

• Integrate water quality data
• Streamline and simplify data transfer
• Provide better, more timely access to water
  quality data
• Aid decision-making
• Improve resource efficiency

3
Goals

• Develop an up to date, searchable inventory of
  ambient water quality monitoring in New Jersey
• Integrate ambient water quality monitoring data
  into NJDEPs enterprise environmental data
  management system to enable more integrated
  environmental decisions
• Make NJ ambient water quality monitoring data
  available to the public
• Report ambient water quality monitoring data to
  USEPA

4
Project Partners

• NJDEP
• Bureau of Freshwater and Biological Monitoring
• Bureau of Marine Water Monitoring
• Bureau of Water Quality Standards and Assessment
• Division of Watershed Management
• Division of Science, Research, and Technology
• USGS
• Department of Health and Senior Services (DHSS)
• County Governments
• Volunteer Monitoring Organizations
• NJ Water Monitoring Coordinating Council
  (Inventory)

5
ExistingAmbient Water Quality Data Producers
6
AMNET Field/Hab dB
Reformat Field/Habitat dB
Field Form
Existing NJDEP Data Entry Mechanisms
Text file
Keypunch Field Msr Habitat Sample Info
MS Access Link
Note Proofreading and correction steps occur for
all data, but are not shown due to space
limitations.
AMNET Taxa dB
Bench Sheet
Reformat Taxa dB
Keypunch Taxa counts
Text file
Text file
Make Final dB
Extensive Manual Reformat Of Text File
FIBI dB
Field Form
Keypunch Field Msr Habitat Taxa counts Sample Info
Text file
Change Result Status To Final
ODBC Connect
DHSS Chem Results
Import e-file Chem Results
Migrate
NJ STORET
FTP Oracle Export
Delete Migration
Lakes dB
Reformat Lakes dB
QC Labs Chem Results
Text file
Import e-file Chem Results
EPA
Text file
Field Staff
Data Staff
Field Form
Keypunch Field Msr Habitat Sample Info
ODBC Connect
Corrections Text file
DHSS Chem Results
Keypunch Chem Results
Proofreading dB
Others
Text file
Others
QC Labs Chem Results
Keypunch Chem Results
Other Projects dBs
Corrections
Proofreading Reports
Manual Proofreading
Field Form
Sample Tracking dB
Keypunch Field Msr Sample Info
7
Water Quality Data Management Model
8
Core project components

• WQX data flow, consisting of the following
• Data-in E2 WQX module with data validation
• Data transmission to and from COMPASS
• Data-out NJ Water Monitoring Inventory / Data
  Portal
• Outbound data transmission to EPA CDX

9
High-Level Data Flow
EPA CDX
E2
COMPASS
ENDS
10
Data System Integration

• E2
• COMPASS
• ENDS
• Data will be transmitted through each component
  using NJs Exchange Network NODE.

11
What is E2?

• The Electronic Environmental Reporting system
  (E2) is a secured XML/Internet based system.

12
What is COMPASS?

• Comprehensive Organizational Monitoring Program
  and Assessment Support System (COMPASS) is an
  enterprise-wide methodology and information
  management solution.

13
What is ENDS?

• Exchange Network Data Services are web-services
  based applications used to exchange data between
  EN-compliant Nodes.

14
FutureAmbient Water Quality Data Producers
EPA USGS Web Services
4
15
Data Flow for Submission of All Publicly
Available DataFrom Multiple OrganizationsEx.
305(b) and 303(d) Assessments
Sampling Data
16
Data Flow When An Organization UsesMultiple
Groups to Generate Results(Ex. Field
Measurements by One Group and Lab Analysis by a
Second Group)
17
Data Flow From a Single Organization(Ex. Field
and Lab Work Done by Same Group)
18
Data Submission Options
19
CIMP CSV-file Import Mapping Procedures
20
Data Field (column) Mapping
21
Data Normalization
22
Mapping to Domain values
23
Save Mapping configurations
24
Water Quality Data Transmission to CDX
25
GIS Integration
Monitoring Inventory
Data Reports
Link
Web-based Maps
Link
Link
26
GIS Proposals

• EN Browser hosted by ECOS, a national solution
• NJ-GeoWeb, a NJ solution

27
EN Browser

• Web services data discovery tool
• Allows users to discover and execute Data
  Requests on Exchange Network Nodes
• Offers users the ability to remove (filter)
  unwanted sections of the XML data set returned
  from a Data Service
• Users have the option of choosing the format that
  they would like to retrieve the data in
• Can integrate with enterprise data reporting tools

28
NJ-GeoWeb

• A New Vision for Seeking Spatial Intelligence.
  Integrating customized GIS and analysis tools
  with enterprise databases for improved
  decision-making on the Internet.

29
What is NJ-GeoWeb

• DEPs next generation i-MapNJ on the Interent.
  Replaces ArcIntelligence on the intranet.
• Uses profiles (tabs) that default data layers
  and reports (ie. Child Care, Whats in My
  Neighborhood, Smart Growth, Business Viewer)
• Allows Internet users to view, interact with,
  analyze and submit points, lines and polygons
• Links to departmental databases (DataMiner,
  eNJEMS, COMPASS..).
• Uses enterprise data from all Agencies (and
  serves from the Web)
• Links to external services on the web.
• Working with NJ OIT make NJ-GeoWeb a state
  agency resource.

30
NJ-GeoWeb
Permitting Compliance Enforcement
Clean Plentiful Water
Water/Sewer Infrastructure Water Allocation Storm
Water Runoff Monitoring
Whats in My Backyard
Surveillance Inspections ePermits
Regulated Sites Incident Mgmt Flood
Mapping Endangered Species

Data Server
Sustainable Growth
SDE
Data
Taking care of Business
Align Economic and Env. Goals Env. Sensitive
Areas Green Technologies
Economic Incentives Employment Info Revitalize
and Redevelop
Live Feeds from State Government
L.awrence Thornton, NJDEP
31
WQDE Project Challenges

• Pulling data from USGS (web services in
  development)
• Pulling data from EPA (web services in
  development)
• Coordinating with a large number of project
  partners
• Requires compromise
• Re-engineering current reporting procedures,
  while designing new WQX reporting procedures
• How to teach an old dog new tricks?

32
Status of NJDEP Data Exchanges

• Water Quality Data Exchange
• Completed Business Process Analysis with 11
  project partners
• Identified Needs and Analyzed Options for
  implementing data exchange
• Completed Requirement Specification
• Design currently in progress
• Volunteer Monitoring
• Application in production
• Currently designing reports

33
Project Phases and Tasks

• Business Process Analysis
• Functional Requirements Specification and System
  Design
• System Development, Data Preparation and
  Implementation
• Acceptance Testing
• User Training, Documentation and Outreach

34
Next Steps

• Complete System Design
• Programming and Development
• User Acceptance Testing
• Documentation and Training
• Production Deployment
• Send data to EPA
• Target Date October 2008

35
Thanks!
Contact Info NJDEP- Office of Info Resources
Management Name Harry Chen Phone
(609)-633-1261 E-mail Harry.Chen_at_dep.state.nj.us