Innovation, Standards, and NOAA Data Management Ted Habermann NOAA National Data Centers

1
Innovation, Standards, and NOAA Data
ManagementTed HabermannNOAA National Data
Centers
There are special management challenges, and I
think that that's an area that we in agencies
such as NOAA, need to spend an extra amount of
time on. We have very talented workers and very
talented employees, many of whom have advanced
degrees, and they have been successful because of
certain behaviors in their field. As you
progress through the system in any organization,
you need to develop other skills Vice Adm.
Lautenbacher
Mature Organizations
2
The Technology S-Curve
We all know that new technologies emerge slowly,
grow quickly (if they catch on) and then fade
away. This common knowledge has been described as
the technology S-curve. Why does it exist?
TIME
3
The Adoption Curve
Luddites
Geoffrey Moore has attributed the S-curve to the
technology adoption life cycle where techies and
visionaries are early adopters, pragmatists make
up the bulk of users, and luddites fill out the
tail of the distribution.
Pragmatists
Visionaries
4
The Chasm
Moore has also described the chasm in the
adoption life cycle. He proposes that many new
technologies do not make it across the chasm
between visionaries and pragmatists. They fall
into the chasm. The technology S-curve with the
chasm might look like
TIME
5
Technology Cycle
Technology Cycle
Technological Disruption
Selection
Era of Ferment
Dominant Design
Disruption 2 (destroys existing competence)
TIME
6
Types of Innovation - 1
Sustaining / Incremental Innovation generally
small innovations in products and processes aimed
at existing customers. Disruptive /
Discontinuous Innovation significant innovations
generally aimed at unknown or non-existent
customers.
7
Unidata Objectives (1998)
Sustaining Innovation
These objectives either respond to users'
current needs or advance Unidata toward meeting
future needs effectively. Most of the
"responsive" items are continuations of current
Unidata objectives, and their importance is well
established. But only by looking beyond present
needs to anticipate future ones, and by pursuing
the most promising technical advances, can
Unidata remain effective. This is true even
though some of these advances involve
uncertainties, and the demand for them may not be
apparent as yet. Unidata, 2003 Proposal.
Disruptive Innovation
Clayton Christensen, The Innovators Dilemma
8
Unidata (netCDF) Evolution
Disruptive Innovation Always includes a decrease
in metrics for current customers so it is
difficult for mature organizations.
In the Unidata case we are now seeing the
disruptive switch to Java play out. The
capabilities of the Java version of the netCDF
libraries have now surpassed the original C
version.
9
Types of Innovation - 2
Component Innovation Making existing components
better. Architectural Innovation putting
existing components together in new ways.
10
Architecture Organization
Structure in mature organizations tends to evolve
to match product architectures. Architectural
Innovation, therefore, many times includes
elements of organizational change.
11
Innovation Technology Cycle
Disruptive Innovation
Component, Architectural, Sustaining
and Process Innovation
Product Innovation Design Competition Community-dr
iven technology change
What do we make?
How do we make it (better)?
TIME
12
Other Differences
Research Prototypes Custom developments Network
building Uncertainty
Operational Systems Product Families Predictabilit
y Partnerships
Standards
Network Effects Value f(N2)
(non-compliance cost increases with time)
TIME
13
How Standards Change The Game

• Expanded Network Externalities (Network effect
  turns on)
• Reduced Uncertainty and Risk in Technology
  Decisions
• Reduced Consumer Lock-In to Particular
  Components
• Competition in the Market vs. Competition for
  the Market
• Competition on Value vs. Features
• Competition to Offer Proprietary Extensions
• Component vs. Systems Competition

Standards shift the locus of competition from
systems development to component development.
Specialists tend to thrive in the mix-and-match
environment created by interface standards.
Generalists and system (stovepipe) developers
tend to thrive in the absence of standards.
In the absence of standards 1) there is no
architectural innovation (no mix-and-match)
and 2) the organization can not benefit from
component innovation. Once a standard has been
agreed on (selection), the organization benefits
from component innovation and architectural
innovation.
14
Andy Grove Communication Overcomes Computing
The framework is changing now. The Internet is
redefining software. The Internet is redefining
the role of computing and communication and their
interaction with each other. I still dont
understand the framework. I dont think any of us
really do. But some aspects of it are pretty
clear. Its proven not to be computing based but
communications based. In it computing is going to
be subordinated to the communications task.
Decisions Dont Wait, Harvard Management Update.
15
Kevin Kelley The Web and Sharing
The revolution launched by Netscapes IPO was
only marginally about hypertext and human
knowledge. At its heart was a new kind of
participation that has since developed into an
emerging culture based on sharing.
We Are The Web, Wired, August 2005
16
Technology Confluence
Geographic Information Systems
Relational Databases
DESKTOP
17
Infrastructural Technologies
IT is, first of all, a transport mechanism it
carries digital information just as railroads
carry goods and power grids carry electricity.
And like any infrastructural technology, it is
far more valuable when shared than when used in
isolation. The history of IT in business has been
a history of increased interconnectivity and
interoperability, from mainframe time-sharing to
minicomputer-based local area networks to broader
Ethernet networks and on to the Internet. Each
stage in that progression has involved greater
standardization of the technology and, at least
recently, greater homogenization of its
functionality. For most business applications
today, the benefits of customization would be
overwhelmed by the costs of isolation. Nicholas
G. Carr, IT Doesnt Matter, Harvard Business
Review, May, 2003
18
Why No Standards?
The longer the market takes to determine a
standard, the more expensive it will be for firms
operating within that market. The more expensive
this competition becomes, the greater the
tendency for firms to cooperate at the beginning.
The difficulty with this reasoning is that it is
difficult for individual firms to determine how
expensive or how long it will take the market to
determine the dominant standard. Nor are
companies willing to cede control of such an
important aspect of their market early in a
competition. Booz Allen Hamilton, 2005.
The science community generally does not value
sharing.
19
Organizational Approaches / Skills
Leadership
Management Detailed Plans
w/ Accountability
Discovery-based Planning Resolving
Critical Unknowns
Long-term Plans Requirements PPBES
TIME
20
Multiple Technologies
NOAA relies on many technologies which are
changing rapidly. To do well in long-term, NOAA
must
1) recognize phases of the technology cycle,
2) develop and use mechanisms for standards
adoption and selection
3) realize that different phases require
different organizational structures /strategies,
4) support multiple structures / strategies
simultaneously.
TIME
21
Innovation, Standards NOAA
There is a considerable innovation literature
that can help NOAA learn the new skills required
to innovate strategically and effectively. Techno
logy is evolving from a computing tool to a
communication tool. It is becoming an
infrastructure technology. Standards are
critical to building value of infrastructure
technologies. Standards are critical to
organizationally effective component and
architectural innovation. NOAA must develop and
use processes for selecting and applying
standards. The requirements and approaches to
planning are very different in the different
phases of the technology cycle. Understanding
and explicitly recognizing the differences in
phases of the technology cycle and the
differences in balance between management and
leadership skills might help NOAA.
22
Examples from NGDC

• NOAA Maps, The NOAA Observing System Database and
  NOSA Website
• Internet mapping in NOAA is a great example of
  an Era of Ferment. Many different tools produce
  maps that look and behave differently. We are
  collecting capabilities for over 400 NOAA
  Internet Maps and comparing those capabilities to
  a single map that includes information for 100
  NOAA Observing Systems. How would a selection
  event change NOAAs approach to internet mapping?
• The Data Processing Pipeline
• An example of components/architecture for data
  processing and ingest in an open source Java
  project initiated at NGDC.

23
NOAA Maps
Over 400 maps have been captured from NOAA Web
pages and described in the NOAA Maps Project.
24
Maps at NGDC
25
NOSA Website _at_ NGDC
26
The Data Processing Pipeline
A pipeline executes a sequence of plug-able data
processing tasks. The NGDC data processing
pipeline provides a set of pipeline utilities
designed around work queues that run in parallel
to sequentially process data objects. The
pipeline is an open source project hosted in the
Jakarta Commons Sandbox (http//jakarta.apache.org
/commons/sandbox/index.html). Processing steps
are specified as a series of stages in an XML
configuration file.
27
DMSP Orbit Processing
Stage 1. Find Matching Files Stage 2. Avoid
Duplicate Processing Stage 3. Read Data / Create
Spatial Objects Stage 4. Low-Res Thinning Stage
5. High-Res Thinning Stage 6. Write Spatial Data
to Database
28
Building on Experience
Louis Liebenberg, The Art of Tracking The Origin
of Science (Wired, June 2003)
29
Leadership Model Positive Deviance
Positive deviance says that if you want to create
change, you must scale it down to the lowest
level of granularity and look for people within
the social system who are already manifesting the
desired future state. Take only the arrows that
are already pointing toward the way you want to
go, and ignore the others. Identify and
differentiate those people who are headed in the
right direction. Give them visibility and
resources. Bring them together. Aggregate them.
Barbara Waugh
30
Ted.Habermann_at_noaa.gov
31
References
Booz Allen Hamilton, Geospatial Interoperability
Return on Investment Study, 2005,
http//gio.gsfc.nasa.gov/docs/ROI20Study.pdf. Ch
ristensen, C., The Innovators Dilemma, Harvard
Business School Press, 1997, 225p. Clark and
Wheelwright, Revolutionizing Product Development,
The Free Press, New York, 1992,
364p. Govindarajan, V. and C. Trimble, Building
Breakthrough Businesses Within Established
Organizations, Harvard Business Review, May 2005,
p. 58-68. Lautenbacher, C., Business of
Government Radio Interview, http//www.businessofg
overnment.org/main/interviews/bios/conrad_lautenba
cher_frt.asp, 2005. Moore, G., Crossing the
Chasm, Marketing and Selling High-Tech Products
to Mainstream Customers, Harper Business, 1991,
211p. OReilly, C.A. and Tushman, M.L., The
Ambidextrous Organizations, Harvard Business
Review, April 2004. The Positive Deviance
Initiative, http//positivedeviance.org/ Pascale,
R.T. and J. Sternin, Your Companys Secret
Change Agents, Harvard Business Review, May 2005,
p. 72-81. Tushman, M.L., Anderson, P., and
OReilly, C.A., Technology Cycles, Innovation
Streams, and Ambidextrous Organizations
Organizational Renewal Through Innovation Streams
and Strategic Change, in Managing Strategic
Innovation and Change, Tushman and Anderson,
eds., Oxford University Press, New York, 1997,
657p.