User studies: contributing to developing regional decision support for climate services

1
User studies contributing to developing regional
decision support for climate services

• Andrea J. Ray, Roger S. Pulwarty and
• Robert S. Webb
• NOAA-CIRES Climate Diagnostics Center
• Climate Prediction Assessments Workshop
• Tallahassee, FL
• 9-11 March 2004

2

• The timely production and delivery of useful
  climate data, information, and knowledge to
  decision makers.
• National Research Council, 2001

3

• NRC recommends that a regional focus is needed to
  address societal needs, developed through
  regional decision support systems.
• Pillars of climate services include
• Regional decision support networks,
• Research,
• Observing systems, and
• Mechanisms to enable user-centric design and
  improvement.

4

• To be effective, regional decision support
  systems must not only provide data and forecasts,
  but also determine
• what problems and questions are important and
  critical, and why, to be determined by consensus
  between users and providers of products
• what information is needed to solve problems, on
  short and long-term time scales
• what transaction costs are involved in developing
  and maintaining long term partnerships and
  interactions.

5

• Many regional decision support, networks exist,
  
• NOAA weather forecast offices,data centers, river
  forecast centers,
• Regional climate centers,
• State climatologist offices,
• Agricultural extension services
• Some private companies
• These networks provide data and forecasts, but in
  order to meet the NRC user-centric goals of
  climate services, these organizations would also
• adapt products to meet specific user needs,
• support major user groups,
• encourage dialogue with users, and
• develop a regional focus to address societal
  needs.
• These networks generally are not set up to do
  effective user-studies

6
Effective user-centric design and implementation

• To achieve user-centric design and implementation
  these interactions must
• identify entry points for information within
  existing decisionmaking and the space for
  communicating this information
• identify research needed, including strategic,
  applied, and adaptive
• create nodes for users to engage with researchers
  and providers, to help determine what products
  are to be developed and tested
• Characterize needed products e.g., scale, units,
  formats, how accessible, when issued.
• Inform the development of alternative
  decisionmaking strategies as needed
• Mechanisms for studying users need to be built in
  to regional networks.

7
Effective user-centric design and implementation

• Sustained and systematic communication is
  required on three fronts
• between users and providers (NOAA and regional)
• between researchers and providers, and
• between researchers and users.
• Why user studies?
• More than consultancy
• More than a simple conversation or communication
  (not anecdotal)
• Understand the complexity of user context and
  differences in perspectives between users and
  scientists

8
Designing effective user studies

• User-study approach
• Based on case studies
• Based on
• need to get past differences in perspective
• the innovation-decision process
• definitions of use
• Dimensions to characterize about a user-case

9
Differences in perspective
Factor Scientists perspective Water Managers Perspective
Identifying a critical issue Based on a broad understanding of the nature of water management Based on experience of a particular system
Time frame Variable Immediate (operations) Long-term (infrastructure)
Spatial resolution Defined by data availability or funding Defined by institutional boundaries or authorities
Goals Prediction Explanation Understanding of natural system Optimization of multiple conditions and minimization of risk
Basis for Decisions Generalizing multiple facts and observations Use of scientific procedures and methods Availability of research funding Disciplinary perspective Tradition Procedure Professional judgment Training Economics Politics Job risks
Expectation Understanding Prediction Ongoing improvement (project is never actually complete) Statistical significance of results Innovations in methods/theory Accuracy of information Appropriate methodology Save money and time Protect the public Protect their jobs, agendas or institutions
Product Characteristics Complex Scientifically defensible As simple as possible without losing accuracy Importance of context
Frame Physical (atmospheric, hydrologic, etc.) conditions as drivers Dependent on scientific discipline Safety and well being Profit Consistency with institutional culture, policy, etc.
Nature of Use Conceptual Applied
10
Innovation-Decision process conceptual steps in
deciding whether to adopt an innovation

• Knowledge phase learns about the existence of an
  innovation, perhaps thru an outreach function
• Persuasion phase forms a favorable/unfavorable
  opinion
• Decision phase engages in activities leading to
  adopting or rejecting
• Implementation puts the innovation into use
• Confirmation phase decision to adopt is
  reinforced, or could be reversed if experience is
  adverse (Rogers,1995)
• ? Path towards adoption and use studies and
  mechanisms for developing user-centric products
  should take advantage of this

11
Definitions of use

• Consult the product is consulted, e.g., looked
  up on a web page or received as a briefing or
  from other source (type1)
• Consider after consulting the product, it is
  considered in management deliberations as a
  factor potentially influencing decisions, but not
  necessarily in operational models (type 2)
• - forecasts may be used in this way when they are
  not in appropriate forms for use in operational
  models
• Incorporate some form of the forecast is
  incorporated into an operational model that is
  utilized in operational decisions (type 3).
• Dialogue about risks communication of risk, i.e.
  the forecast is used to communicate with other
  managers and stakeholders about the risk of
  certain conditions and about the need to take
  actions, or to justify actions (type 4).

12
User Study Approach

• One mechanism for assessing user needs
• Dimensions to characterize about the
  user-context
• Identify critical problems which are sensitive to
  climate
• Identify decisionmakers and their key
  stakeholders
• Assess how climate variability interacts with
  their critical problems
• Decision calendar may help organize recurring
  decisions and show when products are needed
• Identify user groups who may be willing partners
  in testing and prototyping this new technology
• Potential users in a rapidly evolving phase of
  their critical problem may be more open to
  interacting as partners
• Social changes, legal or policy change, or a
  climate event such as drought or flood

13
Case study Multi-purpose Reservoir Management in
the Upper Colorado River Basin
User group is the reservoir managers in the
Colorado headwaters and Gunnison basins who
operate reservoirs originally built to provide
reliable irrigation supply, hydropower, and
recreation
Redlands Diversion dam and fish ladder, Gunnison
river near Grand Junction, CO
14
Dimensions for Aspinall Unit Case

• Critical problem sensitive to climate
• Evolving reservoir management in which demands
  now are closely balanced with supply but new uses
  are being legally proscribed
• More intensely managed water system is more
  sensitive to shortage because there is less
  buffer in the system
• Decisionmakers and their key stakeholders
• USBR has authority to manage, but USFWS, NPS,
  have interacting legal authorities, other stakeho
• How climate variability interacts w/ the critical
  problem
• Decision calendar for annual operating plan
  helped organize recurring decisions and ID
  potential climate
• User groups who may be willing partners/adopters
• Deadlines to write ESA recovery plans and formal
  filing for a federal reserve water right for a
  national park are requiring USBR to accommodate
  these in their operations
• 2002 drought as a climate event (not forecasted)
  also resulted in finding flexibility and new ways
  of operating

15
Potential Uses in the Aspinall case

• Identified a number of potential uses of seasonal
  climate forecasts in their operational models, as
  well as shorter term forecasts
• However, seasonal forecasts are currently not
  available in a form that can be incorporated into
  their operational models. Current use is to
  consult these products and consider them in
  management deliberations
• Aspinall decisionmakers use climate information
  (e.g. during drought) in a dialogue with their
  stakeholders about the risk of certain conditions
  and about the need to take actions, or to justify
  actions.
• Projects with Martyn Clark (WWA) and Dave Brandon
  (CBRFC) are working to translate both MRF
  forecasts and CPC seasonal forecasts

16
Conclusions I

• Regional decision support networks exist, but are
  not designed to carry on the systematic study of
  and interaction with users to ensure the
  development of user-centric products and services
• Mechanisms need to be built into the regional
  decision support networks
• These mechanisms could ensure user-centric design
  and implementation by
• identifying entry points for information within
  existing decisionmaking and the space for
  communicating this information, e.g., through
  basin water management groups, water availability
  task forces, emergency management meetings
• identifying research needed, including strategic,
  applied, and adaptive
• enabling RDS networks to function as nodes for
  users, researchers and providers to engage in
  dialogue and to help determine what products are
  to be developed and tested
• effectively characterizing needed products and
  synthesize user needs from among many
  interactions (beyond consultancy)

17
Conclusions II

• User studies are a means to create and sustain
  the communication and learning required for
  user-centric design and development and also a
  cost.
• create a dialogue about climate risks and not
  just provide information
• create a dialogue about the types of climate
  products desired by a broad spectrum of users by
  integrating knowledge about what climate products
  are needed.
• NOAAs mechanisms for developing user-centric
  products and services need to
• Recognize that differences in perspective exist
  and limit scientists ability to create
  user-centric products unless we deliberately
  work to see other perspectives
• Use of products is broader than just in
  operational models recognize that there is a
  range of uses and create products for all types.
  Evaluation of use should consider all types.
• Adoption is a process develop mechanisms to draw
  potential users along this path and evaluate
  whether activities support evolution of use
• Training needed across the RDSS networks in how
  to conduct user-assessments

18
(No Transcript)
19
EXTRAS
20
Reservoir Management Decision Calendar
Water Year Planning Next Water
Year Planning
Provide for late Summer/early Fall irrigation
while maintaining target flows
Next water year runoff unknown, reserve water
until February snowpack observations
Winter season precipitation forecast for Fall
release decisions
Winter releases based on January/February
snowpack observations
Winter/Spring forecast for Winter release
decisions
Peak Flow Augmentation - fill curve
Summer season forecast for Peak Augmentation
planning
Week 2 forecasts for Peak Augmentation
Peak Flow Augmentation releases
Figure 4-28
Plan releases for Summer irrigation hydropower
Week 2 forecasts for Summer irrigation
hydropower release decisions
Ray et al. (2000)
Provide for Summer irrigation hydropower needs
while maintaining target flows
21
Differences in perspective

• Identifying a critical issue
• Time frame
• Goals
• Basis for decisions
• Expectiation
• Product characteristics
• Frame

22
Characteristics of effective user studies

• Dimensions to characterize about a user-case

23
Understanding Barriers New Services
How to they understand climate? What do they use
now? Can we produce forecasts in a manner
related to how they think about climate?
Desired information not available
Are the problems critical to the user influenced
by climate? If so, are there products available
to address these?
Are there circumstances that minimize
constraints?
User studies seek answers to the right hand
questions