Applied Sciences Program: Decision Support through Earth Science Results Step 1 Review Board

1
Building a User-Driven GEOSS Methods to
Capture, Analyze, and Prioritize User Needs GEO
Task US-09-01a and Earth Observations
Identification Lawrence Friedl, USA-NASA User
Interface Committee Member US-09-01a Task
Lead ISRSE-34 Symposium Sydney,
Australia 10.April .2011
2
Earth Observations NeedsOverview of Presentation
Elements
This presentation discusses a document-based
approach to identify observation needs, using
teams of experts to a) review analytic methods
to assess needs in documents, and b) concur on
priority-setting criteria to rank the
needs. Based on results of GEO Task US-09-01a
Identifying Critical Earth Observation
Priorities. I. Approach and
Meta-analysis II. Results III. Findings IV.
Advantages and Limitations Discussion
2
3
Earth Observations Needs
Section I. Approach and Meta-analysis
3
4
Earth Observations Needs Task US-09-01a
GEO Task US-09-01a Objective Establish and
conduct a process for identifying critical Earth
observation priorities common to many GEO
societal benefit areas, involving scientific and
technical experts, taking account of
socio-economic factors, and building on the
results of existing systems requirements
development processes. GEO Societal Benefit
Areas (SBA) Agriculture Disasters Health Biodiv
ersity Ecosystems Water Climate Energy Weather
4
5
Earth Observations Needs Task US-09-01a
GEO Task US-09-01a Objective Establish and
conduct a process for identifying critical Earth
observation priorities common to many GEO
societal benefit areas, involving scientific and
technical experts, taking account of
socio-economic factors, and building on the
results of existing systems requirements
development processes. GEO Societal Benefit
Areas (SBA) Agriculture Disasters Health Biodiv
ersity Ecosystems Water Climate Energy Weather
5
6
Earth Observations Needs Task US-09-01a

• US-09-01a Process Nine Steps
• The process lists the steps serially, yet some
  are done in parallel.
• Step 1 UIC Members identify Advisory Groups and
  Analysts for each SBA
• Step 2 Determine scope of topics for the
  current priority-setting activity
• Step 3 Identify existing documents regarding
  observation priorities for the SBA
• Step 4 Develop analytic methods and
  priority-setting criteria
• Step 5 Review and analyze documents for
  priority Earth observations needs
• Step 6 Combine the information develop a
  preliminary report on the priorities
• Step 7 Gather feedback on the preliminary
  report
• Step 8 Perform any additional analysis
• Step 9 Complete the final report on Earth
  observations for the SBA

6
7
Earth Observations Needs Task US-09-01a
General Approach Document-based analysis.
Examined observation needs expressed in
publicly-available documents from past 10 years.
Analysis included over 1700 documents. An
Advisory Group and an Analyst worked to
identify documents, analyze them, and prioritize
observations within each SBA. AGs involved 6-23
people from developed developing countries that
represent experts in an SBA. (Invited CoPs, IGOS,
others to be on AGs). 167 AG members 43
countries in total. Developed individual SBA
reports, which specified the observation
priorities for that SBA.
Based on the SBA reports, the Task Team conducted
a meta-analysis across the individual SBA reports
to determine observations priorities common to
many SBAs. The team combined and prioritized
parameters from the SBA lists and prepared an
over-arching report to identify Earth
observation priorities common to many SBAs.
The report includes findings, lessons learned,
and recommendations.
7
8
Earth Observations Needs Task US-09-01a
Cross-SBA Analytic Methodologies Method 1
Tally of All Priorities Frequency analysis is a
simple tally of the SBAs that require a given
observation. (Total of 146 observations were
included in this prioritization.)   Methods 23
Weighted Sums of All Priorities Weighted
frequency analysis is a weighted sum of the
number of SBAs that require a given observation,
taking into account the high/medium/low
importance assigned by SBA Analysts. Different
weighting schemes in the two methods (Same 146
parameters as Method 1.) Method 4 Top 15
Priorities by SBA This key parameters method is
based on each SBA Analyst preparing a list of the
top 15 for that SBA. (Total of 99 observations
were included.) Final Set Ensemble approach
across the methods. Calculated mean rank and
the range of ranks for all 146 parameters across
methods. Natural breaks at top 19 and top 36
parameters.
8
9
Earth Observations Needs
Section II. Results
9
10
Earth Observations Needs Task US-09-01a
SBA-specific Observation Priorities Each SBA
identified a set of observation priorities as
well as the 15 most-critical observations.
Overall, there were 146 observations identified
as priority needs from combining the SBA lists.
The Task Team used an ensemble of 4
prioritization methods to integrate and
prioritize for the Cross-SBA Analysis.
Agriculture SBA
Disasters SBA
Energy SBA
10
11
Earth Observations Needs Task US-09-01a
8 observations (5 of 146 total) are common to 6
or more SBAs 29 observations (20) are common to
4 or more SBAs 100 observations (68) are common
to 2 or more SBAs
11
12
GEO Task US-09-01a
Highest Ranked Observations (1-20) and
Associated SBAs All of the 20 Highest-ranked
observations are common to 4 or more SBAs All
of 30 Highest-ranked obs. are common to 3 or more
SBAs 3 Highest-Ranked Precipitation, Soil
Moisture, Surface Air Temperature
Note Some observations may be relevant to an
SBA even though not included in the SBAs set of
priorities.
Biodiversity SBA Team did not produce a set of
Earth observations priorities.
12
13
Earth Observations Needs
Section III. Findings
13
14
Earth Observations Needs Task US-09-01a

• Task Report Findings
• Precipitation Reigns the Cross-SBA Analysis
• Methods Showed Agreement at Highest-Rankings
• Priorities of a Single SBA May Not Be on the
  Cross-SBA List
• Tasks Approach Produced Users Needs in Users
  Terminology
• Articulation of Observation Needs in Documents
  Varied
• Regional Needs Incorporated but Not Featured
• Availability of Documents by Region Varied
• Insufficient Information across Documents on
  Parameter Characteristics

14
15
Earth Observations Needs Task US-09-01a

• Task Report Findings
• Priorities of a Single SBA May Not Be on the
  Cross-SBA List
• This approach, by design, focused on the
  commonality of priority observations to many
  SBAs. Thus, some observations of critical
  importance to a particular SBA do not appear in
  the Cross-SBA list of priority observations.
• Tasks Approach Produced Users Needs in Users
  Terminology
• Users didnt always use terms that the Earth obs.
  community may be familiar with. Many of the
  needed observation parameters were expressed as
  phenomena of interest rather than technical
  specifications of the parameter. Overall, the
  demand-side, user-based approach of the task
  produced a rich array of observations and
  revealed a need for follow-on user engagement to
  refine parameter characteristics.
• Insufficient Information across Documents on
  Parameter Characteristics
• Some of the documents included quantitative
  information on the required characteristics
  (e.g., accuracy, latency) of critical observation
  parameters, while other documents lacked such
  information. At times it was available. However,
  the required observation parameter
  characteristics vary widely according to the user
  and application.

15
16
Earth Observations Needs Task US-09-01a

• Task Report Recommendations
• Gather information and engage users on specific
  characteristics of the priority Earth
  observations, especially Precipitation.
• Conduct an assessment of the current and planned
  availability of the priority Earth observations.
• GEO and/or Regional Caucuses could consider
  pursuing similar assessments at regional levels.
• Consider additional analytic methods to gathering
  users needs and pursue an ensemble of
  approaches.
• Prescribe the prioritization methods, SBA
  sub-areas, and other aspects of the SBA analyses.
• Pursue broader incorporation of documents in many
  languages.
• Continue the use of ad hoc Advisory Groups, with
  refinements.
• Strongly consider a single organization to manage
  the individual SBA analyses.
• Articulate an SBAs community of users to support
  systematic collection of users needs.

Follow-on Activities
Process Improvements
16
17
Earth Observations Needs Task US-09-01a

• Task Report Recommendations
• GEO and/or Regional Caucuses could consider
  pursuing similar assessments at regional levels.
• User needs vary by geographic region, and user
  needs unique to a single region were not likely
  to appear among highly-ranked observations of the
  Cross-SBA analysis
• Pursue broader incorporation of documents in many
  languages.
• Documents describing user needs, especially
  regional and national needs, exist in languages
  other than English, yet such documents were not
  discovered or were underrepresented. The Task
  Team recommends that future endeavors plan and
  provide sufficient resources for the
  identification of documents in many languages and
  for necessary translations
• Articulate an SBAs community of users to support
  collection of users needs.
• Users and end-users are broad terms. Future Task
  Teams should develop or refine a set of User
  Types for each SBA. The Task Teams should ensure
  that the Analysts employ the User Types as
  guidance in collecting information and
  representing needs, utilizing them to conduct a
  gap analysis of their activities to address any
  bias or gaps.

17
18
Earth Observations Needs
Section IV. Advantages and Limitations
18
19
Earth Observations Needs Task US-09-01a

• Task Leads Comments
• Tasks efforts and results represent significant
  contributions and first steps within GEO to
  articulate Earth observation priorities. GEO has
  documented in a transparent way how Earth
  observation needs have been identified, involving
  numerous organizations and experts.
•  
• The value of the Tasks results and cross-SBA
  report is at least twofold
• Provide a baseline and entry point for further
  engagement with end users on their needs.
• Confirm any expected priority observations as
  backed up by an analysis of the literature.

19
20
Earth Observations Needs Task US-09-01a

• Lessons Learned
• Task Approach Achieved Desired Diversity in
  Prioritization Methods
• Variety in Analysts Approaches Introduced
  Complexities
• Approach to Sponsorship of Analysts Impacted
  Process
• Approach to Selection of SBA Sub-Areas Introduced
  Challenges
• Advisory Groups Played Valuable Yet Variable
  Roles
• Use of existing documents provides a level of
  objectivity
• The ranking of an observation in the Cross-SBA
  list does not imply objective importance of that
  observation as much as commonality in need.

20
21
Earth Observations Needs Task US-09-01a

• A Document-Based Approach
• Process Comments
• Analytic methods to extract observations and
  needs
• - Extraction and inference
• Prioritization methods and priority-setting
  criteria
• - Opportunity to tailor to SBA-specific issues
  (e.g., Daily Adjusted Life Years, World Energy
  Outlook)
• - For individual analyses, need to be internally
  consistent
• - For meta-analyses, need commonality and
  comparability across types (e.g., ordered list,
  tiers, grouped by some factor)
• Note The US-09-01a approach, by design and
  broadly-stated, was to be a natural experiment
  involving a variety of methods methods were tied
  to the
• respective SBA communities as represented by the
  AGs.

21
22
Earth Observations Needs Task US-09-01a

• A Document-Based Approach
• Assumptions
• Documents were produced in an unbiased way
• Documents express needs for existing and new
  observations

22
23
Earth Observations Needs Task US-09-01a

• A Document-Based Approach
• Advantages
• Independent and objective (based on what
  organizations have already stated publicly)
• Reflects collective needs of organizations across
  multiple people
• Shows documented, expressed needs
• Based on a democratic principle in which all
  were eligible to be considered
• Generate observation needs in the terms and
  lexicon of the respective user communities

23
24
Earth Observations Needs Task US-09-01a

• A Document-Based Approach
• Limitations
• Difficult to capture very-recent needs (the time
  aspect of the approach features long-term,
  sustained expressions of need)
• Advisory Group member engagement and potential
  bias
• Need for representative set of documents for all
  regions and SBAs
• Level of specificity and technical information on
  needs may vary across documents inference of
  needs introduces possible error

24
25
Earth Observations Needs Task US-09-01a
A Document-Based Approach
p. 51
25
26
Earth Observations Needs Task US-09-01a

• Recommendations
• Document-based analysis provides an objective
  basis to an identification of Earth observation
  needs
• should be part of a ensemble of analytic
  prioritization methods
• should be employed to identify users
  terminology for their needs
• A process involving a document-based analysis
  across themes should prescribe the
  analytic/prioritization method(s) and
  deliverables as well as allow for SBA-specific
  variants
• Allocate significant time, resources, etc. to
  include range of documents across regions,
  languages
• Articulate what a representative sample
  comprises
• should use and internal gap analysis to ensure
  appropriate breadth
• Allocate sufficient time for review of
  preliminary results by people outside of advisory
  groups this review may stimulate the
  identification of a host of new documents)

26
27
Earth Observations Needs Task US-09-01a
Contacts Acknowledgements Lawrence Friedl,
US-09-01a Task Lead Amy Jo SWANSON, Task
Coordinator Erica ZELL, Lead Analyst Adam
CARPENTER, Asst. Lead Analyst Special
recognition to the Analysts and Advisory Group
members for the respective SBAs.
Final Cross-SBA Report and Individual SBA Reports
are available at the US-09-01a Task Website
http//sbageotask.larc.nasa.gov/
27
28
Earth Observations Needs
Discussion
28
29
Earth Observations Needs
Back-up Materials
29
30
Earth Observations Needs Task US-09-01a
Advisory Groups for SBAs

• Advisory Groups ranged from 6-23 members.
• Sought broad international and regional
  representatives.
• Involved GEO CoPs, former IGOS Themes, GEO
  Countries and GEO Participating Organizations.

30
31
Earth Observations Needs Task US-09-01a
This chart presents the 30 highest-ranked Earth
observations, shown according to score in the
Cross-SBA analysis. The range in ranks is also
shown. Ranks are inverted so highest score
is 146.
31
32
Earth Observations Needs Task US-09-01a
General structure An Advisory Group and an
Analyst work together to identify documents,
analyze them, and prioritize observations within
each SBA.
Advisory Group (1 per SBA) Functions - Will
help to identify documents - Comment on analytic
methods and priority-setting criteria - Review
the analysts findings, priorities, and reports.
Involved 6-23 people from developed and
developing countries that represent experts in an
SBA. Involved Communities of Practice, former
IGOS Themes, GEO Countries and Participating
Organizations.
Analyst (1 per SBA) Functions - Will read and
analyze the documents - Develop an analytic
method and priority-setting criteria - Conduct
the meta-analysis to identify common priorities
within a SBA. The Analyst was the primary
coordinator and organizer of the activity to meet
the schedules and deadlines. Interacted with
and utilized the Advisory Group to vet
prioritization methods and review results and
reports.
32
33
Earth Observations Needs Task US-09-01a
Advisory Groups
33
34
Earth Observations Needs Task US-09-01a
A Document-Based Approach
34
35
Earth Observations Needs Task US-09-01a
Final Report Published October 2010. Delivered
to GEO Plenary VII as annex to UIC Report and
distributed at Plenary. Final Cross-SBA Report
and Individual SBA Reports are available at the
US-09-01a Task Website http//sbageotask.larc.n
asa.gov/
35