Robert M. Groves

1

INDICATOR SELECTION PROCESS Key National
Indicators Initiative Seventh National
Coordinating Committee Meeting November 14-15,
2005 The National Academies

• Robert M. Groves
• Chair, KNII Indicator Set Design Group

2
The Steps in Selecting Indicators

• February, 2005, Indicator Quality Workshop
• May, 2005, NCC Meeting
• Summer, 2005, launch of domain groups
• Fall, 2005, catalogue work of domain groups,
  presented as versions 0.9 today
• 2006 outreach reaction to question/indicator
  catalogue
• Ongoing priority-setting for indicators

3
Important Conclusions from the February, 2005,
Quality Workshop

• Fitness for use is appropriate
• This implies that user play a paramount role in
  quality assessment
• Thus, we begin by collecting questions of
  interest to wide publics to focus on relevance
  and credibility
• We use technical expertise to identify indicators
  and document data and estimation qualities
• Iterating back and forth between users with
  unanswered questions and providers with data
  retains focus on fitness for use

4
May, 2005, NCC Meeting Presentation

Iteration 1
Iteration 2
Iteration N
Prioritization of Available Indicators
of Interest
NCC questions of interest
NCC questions of interest
NCC May 9 questions of interest
Design Group (catalogue of indicators and quality
metadata)
Indicator Framework 0.8
Outreach Group (widescale input on questions of
interests and indicator gaps evident in existing
indicators)
Time
5
Charge to the Domain Groups

• Groups to identify indicators and data series
  that are relevant to the questions contained in
  the revised version 0.8 of the indicator set
  framework
• can add questions that they believe would be of
  interest to stakeholders
• may also identify instances in which indicators
  are not currently being measured
•  
• Domain group members are chosen for their
  personal expertise technical knowledge of
  indicators and/or perspectives on information
  needed by a large constituency.
• they are not agents/representatives of their
  employers or organizations

6
Progress on Indicator Selection
State of the USA Indicator Set Framework (0.8
revised)
THE SOCIETY Population Health Shelter Education Re
search Innovation Personal Safety Families/House
holds, Children Seniors Communities Civic
Engagement Democracy Governance National
Security Values Culture
THE ECONOMY Production, Productivity National
Wealth Individuals, Families Households Employme
nt Labor Markets Business Financial
Markets Prices Inflation Government
Non-Profits The World Economy

• Quality of Life
• Growth
• Opportunity
• Liberty
• Diversity
• Mobility
• Poverty
• Sustainability
• Well-being
• Security

THE ENVIRONMENT The Landscape (including
Submerged Lands) Soil, Water Air Animals,
Plants Ecosystems Goods Services
7
KNII Indicator Set Design Subgroup Economy

• Carol Carson, former Director of Statistics
  Department at the International Monetary Fund,
  and former Director of the Bureau of Economic
  Analysis (Chair)
• Ron Blackwell, Chief Economist, AFL-CIO
• William J. Dennis, Senior Research Fellow,
  National Federation of Independent Business
  Education Foundation
• John Iceland, Professor, Department of Sociology,
  University of Maryland
• Cheryl Kerr, Economist, Bureau of Labor
  Statistics, Department of Labor
• Charles Kolb, President, Committee for Economic
  Development
• Catherine L. Mann, Senior Fellow, Institute for
  International EconomicsRobert McGuckin, Director
  of the Economic Research, The Conference Board
• Rosemary Marcuss, Deputy Director, Bureau of
  Economic Analysis, U.S. Department of Commerce
• Edward B. Montgomery, Dean, College of Behavioral
  and Social Sciences, University of Maryland
• Janet Norwood, former U.S. Commissioner of the
  Bureau of Labor Statistics
• Timothy Smeeding, Director, Center for Policy
  Research, Maxwell School, Syracuse University

8
KNII Indicator Set Design Subgroup Society

• Norman Bradburn, Senior Fellow, National Opinion
  Research Center (Chair)
• David Crowe, Senior Staff Vice President, Federal
  Regulatory and Housing Policy, National
  Association of Home Builders
• Emerson Elliott, Director, Program Standards
  Development Project, National Council for
  Accreditation of Teacher Education
• Jacob Feldman, Senior Fellow, National Opinion
  Research Center
• Roderick Harrison, Director, DataBank, Joint
  Center for Political and Economic Studies
• Sally Hillsman, Executive Officer, American
  Sociological Association
• Katherine Rosich, Policy Analyst, American
  Sociological Association
• Kenneth Hodges, Chief Demographer, Claritas, Inc.
• Gail Leftwich, Executive Director, By the People,
  MacNeil/Lehrer Productions
• Matthew Stagner, Director, Center on Labor, Human
  Services, and Population, The Urban Institute
• James Steinberg, Vice President/Director, Foreign
  Policy Studies Program, The Brookings Institution
• James Svara, Professor, Department of Political
  Science and Public Administration, North Carolina
  State University

9
KNII Indicator Set Design Subgroup Environment
William C. Clark, Harvey Brooks Professor of
International Science, Public Policy and Human
Development, Kennedy School of Government,
Harvard University (Co-Chair) Robin OMalley,
Director, Environmental Reporting Program, The H.
John Heinz III Center for Science, Economics and
the Environment (Co-Chair) Thomas Burke,
Associate Chair and Professor, Department of
Health Policy and Management, Johns Hopkins
Bloomberg School of Public Health Dennis Child,
Professor, Department Forest, Rangeland
Watershed Stewardship, College of Natural
Resources, Colorado State University Clifford
Duke, Director of Science Programs, Ecological
Society of America Paul Freedman, Paul Freedman,
Principal, Limno-Tech, Inc. Michel Gelobter,
Executive Director, Redefining Progress Bob
Green, Kennecott Energy Company Jerry L.
Hatfield, Laboratory Director, USDA/ARS National
Soil Tilth Laboratory H. Theodore Heintz, Jr.,
White House Council on Environmental
Quality Robert L. Hendricks, Policy Analyst, USDA
Forest Service International Programs Brian
Kernohan, Forest Capital Partners Kimberly T.
Nelson, Assistant Administrator and Chief
Information Officer Office of Environmental
Information
10
KNII Indicator Set Design Subgroup Environment,
contd

• Gordon Orians, Professor Emeritus, Department of
  Biology, University of Washington
• Duncan Patten, Research Professor of Ecology, Big
  Sky Institute, Montana State University
• David Policansky, Staff Officer DELS, National
  Research Council
• Walter Reid, Science Director, Millennium
  Ecosystem Assessment
• Don Scavia, Research Associate Dean and Professor
  of Natural Resources and Environment at the
  University of Michigan
• Peggy Shepard, Executive Director and Co-founder
  of West Harlem Environmental Action, Inc.
• Jim Solyst, American Chemistry Council
• John Talberth, Director, Sustainability
  Indicators Program, Redefining Progress
• Terry Young, Senior Consulting Scientist,
  Environmental Defense

11
Practical Needs of the Domain Groups Met through
a Set of Derived Principles

• The domain groups encountered various practical
  issues e.g.
• what to do about redundant questions?
• what to do with more or less abstract questions?
• what to do about missing questions?
• what to do about questions about causes?

12
Principles Guiding KNII Domain Groups

• Unanswered Questions Motivate Indicators
  Indicators Motivate Data Sources
• Action derive indicators from
  questions from users use
• technical experts to identify
  data sources for indicators
• Number of Key Indicators to Be Determined by User
  Demand
• Action user interests will guide the selection
  process
• Indicators of What and How Much but Not Why
• Action focus on questions about prevalence,
  magnitude, distribution, etc. of various
  phenomena, not influences on outcomes

13
Principles Guiding KNII Domain Groups

• Inclusion of Both Factual and Attitudinal
  indicators
• Action when questions pertain to perceptions of
  persons, permit identification of subjective
  indicators
• Preference for Indicators that Allow for Local,
  International, and Demographic Comparisons
• Action groups searched for indicators that had
  data available for diverse comparisons
• Preference for Time Series
• Action groups searched for indicators with long
  histories of measurement
• Providing Context for Locally Available
  Indicators
• Action focus on indicators that may have local
  analogues

14
Principles Guiding KNII Domain Groups

• Use of Existing Data and the Identification of
  Data Gaps
• Action give greater weight to existing
  indicators, but identify indicators currently
  without data sources
• Assembling and Comparing Indicators and Data from
  Different Sources
• Action identify indicators related to one
  another but not normally published together
• Combining Indicators and Data from Government and
  Non-Government Sources
• Emphasis on Collection of and User-friendly
  Access to Metadata
• Quality Criteria Guiding Selection of Indicators
• Action collect metadata on quality criteria,
  prepare for user evaluation

15
KNIIs Information Management System

• Customized relational database
• Provides storage, analysis, and reporting of
• Questions
• Proposed Indicators
• Data References
• Metadata

16
Drill-down to local areas

• Currently working with local indicator groups to
  develop areas of mutual interest
• Local groups may benefit from the framework of a
  well-worked national effort
• They may also benefit from the ability to provide
  context for local efforts and comparability with
  other local projects

17
Cross-cutting Issues

• Beginning work with todays discussion of 4
  cross-cutting issues
• Purpose of this work is to expand question and
  indicator set to include these issues.
• No decision as yet on which cross-cutting issues
  will be included in KNII

18
Indicator Selection Process

• Work is underway to develop a mechanism to make
  final decisions on indicator set
• Next step in this process is review of proposal
  by the KNII Steering Committee