National Institutes of Health

1
National Institutes of Health

• Use of E-Tools for Organizational Performance
  Assessments Evidence-based Management
  NIH GPRA PART
• Deborah Duran, Ph.D.
• Systemic Assessments Branch
• Office of Portfolio Analysis Strategic
  Initiatives

2
Overview of Presentation

• NIH Overview
• Government Performance
• Program Performance Monitoring System (PPMS)
• Information Data Collection
• Knowledge Management
• Reporting
• Portfolio Analysis for goal replacement

3
The Nations Leading Medical Research Agency

• NIH Role
• Worlds premier medical research center and
  steward of medical and behavioral research for
  the nation

• NIH Goal
• Acquire new knowledge to help prevent, detect,
  diagnose, and treat disease and disability

4
NIH Organizational Structure
Office of the Director Staff Offices Office of
Extramural Research Office of Intramural
Research Office of Management/Chief Financial
Officer Office of Science Policy Office of
Communications and Public Liaison Office of
Equal Opportunity and Diversity Management Office
of Program Coordination Office of Legislative
Policy and Analysis Office of Community
Liaison Executive Office Office of the
Ombudsman/Ctr. for Cooperative Resolution NIH
Ethics Office
Immediate Office of the Director
Office of the Director Program Offices Office of
Research on Womens Health Office of AIDS
Research Office of Behavioral and Social Sciences
Research Office of Disease Prevention
National Institute on Alcohol Abuse and Alcoholism
National Institute of Allergy and Infectious
Disease
National Institute of Arthritis and
Musculoskeletal and Skin Diseases
National Human Genome Research Institute
National Heart Lung and Blood Institute
National Eye Institute
National Cancer Institute
National Institute on Aging
National Institute of Biomedical Imaging and
Bioengineering
National Institute of Child Health and Human
Development
National Institute of Dental and Craniofacial
Research
National Institute of Deafness and Other
Communication Disorders
National Institute of Diabetes and Digestive and
Kidney Diseases
National Institute of Environmental Health
Sciences
National Institute of General Medical Sciences
National Institute on Drug Abuse
National Institute of Neurological Diseases and
Stroke
National Institute of Nursing Research
National Institute of Mental Health
National Library of Medicine
John E. Fogarty Center for Advanced Study in the
Health Sciences
National Center for Complementary and Alternative
Medicine
National Center for Minority Health and Health
Disparities
National Center for Research Resources
5
NIH Overview

• 27 Institutes and Centers (ICs) work individually
  and collectively
• Over 29 billion invested annually on medical
  research
• 80 of budget awarded through 50,000 competitive
  grants to support over 325,000 scientists and
  research personnel in the U.S. and abroad
• 10 of budget supports research led by 6,000
  intramural principal investigators

6
NIH HAS ONE PROGRAM
Research

• For NIH level performance reporting
• GPRA
• PART

7
NIH mission to uncover new knowledge that will
lead to better health for everyone.
GPRA Performance Driver Map
FUNCTIONAL AREAS
DRIVERS
PERFORMANCE
Scientific Research Outcomes
Generation of new scientific knowledge of new or
improved prevention, diagnosis, and treatment of
disease

• Short/Long term
• Low/High risk
• Intramural/Extramural

• Knowledge Dissemination
• Translation
• Field input
• Data / Repositories

Communication and Transfer of Results
Shared knowledge for public benefit
Capacity Building and Research Resources

• Training
• Technology
• Facilities
• Partnerships

Highly trained basic and clinical scientists
leadership in technology and facilities
Strategic Management of Human Capital
Availability of highly skilled and diverse
workforce enhanced work environment

• Performance-based approach
• Workplace Management
• Adequate Workforce

Program Oversight and Improvement
Enhanced program management and compliance
with relevant laws

• Results-oriented Management
• Resource Management
• Stewardship

8
Core Criteria for Planning and Reporting
Performance Goals

• Representative (Disease/Disorders/Management)
• Trans-NIH Approach
• Annual Budget Performance Target-Based Approach
• Balanced Scientific Research Portfolio
  (Intramural/Extramural, Basic/Clinical, ICs)
• RD Criteria (Quality, Relevance, Performance)
• 4 Ps Predictive, Preemptive, Personalized and
  Participatory
• Scientific Difficulty (Low-Medium-High)
• Time (Short-Term, Mid-Range, Long-Term)

9
Scientific Risk and Time Matrix
Increasing Level of Difficulty
1-3 years
4-7 years 8-10
years Time
10
Annual Targets

• Goals span from 1 to 10 years therefore, assess
  annual performance
• Prospective Quantifiable Annual Scientific
  Targets
• All IC/Office annual activities or contributions
  are called Strategic Targets
• Reported Targets are called Performance Targets
• Monitor all Targets but only submit Performance
  Targets
• Representative Target Reporting
• SROs 1 performance target per goal
• Non-SROs 1-5 Performance targets per goal
• Adaptive measure scientific justification

11
PART Scores
12
Information Knowledge

• Transitions from
• Age of Information Technology / Data Management
• to
• Age of Knowledge / Collects Connect
• Capture data
• Structure data
• Analyze information
• Design Application

13
Program Performance Monitoring System (PPMS)

• Centralizes performance monitoring reporting
• Enhances coordination communication among
  collaborators
• Reduces workforce burden
• Improves analysis of budget / performance
  information
• Two Components of PPMS
• Systemic Assessments Branch (SAB) Website (VPS
  Interface)
• Useful links to relevant resources and
  performance-related information at NIH
• Provides single point of access to NIHs first
  centralized budget-performance reporting system
• Calendar with relevant guidance
• Visual Performance Suite (VPS)
• Centralized online database for collecting,
  storing, analyzing and reporting trans-NIH
  performance and budget data
• Maintain information used to plan activities at
  NIH
• Central point of access

14
SAB Homepage NIH View
15
SAB Homepage GPRA Goal Performance Reporting
16
INFORMATION DATA COLLECTION
17
Budget Performance Integration (BPI) Functional
Area Level

• Information structured to match NIH business
  process
• Scorecard Configured to display budget and
  performance information
• User can drill down to the next level of
  granularity

18
Functional Level to Goal Level

• Goal level displays all goals and funding
  associated with the functional area
• Displayed box illustrates navigation system
  designed to allow users to
• Navigate to any part of the system
• Display summary of information for goal
• Upload documents or form discussion using blog
  tread

19
Goal Data Entry

• Goal data entry screen has been configured to
  capture data necessary to support reporting
  needs-tailored quantitative and qualitative data
  fields

20
Goal Level to Targets Level
21
Target Data Entry / Tailored Fields

• Tailored fields can be added to collect
  reportable and un-reportable information
• Data collected at target levels automatically
  aggregate to goal levels

22
Budget Reporting

• Budget is integrated with workflow of configuring
  Goals and Targets
• Comment box appears/required when threshold for
  change is crossed
• Email alerts are sent to contacts when a goal is
  rejected or needs revision

23
Performance Page

• Annual performance is collected on the
  performance page
• An automated notification system alerts users
  when annual performance results are due or missed
• Alert contains hyperlink that links users
  directly to reporting page

24
PART Reporting Page
25
KNOWLEDGE MANAGEMENT
26
Analytic Tool
Strategy Map with Attributes

• Goal linkage can be represented dramatically on a
  map
• Attributes are tailored for sorting
• Attributes provide snapshot of how the elements
  are performing

27
Decision-Makers Tools for
Planning and Reporting

• VPS produce graphics of budget-performance data
  in system
• Graphical Analyses supports various types of
  charts graphs and formats
• Summarize budget performance data
• Compare dimensions
• Filter information
• Drill into a dimension
• Trending
• Modeling
• Resource for organizational performance analysis
  and portfolio management

28
Summarized Data(Core Criteria Trans-NIH
Approach)

• X-Axis and Legend can be interchanged dynamically
• Chart labels can also be updated

29
Compare Dimensions(Core Criteria Scientific
Risk)

• Users can navigate from scorecard to graphics
  with a single mouse click
• Preconfigured graphic can be further analyzed
  using menus on left panel
• Control panel can be hidden to draw a full-screen
  chart

30
Filter Information

• Data can be filtered so only top n bottom n
  or middle n is displayed
• Graphics feature has the ability to query the
  database by specifying any number of properties
  of an element in a plan (For example, user can
  configure graphic to display all goals that has a
  specific word such as cancer).

31
Drill Into a Dimension(Core Criteria
Representative Performance Goals)

• Users can drill down to the lowest level of a
  dimension using a filter
• Chart can be display in multiple chart types, in
  3D and with table

32
Charting Multiple Measures

• Multiple measures can be simultaneously charted
  on a stacked vertical bar chart

33
REPORTING
34
VPS Reports

• Analytic graphics can be saved as jpeg or gif
  files imported into PowerPoint
• Customized reports for each module or reporting
  requirement
• 12 tailored budget performance reports are
  currently produced on VPS
• Reports can be generated with a button push
  printed, exported, and/or saved as
• MS Word documents
• MS Excel spreadsheets

35
Sample VPS Report Performance Goal Narrative
36
VPS Summary Report Budget/Performance
37
VPS GPRA Budget Report
38
VPS PART Report
39
PORTFOLIO ANALYSIS FOR GPRA GOAL REPLACEMENT
40
Portfolio Analysis Using Core Criteria

41
Portfolio Analysis Using Funding

• Research, Condition, and Disease Categorization
  (RCDC) system can help identify
• High priority research areas appropriate for
  representation in a GPRA goal
• Potential goal lead

42
RCDC for GPRA reporting
QVR/RCDC can be used to retrieve RCDC categorized
grants based on several criteria (Text search, by
IC, FY etc)
43
GPRA goal 3.4a using RCDC/QVR
Finger Print
By 2013 identify and characterize at least 2
candidate genes influencing substance use,
psychotic disorders using high risk family, twin,
special populations
Concept Name Concept Rank Bipolar
Disorder 95 Schizophrenia 85 Alcohol
dependence 65 proband 65 Substance Use
Disorder 65 endophenotype 60 Genome
Scan 58 Family 58 Conduct Disorder 58 Major
Depressive Disorder 58 Borderline pers
Disorder 58 ADHD 58 Autistic Disorder 58 LD
Mapping 58 Psychotic Disorders 50 Twin
Multiple Birth 50 Alcoholism 50 Twin
Studies 45 Stress Disorders 44 autism spectrum
disorder 44 Pathological Gambling 41 Siblings 4
0 Personality Disorders 40 psychogenetics 40
Project list from intersection
Mental Health Alcoholism Drug abuse
Genetics Human Genome
ICs NIMH, NIDA, NIAAA Key word search
44
Project list GPRA3.4a
5 R01AA015168-03 Alcoholism Susceptibility Genes
in High Density Families 5 P50AA011998-09 MARC
Genetic Epidemiology of Alcoholism
Comorbidity 5 R01MH061613-07 Genetic Linkage
Studies in Bipolar Disorder Families 5
R01DA013240-07 Twin Study of ADHD, CD and
Substance Abuse 5 R01MH042243-16 Genetic Linkage
and Association in Bipolar Disorder 3
R01MH060912-06S1 Genetics of Early-Onset Major
Depression 1 R01MH079799-01 Using Genetics to
Dissect Schizhoprenia, Bipolar Disorder, and
Depression 5 R01MH059552-07 Genetics of Early
Onset Depression 5 R01AA013320-05 Molecular
Epidemiology of Alcoholism 2- Big Sibships 5
R01MH069856-03 Genetics of Bipolar Disorder in
Latino Populations 5 R01MH059535-08 Collaborative
Genetic Study of Bipolar Disorder
45
Analytic Results
It is not a question how much a person knows, but
what use s/he can make of what s/he
knows. Josiah Gilbert Holland 1866
Data Management
Data Analysis
Information
Knowledge
Intuition
Systematic
46
Systemic Assessments Branch Office of Portfolio
Analysis Strategic Initiatives (OPASI)National
Institutes of Health (OD)9000 Rockville
Pike Bldg 2, Rm 1E10Bethesda, MD 
20892ph 301.496.6205fax [phone]email
durand_at_od.nih.gov