Brian Zuckerman

1
Results of the Assessment of the Defense
Experimental Program to Stimulate Competitive
Research (DEPSCoR)
Brian Zuckerman Presented to COSEPUP Evaluation
of the National Science Foundation's Experimental
Program to Stimulate Competitive Research
(EPSCoR) and Similar Programs in Other Federal
Agencies December 17th, 2012
2
Study Origin and Timeline

• FY2008 Defense Authorization Act (Public Law
  110-181), Section 241, instructed the Secretary
  of Defense to utilize a defense Federally Funded
  Research and Development Center (FFRDC) to carry
  out an assessment of the DEPSCoR program.
• Institute for Defense Analyses was tasked to
  conduct study February 2008
• Results briefed to Senate and House Armed
  Services Committee staff (SASC/HASC) November
  2008
• Study results cleared for public distribution
  January 2009

3
Legislative Mandate for Study

• Tangible results and progress toward the
  objectives of the program
• Applications used by, or supportive of,
  operational users
• Expanded national research infrastructure
• Activities consistent with statute
• Assessment of program elements
• Assessment of activities of state committees
• Advantages and disadvantages of institution
  state based formulas
• Mechanisms for improving the management and
  implementation of the program

4
Descriptive Statistics

• Between 1993 and 2008, 729 total DEPSCoR awards
• 546 individual Principal Investigators (PIs)
• 121 PIs with multiple awards
• 42 with three or more
• 1 PI with eight awards
• 22 of PIs have won 42 of awards
• 1993-2008 funding of 243 million
• Decline after 2000 peak partially reversed in
  2008 competition
• 27 states and territories (states) have been
  eligible for at least one year since program
  authorized in current form in 1995, plus Missouri
  (eligible in 1993)
• All eligible states except for the Virgin Islands
  have won awards
• 19 (5) of eligible states have won 35 of awards
• Montana, Alabama, Oklahoma, Nebraska, South
  Carolina
• 7 (5) of institutions have won 28 of awards
• Montana State, U. Nebraska-Lincoln, U. Wyoming,
  West Virginia U., U. Arkansas

5
DEPSCoR Program Objectives

1. To enhance the capabilities of institutions of
   higher education in eligible states to develop,
   plan, and execute science and engineering SE
   research that is competitive under the
   peer-review systems used for awarding federal
   research assistance
2. To increase the probability of long-term growth
   in the competitively awarded financial assistance
   that institutions of higher education in eligible
   states receive from the federal government for
   science and engineering research

6
The DEPSCoR State Share of DOD SE Increased
Source National Science Foundation (NSF) Survey
of Federal Science and Engineering Support to
Universities, Colleges, and Nonprofit
Institutions and the DEPSCoR program database of
funding totals by state and year Note Graph
includes all states ever involved in DEPSCoR
program 1995-2008 Note The dotted lines
represent linear regression models applied to the
data
7
DEPSCoR Funding As a Percentage of DOD Funding in
DEPSCoR States
Sources NSF Survey of Federal Science and
Engineering Support to Universities, Colleges,
and Nonprofit Institutions and the DEPSCoR
program database of funding totals by state and
year

• DEPSCoR has declined in importance as a source of
  funding for eligible states since 2000

8
Success Varied Among States
Sources NSF Survey of Federal Science and
Engineering Support to Universities, Colleges,
and Nonprofit Institutions, the IDA DEPSCoR
database, and the DEPSCoR program database of
funding totals by state and year

• DEPSCoR-eligible states fell into four groups
• 6 states Near or above threshold (AL, HI, LA, MS,
  NM, SC)
• 9 states Rising fast (AK, ID, KY, ME, MT, NE, NV,
  ND, SD)
• 6 states Middle (AR, DE, KS, OK, RI, TN)
• 4 states 2 territories Lagging (NH, PR, VT, VI,
  WV, WY)

9

Average DEPSCoR Funding as a Fraction of DOD
Funding by State
Sources NSF Survey of Federal Science and
Engineering Support to Universities, Colleges,
and Nonprofit Institutions, the IDA DEPSCoR
database, and the DEPSCoR program database of
funding totals by state and year Note Average
calculated as total DEPSCoR funds during eligible
years divided by total DOD funds during eligible
years

• DEPSCoR share of university SE RD funding
  varies substantially by state
• More than 60 for VT, WY
• Rising fast states (AK, ID, KY, ME, MT, NE,
  NV, ND, SD) DEPSCoR shares decline 2001-2005
  compared with 1993-2000

10
Legislative Mandate for Study

• Tangible results and progress toward the
  objectives of the program
• Applications used by, or supportive of,
  operational users
• Expanded national research infrastructure
• Activities consistent with statute
• Assessment of program elements
• Assessment of activities of state committees
• Advantages and disadvantages of institution
  state based formulas
• Mechanisms for improving the management and
  implementation of the program

11
Expanded National Research Infrastructure (1)

• Involving new investigators
• Recent (2006-2008) cohorts had about 60 new PIs
• Most Army-funded DEPSCoR awardees (82) had not
  previously received funding from Army Research
  Office (ARO)
• 56 of PIs had been funded by the NSF either
  previous to or within the same year of their
  first DEPSCoR award
• Training graduate students and postdoctoral
  fellows
• ARO and Office of Naval Research (ONR) data
  suggest that awards fund about 1 PhD, 1 Masters
  degree, 2 postdocs
• Building physical infrastructure
• DEPSCoR awards have supported purchase and
  maintenance of equipment but data not collected
  systematically by services

12
Expanded National Research Infrastructure (2)

• Leveraging new funding for defense-related
  research is limited
• 8 of non-DEPSCoR ARO awardees in DEPSCoR states
  received a DEPSCoR award before (or in the same
  year as) their first non-DEPSCoR ARO award
• 4 DEPSCoR awardees (less than 1) won a DOD
  Multidisciplinary University Research Initiative
  (MURI) award after or in the same year as their
  first DEPSCoR award
• 2 of these investigators received non-DEPSCoR DOD
  funding before their first DEPSCoR award
• Leveraging other funding
• 63 DEPSCoR awardees (12) received their first
  NSF funding subsequent to their first DEPSCoR
  funding

13
Legislative Mandate for Study

• Tangible results and progress toward the
  objectives of the program
• Applications used by, or supportive of,
  operational users
• Expand national research infrastructure
• Activities consistent with statute
• Activities were found to be consistent with
  statute
• Assessment of program elements
• Available data on DEPSCoR program activities and
  outcomes are insufficient for monitoring and
  evaluation purposes
• Assessment of activities of state committees
• State committees prioritized proposals that met
  state infrastructure development goals and
  reflected the mission/research needs of DOD
• Committee processes varied widely from state to
  state and limited and variable data prevented
  detailed assessment
• Advantages and disadvantages of institution
  state based formulas
• Mechanisms for improving the management and
  implementation of the program

14
Legislative Mandate for Study

• Tangible results and progress toward the
  objectives of the program
• Applications used by, or supportive of,
  operational users
• Expanded national research infrastructure
• Activities consistent with statute
• Assessment of program elements
• Assessment of activities of state committees
• Advantages and disadvantages of institution
  state based formulas
• Mechanisms for improving the management and
  implementation of the program

15
Approach Taken

• Assessment focused on comparing the current
  state-based formula with an institution-based
  criterion based on a maximum threshold for DOD
  SE research funding
• As directed in the study legislative mandate,
  particular emphasis was given to supporting
  defense missions and expanding the nation's
  defense research infrastructure
• Since the legislative charge for the assessment
  does not specify how expanding the nations
  defense research infrastructure should be
  interpreted, the assessment considered advantages
  and disadvantages using a variety of possible
  interpretations
• Assessment also considered alternative criteria
• Indicators of state-level ST capacity (SEI,
  Milken Institute)
• Normalization by state population (Census)
• State-based and institution-based criteria can be
  combined
• PI-level criteria are also possible

16
Effect of Institution Based Funding Threshold
Maximum DOD Funding Threshold All Universities With Non-Zero DOD Research Funding in 2005 Eligible for the 2008 Competition Potentially Eligible for the 2009 Competition
No limit 360 77 114
10 million 316 75 109
5 million 269 65 90
3 million 231 55 76
1 million 157 36 52

• 77 institutions in 2008 DEPSCoR-eligible states
  (360 total) received nonzero research funding
  from DOD in 2005
• 38 Carnegie Very High or High research
  institutions in DEPSCoR states
• 5 million threshold would make 269 institutions
  eligible
• Twelve of the 77 institutions in currently
  eligible jurisdictions (e.g., University of
  Delaware, University of Nevada, Brown, Clemson,
  Vanderbilt, University of Nebraska) would become
  ineligible
• Considering only Carnegie Very High or High
  research universities, shift would increase
  number of eligible universities from 38 to 121

17
Definition of Expanding National Research
Infrastructure

• If interpreted as increasing equity in funding
  among states or achieving state-level
  infrastructure goals, a state-based formula would
  be advantageous
• Current state-based formula for eligibility
  harnesses the state EPSCoR committees to
  coordinate infrastructure and capacity-building
  at the state level
• If intent is to increase the competitiveness of
  historically-underrepresented states, eligibility
  can easily be determined at a state level
• If interpreted as involving new investigators or
  institutions in defense-related research, an
  institution-based formula would be advantageous
• Allows targeting of programmatic resources toward
  investigators at institutions that have not
  historically built relationships with DOD
• Approach taken by late 1980s/early 1990s DOD
  Research Initiation Program
• While a state-based approach includes the
  flexibility to channel DEPSCoR proposals toward
  historically underrepresented universities or new
  investigators within an eligible state, the
  institution-based approach allows greater
  flexibility to target underrepresented
  universities and investigators throughout the
  entire country
• Could not be determined whether state-based or
  institution-based approach would elicit more
  qualified applications to support defense
  missions
• Larger number of eligible institutions implies
  more proposals, but quality indeterminate

18
Effect of State Population Normalization
05 DOD SE RD Funding, by State

• Green states were eligible in 2008, red states
  are graduates, blue states were never eligible
• Comparing top chart with bottom shows dramatic
  difference in order
• DEPSCoR graduates among highest per capita
  recipients
• Several DEPSCoR-eligible states (e.g., AK, DE,
  ND, RI, MT, SD) above average in funding per
  capita

05 Per capita DOD SE RD Funding, by State
19
Other Capacity Indicators

• Some DEPSCoR-eligible states (e.g., NH, RI, DE,
  KS) above average in Milken Institute state ST
  index
• Some graduated states below average
• Some DEPSCoR-eligible states (e.g., DE, KS, NE,
  NH, RI, VT) in the top or second quartiles for
  more than half of Science and Engineering
  Indicators seven RD output measures

20
Legislative Mandate for Study

• Tangible results and progress toward the
  objectives of the program
• Applications used by, or supportive of,
  operational users
• Expanded national research infrastructure
• Activities consistent with statute
• Assessment of program elements
• Assessment of activities of state committees
• Advantages and disadvantages of institution
  state based formulas
• Mechanisms for improving the management and
  implementation of the program

21
IDA Recommendations

• DOD should change the current process for review
  of proposals to focus more heavily on
  investigators future potential to conduct
  research rather than on their current research
  capabilities
• DOD program managers should be formally
  encouraged to serve as mentors and facilitators
  for DEPSCoR investigators seeking to engage in
  further defense-related research
• DOD should create data systems that will allow
  systematic tracking of DEPSCoR activities and
  outcomes
• Congress should re-examine and consider
  clarifying ambiguities in the DEPSCoR legislative
  mandate
• Once the DEPSCoR objectives have been clarified,
  redesign the program with a strategy for
  enhancing competitiveness at relevant level
  (e.g., individual, institution, state)

22
Some DEPSCOR Legislative Language Is Inconsistent

• Objectives specify that the research institution
  is the level at which competitiveness is to be
  enhanced but authorizing legislation also
  specifies that eligibility for DEPSCoR be
  determined at the state level
• 2008 change to eligibility criteria leaves
  ambiguous whether Congress intends DEPSCoR to
  target competitiveness at the institution or
  state level
• Objectives specify that goal is to increase
  probability of long-term growth in the
  competitively awarded financial assistance that
  institutions of higher education in eligible
  states receive from the federal government but
  eligibility determined based on DOD funding
• Statute does not discuss whether Congress intends
  DEPSCoR should focus narrowly on 6.1 research
  that is relevant to DOD missions and priorities
  although this is how program is run

23
Recommendations for Legislative Clarification

• To ensure that the program is implemented in a
  manner that meets current legislative priorities,
  Congress should clarify whether
• The program is intended to increase
  competitiveness for federal research funding in
  general or for particular types of research
  funding
• The program is intended to fund primarily basic
  research, primarily applied research, or a
  combination
• The primary unit at which competitiveness should
  be enhanced is the institution, state, or other
  (e.g., individual investigator)
• Eligibility criteria, funded activities, and
  other program elements should be structured in
  accordance with the programs objectives

24
Competitiveness can be Enhanced at Different
Levels

• DEPSCoR supports individual or small-group
  research projects and can therefore be understood
  to primarily target capacity-building at the
  level of the individual
• Though supporting training and purchase of
  equipment target capacity-building at level of
  the institution as well
• DEPSCoR also operates at state level (e.g.,
  involvement of EPSCoR committees, state-based
  eligibility criteria)
• While it might be argued that institutional
  competitiveness depends on individual
  competitiveness and state competitiveness depends
  on institutional competitiveness, these
  dependencies are neither straightforward nor
  self-evident

25
Structure Program to Enhance at the Desired
Organizational Level

• Assessment identified other programs that might
  serve as potential models for DEPSCoR
• State-level NSF EPSCoR/Research Infrastructure
  Improvement (RII) Awards
• One per state integrated into state ST plan
  EPSCoR committee involvement capacity building
  rather than research
• Centers NIH IDeA/Centers of Biomedical Research
  Excellence (COBRE)
• Large-team research, equipment, mentoring
  milestones for transition to support by standard
  NIH mechanisms
• Investigator-level DOD Research Initiation
  Program (RIP) or NIH/IDeA Academic Research
  Enhancement Award (AREA) Program
• Institution-based eligibility criteria small
  research awards to individual investigators
  build relationships/expertise to allow for
  transition to support by standard research
  mechanisms

26
Backup Slides
27
State Eligibility over Time
Source IDA analysis of DEPSCoR BAAs and DOD
DEPSCoR press releases Note Red cells denote
years in which jurisdictions were not eligible,
and blue cells denote years in which states were
eligible. Note Missouri, which was eligible only
in the 1993-4 competition, was not included in
the table.
28
No Obvious Correlation

Number of DEPSCoR awards and program funding
Linear regression model of DEPSCoR state share of
DOD university funding
29
DEPSCoR Emphasis is on Basic Research

• Research oriented towards developing
  operationally useful devices or components is not
  a legislatively-mandated program goal
• DEPSCoR projects mostly fund basic research
• Not a statutory requirement
• Administered by basic research organizations
  within Air Force and Army, plus Office of Naval
  Research (ONR)
• Interviews with DEPSCoR program officers suggest
  that more applied research does not perform well
  in review
• DEPSCoR program officers tend to track
  transitions to 6.2 or 6.3 research, but not
  further
• Two transitions to operational use were found

30
Two Transitions to Operational Use Were Found

• Ronald DeVore, University of South Carolina
  Wavelet mathematics for image compression for
  tactical applications
• DeVore and colleagues collaborate with program
  managers at Naval Air Warfare Center NAWC at
  China Lake to deliver wavelet-based image
  processing platform
• Charles Creussere of NAWC implements
  wavelet-based image processing system for
  navigation in the Tomahawk Block II program
• Michael Pursley, Clemson University wireless,
  mobile, distributed, multimedia communication
  networks
• Pursley and colleagues working since 1970s with
  ITT on tactical radio development
• Group used DEPScoR funding to support research
  that provided better anti-jam communications and
  greater multiple-access capability
• Research led to the Soldier Level Integrated
  Communications Environment (SLICE) wideband
  networking waveforms that have been integrated
  into the SINCGARS radio