Cycle 3 Survey Results

1
Results from the Performance Management
Survey Prepared by Janice Rouiller and Laura
Stouffer, SAIC and Joe Wolski, OQM 9 May 2005
2
Table of Contents

• Survey Background....3
• Respondent Information.7
• Perceptions of Cycle 3 PM Implementation
  ..15
• Perceptions of Needs for Cycle 4
  ...52
• Organizational Climate Perceptions
  ...68
• Organizational Climate Perceptions by Program
  Area78
• Organizational Climate Perceptions by
  Role84
• Customer Scorecard Ratings..8
  8
• Summary of Comments.....101
• Summary.........105
• Appendix - Individual Comments from
  Survey...111

3
Survey Background
4
Purpose

• Administer a customized version of the ORS
  Customer Scorecard to
• Gather feedback on past Performance Management
  (PM) implementation efforts
• Collect needs assessment information to tailor
  the on-going implementation of PM
• Gather data on the required Customer Scorecard
  dimensions
• Collect organizational culture change information
  for the Evaluation Study

5
Methodology

• Administer the survey at the completion of the
  third cycle of Performance Management
  implementation on an organization-wide basis
• Which occurred in February 2005
• Designed and tested web survey during February
  2005
• Survey went live on 3 February, 2005
• Email requests sent to all PM Team Leaders,
  Members, and additional senior ORS/ORF managers
• First reminder email sent 17 February 2005
• Closed survey field period 24 February 2005
• Data file sent to contractor for analysis

6
Survey Distribution
Number of email recipients 186 Numb
er of respondents 41
Response Rate 22
7
Respondent Information
8
Respondents by Role
N 45
Number of Respondents
Note Multiple responses allowed.
9
Respondents by Organization
N 41
Number of Respondents
10
Respondents by Program Area
N 41
Number of Respondents
11
Experience with the PM Process
N 41
N 40
N 40
N 36
N 35
Percent Yes
12
Approximately what percent of time did you spend
on PM-related activities during FY04?
N 41
Number of Respondents
Percent of time Spent
13
On average, how often did your PM team meet
during FY04?
N 41
Number of Respondents
How Often PM Team Met
14
To what extent did your team members work between
meetings to implement PM?
N 41 Mean 6.71 Median 6.5
To No Extent
To a Great Extent
15
Perceptions of Cycle 3 PM Implementation (January
2004 - January 2005)
16
How helpful did you find the following in
implementing PM?
Not at all Helpful
Extremely Helpful
Mean Ratings
17
How helpful did you find the following tools in
implementing PM?
N 41
N 41
N 40
N 41
N 41
N 41
N 38
Not at all Helpful
Extremely Helpful
Mean Ratings
18
How helpful did you find the following tools in
implementing PM? Listings of the Services
Hierarchy
N 41 Mean 5.46 Median 6
Not at all Helpful
Extremely Helpful
19
How helpful did you find the following tools in
implementing PM? Performance Management Plan
(PMP) Template
N 41 Mean 7.15 Median 8
Extremely Helpful
Not at all Helpful
20
How helpful did you find the following tools in
implementing PM? Strategy Mapping
N 40 Mean 6.29 Median 7
Not at all Helpful
Extremely Helpful
21
How helpful did you find the following tools in
implementing PM? Measures Roadmap Template
N 41 Mean 6.18 Median 7
Not at all Helpful
Extremely Helpful
22
How helpful did you find the following tools in
implementing PM? Initiative Roadmap Template
N 41 Mean 6.25 Median 7
Not at all Helpful
Extremely Helpful
23
How helpful did you find the following tools in
implementing PM? Measures Reporting Form
N 41 Mean 5.88 Median 7
Not at all Helpful
Extremely Helpful
24
How helpful did you find the following tools in
implementing PM? PM Presentation Template for
Conference
N 38 Mean 6.52 Median 8
Not at all Helpful
Extremely Helpful
25
How helpful did you find the following data
analysis services in implementing PM?
N 41
N 41
N 40
N 41
N 41
Extremely Helpful
Not at all Helpful
Mean Ratings
26
How helpful did you find the following data
analysis services in implementing PM Design of
Data Collection Forms
N 41 Mean 5.38 Median 5.5
Not at all Helpful
Extremely Helpful
27
How helpful did you find the following data
analysis services in implementing PM
Establishing Data Collection Methodologies
N 41 Mean 6.00 Median 7
Not at all Helpful
Extremely Helpful
28
How helpful did you find the following data
analysis services in implementing PM Graphing
Data
N 40 Mean 6.06 Median 7
Not at all Helpful
Extremely Helpful
29
How helpful did you find the following data
analysis services in implementing PM Process
Behavior Chart Analysis
N 41 Mean 5.86 Median 7
Not at all Helpful
Extremely Helpful
30
How helpful did you find the following data
analysis services in implementing PM ORS
Customer Scorecards
N 41 Mean 6.45 Median 7
Not at all Helpful
Extremely Helpful
31
How helpful did you find the following
communication vehicles in implementing PM?
N 40
N 40
N 40
Not at all Helpful
Extremely Helpful
Mean Ratings
32
How helpful did you find the following
communication vehicles in implementing
PM Performance Management Website
N 40 Mean 4.78 Median 4
Not at all Helpful
Extremely Helpful
33
How helpful did you find the following
communication vehicles in implementing PM PM
Related Emails from OQM
N 40 Mean 5.74 Median 6
Not at all Helpful
Extremely Helpful
34
How helpful did you find the following
communication vehicles in implementing PM PM
Conference
N 40 Mean 5.44 Median 6
Not at all Helpful
Extremely Helpful
35
How helpful did you find assistance from the
following groups in implementing PM?
N 39
N 40
N 38
N 38
Not at all Helpful
Extremely Helpful
Mean Ratings
36
How helpful did you find assistance from the
following groups in implementing PM? OQM Staff
N 39 Mean 6.38 Median 8
Not at all Helpful
Extremely Helpful
37
How helpful did you find assistance from the
following groups in implementing PM? PM
Consultants
N 40 Mean 7.97 Median 9
Not at all Helpful
Extremely Helpful
38
How helpful did you find assistance from the
following groups in implementing PM? Other PM
Teams
N 38 Mean 4.96 Median 6
Not at all Helpful
Extremely Helpful
39
How helpful did you find assistance from the
following groups in implementing PM? Your
Supervisor
N 38 Mean 7.82 Median 8
Not at all Helpful
Extremely Helpful
40
Immediate Supervisor Involvement in PM
Mean Ratings
N 38
To a Great Extent
To no Extent
N 39
Extremely Helpful
Not at all Helpful
41
What extent was your immediate supervisor
involved in your PM process during this PM cycle?
N 38 Mean 7.53 Median 8
To No Extent
To a Great Extent
42
How helpful was the involvement of your immediate
supervisor in PM related matters during this
cycle?
N 39 Mean 7.60 Median 8
Not at all Helpful
Extremely Helpful
43
To what extent have your knowledge/skills/abilitie
s (KSAs) improved in the following areas during
this PM cycle?
N 38
N 40
N 40
N 40
N 39
N 38
N 39
N 38
To No Extent
To a Great Extent
Mean Ratings
44
To what extent have your knowledge/skills/abilitie
s (KSAs) improved in the following areas during
this PM cycle? Balanced Scorecard Approach to PM
N 38 Mean 6.03 Median 7
To No Extent
To a Great Extent
45
To what extent have your knowledge/skills/abilitie
s (KSAs) improved in the following areas during
this PM cycle? Methods to Gather Data
N 40 Mean 6.18 Median 7
To No Extent
To a Great Extent
46
To what extent have your knowledge/skills/abilitie
s (KSAs) improved in the following areas during
this PM cycle? Methods to Analyze Data
N 40 Mean 5.68 Median 7
To No Extent
To a Great Extent
47
To what extent have your knowledge/skills/abilitie
s (KSAs) improved in the following areas during
this PM cycle? Presentation Preparation and
Delivery
N 40 Mean 5.54 Median 7
To No Extent
To a Great Extent
48
To what extent have your knowledge/skills/abilitie
s (KSAs) improved in the following areas during
this PM cycle? Working in a Team
N 39 Mean 5.54 Median 6
To No Extent
To a Great Extent
49
To what extent have your knowledge/skills/abilitie
s (KSAs) improved in the following areas during
this PM cycle? Using Excel
N 38 Mean 5.03 Median 6
To No Extent
To a Great Extent
50
To what extent have your knowledge/skills/abilitie
s (KSAs) improved in the following areas during
this PM cycle? Using PowerPoint
N 39 Mean 4.79 Median 5
To No Extent
To a Great Extent
51
To what extent have your knowledge/skills/abilitie
s (KSAs) improved in the following areas during
this PM cycle? Using Visio
N 38 Mean 4.19 Median 5
To No Extent
To a Great Extent
52
Perceptions of Needs for Cycle 4
53
In planning for the next PM cycle, how helpful
would each of the following be to your PM Team?
Not at all Helpful
Extremely Helpful
Mean Ratings
54
In planning for the next PM cycle, how helpful
would each of the following tools/resources be to
your PM Team?
N 40
N 40
N 37
N 39
N 38
N 37
Not at all Helpful
Extremely Helpful
Mean Ratings
55
In planning for the next PM cycle, how helpful
would each of the following tools/resources be to
your PM Team PM Consultants Provided by OQM
N 40 Mean 7.84 Median 9
Not at all Helpful
Extremely Helpful
56
In planning for the next PM cycle, how helpful
would each of the following tools/resources be to
your PM Team OQM Staff
N 40 Mean 6.57 Median 7
Not at all Helpful
Extremely Helpful
57
In planning for the next PM cycle, how helpful
would each of the following tools/resources be to
your PM Team Regular Meetings with your
Management
N 37 Mean 7.03 Median 8
Not at all Helpful
Extremely Helpful
58
In planning for the next PM cycle, how helpful
would each of the following tools/resources be to
your PM Team Rotating Quarterly Presentation
Meetings
N 39 Mean 5.94 Median 7
Not at all Helpful
Extremely Helpful
59
In planning for the next PM cycle, how helpful
would each of the following tools/resources be to
your PM Team Yearly Performance Management
Conference
N 38 Mean 5.59 Median 7
Not at all Helpful
Extremely Helpful
60
In planning for the next PM cycle, how helpful
would each of the following tools/resources be to
your PM Team IT Support to Establish Data
Collection Systems
N 37 Mean 7.47 Median 8
Not at all Helpful
Extremely Helpful
61
In planning for the next PM cycle, how helpful
would each of the following training sessions be
to your PM Team?
N 38
N 39
N 40
N 39
N 40
N 40
Not at all Helpful
Extremely Helpful
Mean Ratings
62
In planning for the next PM cycle, how helpful
would each of the following training sessions be
to your PM Team? Balanced Scorecard Methodology
N 38 Mean 5.97 Median 6
Not at all Helpful
Extremely Helpful
63
In planning for the next PM cycle, how helpful
would each of the following training sessions be
to your PM Team? Setting up Data Collection
Systems and Forms
N 39 Mean 6.45 Median 7
Not at all Helpful
Extremely Helpful
64
In planning for the next PM cycle, how helpful
would each of the following training sessions be
to your PM Team? Analyzing Data Through Graphs
N 40 Mean 6.58 Median 8
Not at all Helpful
Extremely Helpful
65
In planning for the next PM cycle, how helpful
would each of the following training sessions be
to your PM Team? Analyzing Processes with
Process Behavior Charts
N 39 Mean 6.54 Median 8
Not at all Helpful
Extremely Helpful
66
In planning for the next PM cycle, how helpful
would each of the following training sessions be
to your PM Team? Customer Assessment through
Interviews, Focus Groups and Surveys
N 40 Mean 6.86 Median 8
Not at all Helpful
Extremely Helpful
67
In planning for the next PM cycle, how helpful
would each of the following training sessions be
to your PM Team? ORS ABC Cost Accounting Model
N 40 Mean 6.56 Median 8
Not at all Helpful
Extremely Helpful
68
Organizational Climate Perceptions
69
Organizational Climate PerceptionsSummary
N 40
N 39
N 40
N 40
N 39
N 40
N 40
N 40
Strongly Disagree
Strongly Agree
Neither Agree nor Disagree
70
I see the value of performance management for
improving business operations in my area.
N 40 Mean 3.59 Median 4
Strongly Disagree
Strongly Agree
Neither Agree nor Disagree
71
My organization (ORS or ORF) is committed to the
performance management effort.
N 39 Mean 4.03 Median 4
Strongly Disagree
Strongly Agree
Neither Agree nor Disagree
72
I understand what the Services Hierarchy is and
its purpose.
N 40 Mean 3.50 Median 4
Strongly Disagree
Strongly Agree
Neither Agree nor Disagree
73
The culture of my organization (ORS or ORF) is
changing to be more results-oriented.
N 40 Mean 3.70 Median 4
Strongly Disagree
Strongly Agree
Neither Agree nor Disagree
74
Managers in my organization (ORS or ORF) believe
accountability is an important organizational
value.
N 39 Mean 3.84 Median 4
Strongly Disagree
Strongly Agree
Neither Agree nor Disagree
75
Performance management has contributed to
improvements in my area.
N 40 Mean 3.21 Median 4
Strongly Disagree
Strongly Agree
Neither Agree nor Disagree
76
Performance management will assist my
organization if we have to go through an A-76
competition.
N 40 Mean 3.56 Median 4
Strongly Disagree
Strongly Agree
Neither Agree nor Disagree
77
My PM Team (or the groups I am responsible for)
are actively involved in data collection and
analysis.
N 40 Mean 4.15 Median 5
Strongly Disagree
Strongly Agree
Neither Agree nor Disagree
78
Organizational Climate Perceptions by Program Area
79
Program and Employee Services Climate
PerceptionsPositive Responses
Percent Agree or Strongly Agree
N 7
N 7
N 7
N 7
N 7
N 7
N 7
N 7
80
Scientific Resources Climate PerceptionsPositive
Responses
Percent Agree or Strongly Agree
N 6
N 6
N 6
N 6
N 6
N 6
N 6
N 6
81
Security and Emergency Response Services Climate
PerceptionsPositive Responses
Percent Agree or Strongly Agree
N 8
N 8
N 8
N 8
N 8
N 8
N 8
N 8
82
Management Services Climate PerceptionsPositive
Responses
Percent Agree or Strongly Agree
N 3
N 3
N 3
N 3
N 3
N 3
N 3
N 3
83
Real Estate and Facilities Climate
PerceptionsPositive Responses
Percent Agree or Strongly Agree
N 4
N 4
N 4
N 4
N 4
N 4
N 4
N 4
84
Organizational Climate Perceptions by Role
85
Division Directors/Office Directors Climate
Perceptions
Percent Agree or Strongly Agree
N 5
N 5
N 5
N 5
N 5
N 5
N 5
N 5
86
PM Team Leaders Climate Perceptions
Percent Agree or Strongly Agree
N 7
N 7
N 7
N 7
N 7
N 7
N 7
N 7
87
PM Team Members Perceptions
Percent Agree or Strongly Agree
N 27
N 27
N 27
N 27
N 26
N 27
N 27
N 27
88
Customer Scorecard Ratings
89
Customer Scorecard Ratings for PMOverview for
Cycle 3
N 40
N 40
N 39
N 38
N 40
N 40
N 39
N 40
N 40
Unsatisfactory
Outstanding
Mean Ratings
90
Cost Satisfaction Ratings
N 40 Mean 5.92 Median 7
Unsatisfactory
Outstanding
91
Quality Satisfaction Ratings
N 40 Mean 6.77 Median 8
Unsatisfactory
Outstanding
92
Timeliness Satisfaction Ratings
N 39 Mean 6.67 Median 7
Unsatisfactory
Outstanding
93
Reliability Satisfaction Ratings
N 38 Mean 6.82 Median 8
Unsatisfactory
Outstanding
94
Availability Satisfaction Ratings
N 40 Mean 7.16 Median 8
Unsatisfactory
Outstanding
95
Responsiveness Satisfaction Ratings
N 40 Mean 7.16 Median 8
Unsatisfactory
Outstanding
96
Convenience Satisfaction Ratings
N 39 Mean 7.23 Median 8
Unsatisfactory
Outstanding
97
Competence Satisfaction Ratings
N 40 Mean 7.71 Median 8
Unsatisfactory
Outstanding
98
Handling of Problems Satisfaction Ratings
N 40 Mean 7.18 Median 8
Unsatisfactory
Outstanding
99
Customer Scorecard Ratings for PMTrend Analysis
Mean Ratings
N 40
N 68
N 37
N 40
N 68
N 70
N 39
N 67
N 67
N 38
N 68
N 67
Unsatisfactory
Outstanding
100
Customer Scorecard Ratings for PMTrend Analysis
(cont.)
Mean Ratings
N 40
N 68
N 69
N 40
N 68
N 68
N 39
N 68
N 66
N 40
N 68
N 69
N 40
N 65
N 65
Unsatisfactory
Outstanding
101
Summary of Comments
102
What did you value most about this cycle of
performance management?

• Knowledge and Experience Gained (4)
• Results show that we have improved since last
  year.
• Surveys have provided an opportunity to find out
  how customers perceive quality of service.
• Data gathering has helped quantify quality of
  service.
• Assistance provided by consultants and OQM staff
  (3)
• Interaction with OQM staff consultants in
  analyzing performance.
• Knowledgeable and helpful consultants.
• Measurement Tools (3)
• The use of data reporting template and the
  initiatives template.
• The Measures Reporting Form allowed us to look at
  the objectives and measures in a more relevant
  perspective.

Note Comments were sorted into categories. Most
frequently mentioned categories listed first.
103
What obstacles were most challenging to you
during this cycle of performance management?

• Lack of Time and/or Resources (6)
• Lack of time to devote to PMP and keep up with
  workload
• Short notice
• OQM website is out of date
• Miscellaneous (4)
• Learning to use Visio and Excel
• Preparing the report
• Lack of Management Involvement (3)
• Not having discussions about PM in staff meetings
• Managers need to use data in decision making
• Data Collection and Measurement (2)
• Obtaining data
• Lack of commitment from team members (2)
• Getting all team members to contribute

Note Comments were sorted into categories. Most
frequently mentioned categories listed first.
104
How can we help integrate PM into your daily
business activities?

• Increase support currently provided (6)
• Need help in learning how to distribute data
  collection throughout the division.
• Need help collecting and interpreting data.
• Miscellaneous (3)
• Work to establish timelines so that we see the
  year at a glance.
• Continue emphasis on the importance of
  performance management.

105
Summary
106
SummarySurvey Respondents

• Response rate of 22
• Respondents were primarily from ORS (73)
• The majority of respondents were team members
  (68)
• About 75 of respondents were experienced with PM
  (i.e., participated in FY03)
• About 60 of respondents attended FY04 training
• About 63 of respondents spent 20 or less of
  their time during the past year on PM
• Team meetings typically occurred once a month
  (41), biweekly (22), and quarterly (20)

107
Summary (cont.)Feedback on Past Cycle

• Most Helpful Tools/Resources
• PM consultants
• Service group supervisors
• PM template
• PM presentation template
• ORS Customer Scorecards
• OQM Staff.
• These are the same tools/resources receiving the
  highest ratings in FY03.
• KSA Improvements
• Just above mid-scale improvements noted on all
  topics with the following exceptions PowerPoint
  and Visio.
• Perhaps respondent familiarity with these tools
  has leveled off.

108
Summary (cont.)Planning for Next PM Cycle

• Future needs
• Most helpful to teams for the future would be
• PM consultants
• IT support for data collection systems
• Regular meetings with management to discuss PM
  data
• Training on customer assessment
• Training on analyzing data
• These requests are similar to previous years

109
Summary (cont.)Organizational Climate

• Majority of respondents either agree or strongly
  agree that
• PM Teams are actively involved in data collection
  (85)
• Their organization is committed to PM (77)
• Managers in their organization believe
  accountability is important (64)
• They see the value of PM for improving business
  operations (60)
• The culture of their organization is changing to
  be more results-oriented (58)

110
Summary (cont.)Customer Scorecard Trend Analysis

• Ratings on dimensions show minor variance from
  cycle to cycle but none of the differences are
  statistically significant.
• Current ratings
• Highest on competence, convenience, and handling
  of problems,
• Lowest on cost

111
Appendix

• Individual Comments From Survey

112
What did you value most about this cycle of
performance management? Individual Comments

• The opportunity to find out how my customers
  perceive the quality of the job I'm doing.
• Data gathering enabled us to quantify that
  construction inspection was decreasing our
  submittal review response time.
• Our consultant Janice Rouiller.
• The survey provided verification of the quality
  of service DMCS is providing the NIH community.
• The numbers show that last year's improvements
  have not only been sustained, but improved upon.
• Honestly, I must state NOTHING! This ploy that
  the PMP process is management tool is hog-wash,
  the true intent of the PMP is to tie it to how
  much time staff spend in each function and to cut
  staff.
• Interaction with OQM staff consultants in
  analyzing performance.

113
What did you value most about this cycle of
performance management? (cont.) Individual
Comments

• Patrick Patterson, is the consultant with whom
  our team works on PMP. His skills knowledge of
  all aspect of the program are excellent. He
  ensures that we see the process purpose
  rather than just do it.
• The willingness of the ORS Scientific resources
  to be first in the rotation of PM conferences.
• The use of data reporting template and the
  initiatives template.
• Meeting as a group to plan ahead.
• the introduction of the use of the Business
  Framework and the concept of the Balanced
  Scorecard.
• It was helpful to focus just on the high impact
  objectives.
• The Measures Reporting Form allowed us to look at
  the objectives and measures in a more relevant
  perspective.

114
What obstacles were most challenging to you
during this cycle of performance
management?Individual Comments

• Preparing the report is painful.
• Being taken away from my primary duties created
  undue stress and hardship when OQM wanted their
  deadlines met, yet they couldn't seem to get
  their parts done in a timely fashion.
• We did not have the time to do the customer
  survey focus groups we had planned on doing.
• OQM website is out of date and needs updating.
• The challenge of continuing performance gains
  without losing any quality.
• learning to use Excel and Visio.
• Lack of time to devote to PMP AND perform ones
  own work. PMP is almost a full time job. This
  is an OQM function and should be performed by
  that staff.
• Supervisors and managers should use data more
  often in their decision-making.

115
What obstacles were most challenging to you
during this cycle of performance management?
(cont.) Individual Comments

• Commitment from all the team members.
• obtaining data.
• Getting all team members to contribute to the
  tasks of identifying relevant data and analyzing
  it.
• The way the PMP was created in Excel with an
  entire page in one row of cells made it extremely
  time consuming tedious to update.
• Switching the rules in the middle of the game!
  We went from PMP, to score card, then to the dash
  board. These aberrations made staff wonder what
  value all of this has to our the ultimate duty to
  provide the service to the Institutes.
• Not having discussions about the performance of
  service groups/discrete services in staff
  meetings.
• Lack of a program area manager who accepts PM
  plans and manage their implementation.

116
What obstacles were most challenging to you
during this cycle of performance management?
(cont.) Individual Comments

• Concentration on the Measures Reporting Form
  throughout the year caused the PMP to be
  neglected. At the end of the cycle, it took a
  lot of time to update the PMP to match the
  Measures Reporting Form.
• The short notice and scheduling a report due for
  the week of January 17 was inane!

117
How can we help integrate PM into your daily
business activities? Individual Comments

• We have always been collecting and analyzing data
  to improve our program--no value added from this
  formal process.
• In my opinion performance management might be
  applicable for some businesses in private
  industry but not for the US Gov,t.
• It already is, and was before I was tasked with
  being a part of this initiative.
• Work to establish timelines so that we see the
  year at a glance. Several times my consultant
  worked with me to make deadlines at the last
  minute.
• Implement the ORF design review and inspection
  processes
• Already done.
• Have the ORS Office of Quality Management (OQM)
  staff do the work. In many agencies and in
  private sector companies this function is not put
  on the backs of the line staff but is performed
  by either consultants or an office such as OQM.

118
How can we help integrate PM into your daily
business activities? (cont.) Individual Comments

• Need help in learning how to distribute data
  collection throughout the division.
• Follow up with Quarterly team meetings.
• Continue emphasis on the importance of
  performance management to efficient 7 effective
  business operations.
• Need help collecting observation type data.
• Need help interpreting data.