Karlene Roberts, PhD

1
CPMR/USRCycle 1 Phase II Progress Review
.

• Karlene Roberts, PhD
• Haas School of Business
• UC Berkeley
• Anthony Ciavarelli, Ed.D
• Naval Postgraduate School
• Human Factors Associates, Inc.

2
AGENDA

• High-Reliability Organizations (HRO)
• HR0 Survey Development and Validation
• Examples of Survey Analysis Aerospace
  Benchmarking
• Key Accomplishments
• Problem Areas Encountered
• NASA HRO-Safety Climate Web Site
• Future Plans and Schedule

3
(No Transcript)
4
Columbia Accident InvestigationExcerpts from
the CAIB Report

• In the boards view, NASAs organizational
  culture and structure had as much to do with the
  accident as the External Tank Foam (CAIB, p.177)
• The Board determined that high-reliability theory
  is extremely useful in describing the culture
  that should exist in the human space flight
  organization (CAIB, p.181)
• CAIB discusses differences between the U.S. Navy
  and NASA in terms of safety culture and operation
  as an HRO. CAIB concluded that NASA could
  substantially benefit by following the Navys
  example and best practices (New York Times, July
  21, 2003)

5
Other Aerospace Accidents

• Genesis vehicle slammed into Utah desert,
  probably because aerospace engineers installed
  four small switches backward
• Climate orbiter crashed into Mars because the
  aerospace contractor used English measurement
  while NASA used metric
• Mars polar vehicle crashed when descent rockets
  shut off prematurely
• Television infrared observation satellite (TIROS)
  fell off its transport stand because an adapter
  plate was not properly secured (procedures not
  followed)

6
What Are Common Root Causes?

• Poor risk perception and threat recognition
• Lack of leadership commitment to risk management
• Inadequate management oversight and control
• Inadequate or unclear risk-decision criteria
• Difficulty in reporting at-risk
  decisions/behavior
• Policy and incentives reward excessive risk
  taking
• Culture does not support desired
  attitudes/behavior

7
Characteristics of an HRO(just a few of many
ideas about HRO)

• Leadership commitment to safe operations (failure
  conscious)
• Accurate perception of risk and risk mitigating
  processes (sense-making)
• A culture of trust and shared values
  (transmission of values-beliefs)
• Risk mitigating communication processes with open
  discussion of errors and corrective procedures
• .
• Distributed decision making, where the buck
  stops everywhere.
• Operating standards, with built-in redundancies,
  and continuous training
• Reward and discipline to reinforce safe behavior

8
Organizational Culture

• Shared Values (What is important) and Beliefs
  (How things work) that interact with an
  organizations structures and control systems to
  produce Behavioral Norms (The way things work
  around here).

9
Safety Culture Components

• Shared values about what is safe and unsafe
• Common beliefs about how to conduct safe
  operations
• Behavioral norms that govern risk-taking,
  everyday procedures and precautions
• Transmission of values, beliefs and accepted
  practices to others.

10
Five Element HRO-Safety Climate Model

• Process auditing - a system of on-going checks to
  monitor hazardous conditions
• Reward system expected social compensation or
  disciplinary action to reinforce or correct
  behavior
• Quality assurance policies and procedures that
  promote high quality performance
• Risk management how the organization perceives
  risk and takes corrective action
• Leadership and Supervision policies,
  procedures, and communication processes used to
  mitigate risk

11
Validation based on 10 years of US Navy
Research(Demonstrated Measurement Reliability
and Validity)
12
Summary of Relationship

• Units with survey average scores one sigma less
  than average, the risk of mishap increases 28
• Units with survey average scores two sigma less
  than average, the risk of mishap increases 65
• Units with survey average scores three sigma less
  than average, the risk of mishap increases by a
  factor of two(2)
• Units with survey average scores one sigma above
  average, the risk of mishap decreases 22
• Units with survey average scores two sigmas
  above average, the risk of mishap decreases 39

13
HRO Survey Development Process

• Meet with key staff to discuss risks and
  conditions of clients mission or business
  operations.
• Interview sample of management and operational
  personnel.
• Tailor baseline survey items to fit clients
  business line and the local situation.
• Following client approval, web-based version is
  created and web site constructed.
• Beta Test on a selected (small) sample. (WE ARE
  HERE)
• Large-scale data collection and performance
  feedback on scheduled roll out.

14
SAMPLE DATA FROM AEROSPACE SECTOR

• Summary Statistics for Selected Ratings
• Selected Comments on Prevailing Hazards
• Selected Comments on the Survey

15
Quick Look Analysis Survey Items Items are on a
Likert Scale 1-5, running from Strongly Disagree
(1) to Strongly Agree (5)

• 21 Workers in my organization carefully assess
  hazards during planning of daily work operations.
• 24 My immediate supervisor can be relied upon
  to keep his/her word.
• 25 Our safety professionals are very
  influential in promoting safety.
• 27 Management puts safety at the same level of
  importance as production.
• 35 I am adequately trained to safely conduct
  all of my job assignments.

16
LM Results Selected Survey Items (2)Combined
Organizations (n 356)
17
35 I am adequately trainedI am adequately
trained to safely conduct all of my job
assignments.
Low Rating Example
Average Rating (1-5 Scale)
18
20 Would not hesitate to advise supervisorI
would not hesitate to advise my supervisor when I
believe that the schedule pressure might
compromise safety.
High Rating Example
Average Rating (1-5 Scale)
19
Hazards Identified Typical comments

• Overworked, lack of personnel pressured to work
  outside their job description
• Old worn out equipment
• Poor, out of date equipment. Not enough workers
• Lack of proper tools and equipment
• Old equipment. Under staffed
• Not enough people
• Employee fatigue, operator error, failed to read
  procedure
• Tripping hazards, heavy materials to lift
  (slippery surfaces)
• Fatigue, overwork, too much overtime
• Lifting injuries
• Long hours
• Rush to deadline, last minute changes, and
  improper equipment
• Ignorance about procedures
• Being rushed by time

20
Actions to improve safety Typical comments

• Hire more people
• Dont accept more jobs than we can handle
• More communication and understanding of command
  media
• Retire old equipment
• Update equipment. More people
• Slow down and take time to process proof prior to
  starting work (better) planning and communication
• Mini huddles, use safety checklist
• Monthly safety inspections
• Communication, communication, communication
• Talk about safety at every meeting
• Better planning and scheduling
• Slow down
• Maintain safety focus
• Closer supervision

21
Further safety/mission success Typical Comments

• Need to schedule workload more evenly
• Management is dedicated toward a safe work place
• Add portable defibrillators
• We need to have more people
• Provide anonymous line of reporting for safety
• Work space is inadequate
• Follow through with promises
• Too many conflicting initiatives
• Follow procedures
• Make safety (ESH) more visible at the job location

22
Improvements to survey Typical comments

• The survey is great, it will be interesting to
  see how we use this tool to better ourselves
• Questions asked do not extract the most effective
  answers. (from the workers perspective).
• Keep questions worded so that they are less
  complex.
• Survey was fine.
• Survey looks pretty good as is.
• Survey is too general.
• The survey was good.
• Make fewer questions.
• Add items about completing goals
• Clearly define the term supervisor
• Make it shorter.
• It was an easy survey to take.
• (some confused that the survey targeted only
  Denver)
• I dont believe this is really anonymous
• Survey is too long
• Numerous questions do not relate to my work area
  (desk job)

23
KEY ACCOMPLISHMENTS

• Research protocols (interviews and survey
  construction)
• Published papers and presentations (see handout)
• Graduate Student Involvement
• Related work in Aerospace and Medicine
• Scheduled Beta test and Survey roll out

24
HRO Survey Demonstration NASA Web Site
OSES_PHI_screens.ppt
Copy link and paste to Browser https//65.104.11
9.152/nasa/login.html
25
NEXT STEPS

• Launch HRO-Survey at Goddard Space
• Provide Diagnostic Analysis and Feedback
• Focus Intervention Strategy on Feedback

26
Survey Serves as Diagnostic tool

• Points to areas where problems exist at the
  individual, team and organization level
• Identifies units who respond outside the norm to
  particular survey items
• Initial survey serves as a performance baseline
  and benchmark for comparison following
  intervention

27
Intervention Approach

• The Six Boxes methodology, developed by
  Binder-Riha Associates, is designed to improve
  organization performance, effectively manage
  organizational change, and to develop an HRO
  culture
• Based on research-based Behavior Engineering
  Model, it is a plain English classification of
  all factors that can influence behaviors which
  ultimately produce organizational results.
• Teach Six Boxes to all involved
• Common model and language
• Enables organization alignment leaders,
  managers, supervisors, individual contributors
• Toolbox for performance problem solving and
  continuous improvement

28
FUTURE PLANS AND SCHEDULE

• NASA Goddard Survey Application (July 2006)
• NASA Goddard Intervention Program (October
  December 2006)
• NASA Expansion of Survey and Intervention
  process to other NASA organizations (2007)

29
The Six Boxes Model