Case Studies in Prediction and Prevention of Failures

1
Case Studies in Prediction and Prevention of
Failures

• Chandra X-ray Observatory Propulsion Subsystem
  Anomalies
• Sabina Bucher

2
Purpose

• Share Chandra program Propulsion anomaly
  experience
• Trace two anomalies through
• Initial Anomaly detection
• Analysis techniques
• Mitigation strategies
• Implementation of mitigating actions
• Review Best Practices and Lessons Learned

3
Background

• Chandra is a Space based telescope
• Maneuvers several times a day
• One side always faces sun
• Passive thermal controls degrading
• Changing sun angles cause large temperature
  variations on some components

4
Case 1 Reduction in Thrust

• The Momentum Unloading Propulsion Subsystem
  (MUPS) thrusters are used to unload accumulated
  angular momentum
• An operational unload took 60 longer to complete
  than expected
• Attributed to reduced thrust from one of the
  sun-side thrusters
• Pulse-by-pulse performance of individual
  thrusters assessed
• Contributing factors identified
• Scheduling of operational momentum unloads
  modified
• Scheduling modifications have prevented
  reoccurrence of the anomaly

5
Momentum Management

• Reaction Wheels provide maneuver capability and
  pointing control for Chandra
• Reaction wheels store angular momentum
  accumulated from solar wind and gravity gradient
  effects
• Propulsion subsystem unloads accumulated angular
  momentum
• All unloads planned and executed by stored
  command sequence
• Stored angular momentum tracked and propagated
  carefully
• An unload is planned whenever the stored momentum
  is predicted to exceed the operation threshold

6
Calibration of Thruster Performance

• Derivation
• Choose 12 unloads
• All axes well represented
• All thrusters well represented
• Perform least square fit of on-time to delta
  momentum telemetry

• Uses
• Provide accurate ground based modeling of
  momentum unloads
• Invaluable in monitoring and measuring thruster
  performance

7
Observed Reduction in Thrust
Nominal unloads achieve near linear transition
from one state to the next
Nominal Unload

• Anomaly best modeled by cutting the thrust
  provided by one of the thrusters to 55 of its
  nominal value 360 seconds into the unload

8
Pulse-by-Pulse Performance
9
Identifying Contributing Factors

• Thruster efficiency calculation used to process
  all momentum unloads
• Six exhibited the anomaly signature
• Three showed indications of the anomaly
• Anomalous unloads searched for common traits
• Temperature and unload duration identified
• Modeling performed at the factory supported these
  observations

10
Defining and Implementing Scheduling Constraint

• New Constraint do not perform momentum unloads
  with duration over 600 s or starting temperatures
  over 120º F
• Limiting Durations
• Choose the unload target to shorten the unload
  duration
• Use deadman to cut off any unload that runs long
• Limiting Temperatures
• Required a model that could predict thruster
  temperature
• Developed an empirical model
• Incorporated model into the Mission Planning
  suite of tools

11
Monitoring Thruster Performance

• Every unload since anomaly detection has been
  checked with the thruster efficiency calculation
• No additional occurrences have been found
• Implementing such a technique on other programs
• Several weeks of up-front effort
• Now takes less than 30 s to run
• Useful in trending thruster performance
• Can highlight performance changes indicative of
  impending failure

12
Case 2 Cold Sun-Side Feedlines

• Caution low limit violations on two sun-side
  propulsion line thermistors
• Brief
• Infrequent
• Always on attitudes that put the sun on the tail
  of the vehicle
• Trending data dominated by solar heating
• Heater cycle turn on temperatures isolated
• Revealed steady, mission long cooling trend
• Feedline temperature profiles characterized with
  respect to time and attitude
• Limited duration of dwells at orientations
  requiring the heaters
• Cooling trend successfully halted

13
Propulsion Thermal Protection

• Propulsion lines on Chandra are spiral wrapped in
  multi- layer insulation (MLI), heaters, and
  aluminum tape
• Heaters controlled by bi-metallic thermostats
• Cannot be commanded on
• Cannot be re-programmed
• Every set of thermostats controls a circuit of
  heaters
• Temperature telemetry provided by thermistors
  located at various points along the lines
• The propulsion subsystem wraps around the front
  of the spacecraft bus
• Attitudes that put the sun at the tail of the
  vehicle (tail-sun) put the propulsion components
  into shadow

14
Cold Temperatures on Sun Side Feedlines
Sun
Limit Violations on Thermistors B and C

• Brief and Infrequent
• Always at tail-sun attitudes
• Heaters turning on late
• Once on, heaters functioning well

Attitude Dependence

• To keep other units cool, time spent at tail-sun
  attitudes increasing
• Thermistors B and C well removed from the
  thermostats
• On orbit changes have caused the thermostats to
  stay warm longer than remote sections of line
  once in shadow
• Allows portions of the propulsion lines to be
  exposed to cold temperatures

15
Isolating Heater Cycles

• Mission long data set split into attitude by
  attitude segments
• Temperatures of each segment analyzed
  independently
• Calculated rate of temperature change for each
  attitude
• Makes heater turn-on obvious
• Used to collect statistics on heater cycles
• Revealed mission long cooling trend
• Attitude-by-attitude telemetry showed Thermistors
  A and B behaved differently

16
Temperature Behavior
17
Defining and Implementing a Scheduling Constraint

• Thermistor A constraint do not schedule
  attitudes past 170 deg sun-pitch
• Small operational impact
• Easily implemented
• Thermistor B constraint do not schedule
  attitudes past 150 deg sun-pitch
• Eliminates attitudes used to cool other
  spacecraft components
• Makes some time-constrained science observations
  impossible
• Constraint needed more balance
• Keep propulsion lines safely above freezing,
• Do no eliminate tail-sun attitudes
• Thermal model considered
• Data too sparse
• Too many factors contributing to the final
  temperature of the lines
• Required high degree of accuracy

18
Determining Maximal Cooling Rates
Cooling envelope establishes the maximum cooling
rate from one temperature to another
Maximal cooling rates used to set do not exceed
limits on time at cold attitudes
19
Implementing Scheduling Constraint
New Constraint

• Preheat lines before maneuvering to a cold
  attitude
• Minimum preheating times set with plots of
  temperature vs. elapsed time at attitude
• Implementation without software very labor
  intensive
• Additions to existing software tools made
  implementation manageable

Plot sun angle vs. time
Show transitions into and out of propulsion line
regions
Issue red warning if 1) pre-heating
requirement not met 2) maximum duration
exceeded
New Tool
20
Conclusions

• Successful mitigation of two Chandra Propulsion
  Anomalies
• Initial indications subtle
• In-depth analysis revealed distinct performance
  changes
• Adaptive Mission Scheduling process allowed
  successful mitigation
• Incorporate in-depth analysis methods into
  day-to-day operations
• Software and Hardware advancements making this
  increasingly feasible
• Identify problems before they become failures
• Mission Scheduling can evolve gracefully as the
  vehicle ages
• Provide the scheduler with all of the information
  that goes into scheduling
• Allow the scheduler to specify how requests are
  scheduled
• Use optimization routines aid, but not replace,
  the scheduler
• Chandras remarkable safety and efficiency record
  contributed to by
• An environment that fosters in-depth analysis
• An adaptive approach to mission scheduling