Flight Software

1
Section 9 Flight Software
. . . Ray Whitley EO-1 Flight Software Systems
2
Ray, why dont you just say that flight
software is never done, it just becomes
operational? John Azzolini
3
Agenda

• Flight Software Overview
• ACS/EFF CDH RSNs (ACE Comm HK PSE WARP
  X-Band) WARP
• Heritage Experience
• Verification Validation
• ACS/EFF CDH RSNs (ACE Comm HK PSE WARP
  X-Band) WARP
• Metrics Margins
• Special Topic FSW Configuration Management
• Control Process Procedures
• Tools
• Special Topic FSW Sustaining Engineering
• Team
• Maintenance Facility
• Conclusion

4
FSW Overview

• Mission Requirements
• 2 kbs r/t command up-link stream in CCSDS
• 2, 32, 1000, or 2000 kbs telemetry in CCSDS AOS
• Attitude Determination Control with
  Independent Safehold
• 24 hr autonomous operations
• Spacecraft subsystem safing
• Science data record playback
• Patch and/or partially re-program all processors
  on-orbit

5
FSW Overview
6
FSW Overview

• ACS/CDH Mongoose V
• Attitude determination control modes
• Real-time stored command distribution to all
  subsystems
• Telemetry acquisition and frame generation
• Time maintenance distribution
• SSR record and playback (S/C instrument
  housekeeping data)
• Enhanced Formation Flying Package
• Processor/software/ s-c subsystem health safety
  monitoring
• ACE RSN
• Sensor data acquisition
• Actuator commanding
• Independent safehold
• Comm RSN
• Command reception partial de-blocking
• Telemetry transmission
• Spacecraft/Ground time correlation data collection

7
FSW Overview (continued)

• HK RSN
• GPS time/position data acquisition
• Thermister data collection
• High Output Paraffins (HOPS) deployment control
  monitoring
• Miscellaneous LVPC services
• PSE RSN
• Power subsystem housekeeping data acquisition
  command distribution
• Power subsystem monitoring control
• WARP RSN
• Command distribution for LVPS services
• Housekeeping data acquisition monitoring

8
FSW Overview (continued)

• X-Band RSN
• Phased Array Antenna command distribution
• Phased Array Antenna housekeeping data collection
  monitoring
• WARP Mongoose V
• Science data record (set-up and control)
• Science data playback control for both X-band and
  S-band playbacks

9
Heritage Experience

• ACS/EFF
• Direct heritage from TRMM ACS same design with
  approximately 40 code re-use. (ie., Kalman
  Filter)
• The EFF technology code is an entirely new
  application however is derived from a ground
  version of the software
• Key personnel were members of TRMM ACS software
  development team
• J. DAgostino (13 years)
• K. Blackman (8 years)
• A. Hawkins (12 years)
• D. Kobe (14 years)

10
Heritage Experience

• CDH
• SAMPEX to XTE to TRMM to MAP The core CDH
  software (including the operating system,
  software bus, 1553/1773 Bus control, S/C time
  management distribution, command ingest
  distribution, telemetry acquisition
  transmission, and SSR management) was originally
  developed for SAMPEXs 386 S/C processor under
  the VRTX OS kernel. The transition to XTE/TRMM
  applied many lessons learned from an operational
  perspective. The TRMM version fixed a couple of
  bugs discovered during XTE IOC.
• MAP EO-1 software was developed by a joint team
  of civil servants and Litton personnel under a
  Space Act Agreement, where the primary task was
  to re-host proven software to a new processor
  (MV) with a new operating system (VxWorks). Key
  Personnel
• Ken Rehm (17 years) J. Marquart ( 20 years)
• G. Smith (18 years) T. Miller (14 years)
• L. Bashar (8 years) A. Cudmore (11 years)
• E. Stagmer (7 years) M. Bartholomew (12 years)

11
CDH Software Heritage
12
Heritage Experience

• RSNs
• Code Baseline builds received from MAP for Comm,
  HK, and PSE in Dec. 98. These deliveries
  included a common OS kernel developed and tested
  under a previous GSFC contract and a set of
  generic CDH services including 1773 RT, S/C time
  processing, and memory load/dump routines.
• The ACE, WARP, and X-band software applications
  are entirely EO-1 specific. That is, there was
  no associated MAP baseline.
• J. Marquart (20 years) ROS, GRSN applications
• T. Miller (14 years), Comm RSN applications
• M. Maldonado (16 years) WARP RSN applications
• Steve Mann (14 years) ACE RSN applications
• G. Smith (18 years) HK RSN applications
• C. Xenophontas (6 years) PSE applications Xband
  applications

13
Heritage Experience

• WARP
• WARP MV software shares several software
  components with the CDH. These include the OS
  layers of VxWorks and the software bus, processor
  health safety, and memory monitor/load/dump
  services.
• The WARP specific applications includes science
  data recorder management services for record
  playback, and communications with the CDH as an
  1773 RT.
• The core development team for the WARP specific
  applications were responsible for the FSW
  associated with HSTs recently installed SSRs
• S. Slegel (8 years) Chris Xenophontas (6 years)
• D. Molock (5 years)
• R.J. McGraw (10 years)

14
Conclusion

• FSW Readiness
• All software systems have been tested and
  re-tested over the last 4-9 months with no
  mission threatening discrepancies revealed.
• Yes, changes have been made as the result of new
  operational requirements or to correct minor
  operational specifics. These would have probably
  been made on-orbit had launch not been delayed
• Residual Risk
• There is no residual risk in software due to the
  lack of verification validation
• Residual risk, if it exists, is in our ability to
  reproduce or simulate hardware anomalies in the
  components for which we have no ETU nor access to
  one (eg. HK Xband WARP).

15
Special Topic Verification Validation
. . . Ray Whitley EO-1 Flight Software Systems
16
Verification Validation (ACS/EFF)

• Software requirements design material presented
  at peer review subsystem level CDR in June 97
• Code walk-throughs conducted by ACS software team
  members and the subsystem lead on an as needed
  basis
• Extensive test suite developed and maintained by
  dedicated subsystem test personnel for closed
  loop testing at both software lab and the
  spacecraft.
• Results
• Critical RFAs 0
• Critical PRs 2 (790-30-3 846-50-1) warm
  restart delay set too long in ACS task. Corrected
  start-up timing. Closed.
• Fully verified no residual risk

17
Verification Validation (CDH)

• The requirements design of the CDH software
  were presented at every major EO-1 project-level
  review
• Since the code was originally received from MAP,
  no formal code walk-throughs was performed for
  EO-1 however, EO-1 software engineers attended
  all code walk-throughs hosted by the MAP software
  team
• A well documented and repeatable set of STOL test
  procedures was received augmented with EO-1
  specifics. Tests were originally developed
  against requirements-to-test matrices generated
  from software specifications at the component
  level
• Set of 19 automated STOL test procedures are
  executed in the software facility prior to each
  build delivery to the spacecraft and again
  repeated upon delivery to spacecraft (regression
  test philosophy)

18
Verification Validation (CDH) (continued)

• Special Tests developed for execution at the
  spacecraft
• 1773 bus error injection independent S/C timing
  test processor stress tests
• Independent test developed at spacecraft IT by
  subsystem leads and/or spacecraft test conductors
  as part of box-level function and/or CPTs
• Major Results
• RFAs 5
• 3 are special topics during this review
• (18.06) verify TSMs from FOR
• (4.17) Maint. Plan from Mission CDR
• (17.19) Regression tests from MOR
• 2 design change considerations
• (4.19) consider re-booting from DRAM or EEPROM
  from Mission CDR
• (4.12) consider re-booting around errors in DRAM
  from Mission CDR

19
Verification Validation (CDH) (continued)

• Responses to Design Altering RFAs
• (4.12) Consider including capability to check
  DRAM from EEPROM before downloading executable
  code
• The old code 735 made every effort to remove
  dependencies upon a given area of DRAM during the
  boot process however, there are still some
  dependencies remaining due to the architecture of
  the R3000 processor (ie., the fixed location of
  exception handlers).

20
Verification Validation (CDH) (continued)

• Responses to Design Altering RFAs (continued)
• (4.19) Consider rebooting from protected EEPROM
  or writeable EEPROM using hardware discrete
• When this RFA was written, there was no
  independent safemode (in the ACE RSN) baseline
  one now exists
• The current MV design will boot from protected
  EEPROM or writeable EEPROM based on the setting
  of a special command sent from the ground
  however, for the power-on reset condition, the
  hardware always defaults to the protected
  version. The two versions of code are downloaded
  into different areas of RAM for execution
  however, they share common areas for exception
  handlers (hard-wired) and the mode transition
  log.
• The Project decided that the EO-1 and MAP
  baseline MV boards should be as close to
  identical as possible to mitigate any future risk
  to EO-1.

21
Verification Validation (CDH) (continued)

• Results (continued)
• Critical PRs 14 all are currently closed except
  where noted

22
Verification Validation (CDH) (continued)

• 674-20-26 674-20-44
• Found to be non-critical problem with command
  ingest reporting partial command receipts after
  a commanded MV warm restart. Code was not
  flushing data from Comm RSN before searching for
  new commands. Problem fixed and re-tested. No
  residual risk.
• 768-20-3
• Time code software logic error discovered
  screening GPS data. Problem corrected
  re-tested. There is no residual risk here.
• 798-20-2
• Result of special time epoch test. This
  particular error event message is to be expected
  for duration of the test no further action
  required
• 678-20-4 795-20-1
• Discovered and corrected GPS data synchronization
  time roll-over problems in the HK RSN when
  using GPS time corrections and the MV is
  commanded to re-boot ( temporary loss of time
  broadcast) to all RSNs.
• Time roll-over protection added to HK software
  re-tested. PRs are now closed with no residual
  risk.

23
Verification Validation (CDH) (continued)

• 674-20-10
• Telemetry drop-outs. This was found to be a
  ground procedural error. The S/C was
  transmitting 2kbs however, the software was
  configured for 32kbs of real housekeeping data.
  Procedure corrected, operational constraint
  documented.
• 674-20-38
• PR written during the initial execution of the
  MV Stress Test procedure whose purpose is to
  verify task priority setting within the system.
• As CPU utilization increases, system performance
  should gracefully degrade and report this
  degradation ( low priority task CPU starvation)
  in the form of error event messages and error
  counters in telemetry.
• Analysis of test data reveals that the system
  performed as required. No further action is
  required.
• 679-20-7
• PR written because TC could determine if the
  re-transmit telemetry transfer frame segment
  command was working properly. Data analysis or
  the original test and re-execution by FSW
  personnel determined that the command executes
  correctly. No further action required, and no
  residual risk.

24
Verification Validation (CDH) (continued)

• 943-40-1
• During a recent Mission Sim, FOT personnel
  attempted to fine tune the down link frequency
  of specific packets and discovered that the
  command to accomplish this fails for ap-ids
  greater than a specific value.
• Code inspection determined that an ap-id limit
  validation check in the software is set too low,
  (probably the MAP limit)
• Operational work-around is to perform partial
  table load to get the new values in however, we
  decided to develop a EEPROM code patch as an
  exercise to correct the problem. No residual
  risk anticipated.
• 794-20-6
• FPU exception in ACS task normalization routine.
  Problem corrected and re-tested at software lab
  and the S/C. There is no residual risk in this
  area.

25
Verification Validation (CDH) (continued)

• 876-20-1 876-20-2 876-20-3
• Three separate occurrences of the same problem.
  The 1773 Bus Controller software would began to
  continuously crash when 1773 fiber optic cables
  were cabled incorrectly for the VirtualSat. At
  least one of the cables was connected to an
  unused bus monitor. Although VirtualSat is not a
  flight component, it contains a Mil-Std 1553/1773
  compliant protocol chip.
• Problem deemed critical because of loss of S/C
  communications.
• Problem was reproduced in S/W lab and discovered
  to be centered around a single word message from
  the MVs BC to the Vsat RT. It was determined
  that increasing this message size to 2 words
  eliminated the problem however we dont
  understand why it works.
• The word message has been promoted to the
  spacecraft
• Additional test and analysis is pending. There
  is low residual risk here

26
Verification Validation (ACE RSN)

• Code walk-throughs with panel of FSW peers, and
  subsystem lead on an as needed basis
• ACE software is thoroughly regression tested at
  the spacecraft immediately after each delivery by
  dedicated ACS subsystem team member
• Results
• Critical RFAs 0
• Critical PRs 1 (790-30-5) ACE SH inhibited due
  to diagnostic test code. Test code removed.
  Closed.
• Fully verified no residual risk

27
Verification Validation (Comm RSN)

• Comm RSN detail requirements design reviews
  conducted early in MAP Program development
• Well documented test procedures developed by MAP
  team and delivered with code to EO-1 engineers
  for regression testing
• Code walk-through with panel of FSW peers,
  subsystem lead, S/C systems engineering (all
  under MAP)
• Results
• Critical RFAs 0
• Critical PRs 1 (684-20-17) Defaulted to MAP
  encoding instead of EO-1s. Corrected default
  start-up settings and re-tested. Closed.
• Fully verified no residual risk

28
Verification Validation (HK RSN)

• An extensive code walk-through was conducted over
  3 days in October 99, resulting in approved 33
  w/t RFAs, including a major re-design
  recommendation to accommodate GPS operation.
  These were implemented reviewed by mid-November
  99
• Box-level engineer tasks to develop comprehensive
  test suite to verify validate all software
  requirements at the spacecraft independently of
  assigned software engineer.
• Results
• Critical RFAs 0
• Critical PRs 20 majority are concerned with
  failures to resume the correct operational
  posture after a hk rsn processor re-boot has
  occurred. All 20 are now closed!

29
Verification Validation (HK RSN) (continued)

• Results (continued)

30
Verification Validation (HK RSN) (continued)

• 767-20-2 779-20-1 790-30-4 800-20-1 807-20-1
  808-20-2 826-20-2 835-20-4
• All are related to the HK RSN softwares warm
  restart requirements.
• Warm restart for an RSN means that key
  processing parameters are check-pointed
  periodically to a non-volatile area of RAM. If
  the processor then suffers an upset, upon
  re-booting these parameters are restored used to
  continue with operations.
• The software lab does not support thorough
  testing of HK RSN software prior to S/C delivery.
• Most were written during a 3 day period while
  trying to verify a new HK load to the S/C
  resulting from a previously held code
  walk-through. The code w/t had revealed that
  warm restart had never been tested. The
  assigned box-level engineer developed a more
  comprehensive test as a result of w/t
  recommendations. A major change in the HK
  functional test was that HK processor warm
  restarts were inserted in each step of the
  launch sequence to verify that the HK RSN would
  resume the sequence correctly.
• All warm restart requirements have now been
  verified and validated at the spacecraft. There
  is no residual risk in this area.

31
Verification Validation (HK RSN) (continued)

• 801-20-2 806-20-1 806-20-3 806-20-4 808-20-1
  808-20-3809-20-1
• This series of PRs were written against various
  commands and timed functions that did not operate
  as required or desired. All functions were
  updated and re-tested by the subsystem lead at
  the S/C.
• There is no known residual risk in code however,
  HK software specification document updates are
  still pending.
• 835-20-1
• Code inspection revealed coding error in
  operation of the DE01DISALL command. Code
  corrected and re-tested at S/C. No residual risk
• 811-20-2 811-20-3
• Two manifestations of the same problem. HK
  software contained an error in the handling of
  circular buffers used to store telemetry data
  prior to sending the data to the MV. Coding
  error fixed and verified at S/C during regression
  testing. No residual risk remains.
• 679-20-1
• Telemetry point not being stored in packet.
  Corrected and verified at S/C.

32
Verification Validation (PSE RSN)

• An extensive code walk-through was conducted over
  4 days in August 99, resulting in approved 19
  w/t RFAs. These were implemented in early
  September 99
• Box-level engineer tasked to augment
  comprehensive test suite to verify validate all
  software requirements at the spacecraft
  independently of assigned software engineer
• Results
• Formal RFAs 0
• PRs 2 (673-40-2 678-20-1) Both were coding
  errors for PSE command interpretations. The
  errors were corrected and re-verified at the S/C,
  and there is no residual risk.

33
Verification Validation (WARP RSN)

• Given the simplicity of the application, only a
  limited code walk-through was conducted with
  other RSN software engineers to verify use of
  operating system services
• The software developer developed, executed, and
  currently maintains (if necessary) a
  comprehensive software acceptance test suite
• Software is independently verified by numerous
  execution of box-level functional and CPT
  procedures at the spacecraft
• Results
• Critical RFAs 0
• PRs 0

34
Verification Validation (X-Band RSN)

• Peer-level requirements and code walk-through
  reviews were performed internal to the CDH
  software team
• Verification and validation is achieved only
  through successful execution of box-level tests
  at the spacecraft
• Results
• Critical RFAs 0
• Critical PRs 3 (676-20-5 679-20-10
  681-20-17) PAA temperature limit inappropriately
  set. Limits widened. Re-tested. All are now
  closed!
• Fully verified no residual risk

35
Verification Validation (WARP)

• WARP software requirements detail design were
  reviewed during several formal box-level reviews
  in 98
• Code walk-throughs of the WARP-specific
  applications were waived due to demonstrated
  successful operation of the software during
  box-level IT
• An independent acceptance test team was
  contracted between Oct 98 March 99 to test
  the WARP MV software. The team consisted of
  engineers associated with MAP s/w testing and
  on-orbit maintenance of other missions. R-to-T
  matrices were developed from previously
  signed-off requirements specifications users
  guides.
• Tests executed both prior and after initial
  delivery to the spacecraft
• Special re-programmability tests included
• Results
• Critical RFAs 0
• Critical PRs 0

36
Metrics

• Problem Tracking Reports
• 927 problem reports have been recorded in the
  Hammers/Litton database against all non-WARP
  related flight software
• Discrepancy Reports
• The WARP team recorded 54 software related
  discrepancies through 5/99 in a separately
  maintained database. No discrepancies have been
  recorded since that time.
• All requiring S/W changes have been addressed and
  closed. Remainder are one-of-a-kinds or
  documentation related.

37
Margins
38
Operational Hours

• Processor Prior Versions Launch Version

• ACS/CDH/EFF Mongoose V 1813 274
• ACE RSN 1298 196
• Communications RSN 1813 274
• Housekeeping RSN 1813 274
• PSE RSN 1813 274
• WARP RSN N/A 523 59
• X-Band RSN 419 47
• WARP Mongoose V N/A 523 59

39
Major FSW Releases
40
FSW Configuration Management (Process)

• Discrepancy identified as result of level III
  testing or as result of S/C level test activity
  (level II) as a problem or as a new requirement
• Board convenes on as needed basis to access
  need for change
• Risk impact if not done possible operational
  work-arounds
• Technical risk assessment due to complexity of
  change
• Schedule resource requirements (including
  regression tests at the spacecraft)
• If change is approved, PTR or DR, is inserted in
  level III database.
• Developers make change and regression test at
  software lab
• Software lead prepares Version Description
  Document
• Lists all changes and optionally includes code
  listing differences (if requested)
• Software lead and/or subsystem lead prepares
  level II WOA, attaches VDD, and obtains required
  signatures (PM, Systems Engineering, QA)

41
FSW Configuration Management (Process)

• New release is installed and regression tested at
  the spacecraft with results reported in daily TC
  activity log and is also distributed
  electronically
• WOA package filed by spacecraft QA
• Level III Level II (PRs) close-out at
  subsequent meetings

42
FSW Configuration Management (Tools)

• Electronic, web-based problem reporting and
  tracking systems
• All non-WARP flight software maintained by the
  Hammers Corp
• To date, GSFC has maintained an equivalent system
  for WARP software
• Systems track reqs, design, code error, test,
  documentation issues
• For on-orbit maintenance, the Hammers system will
  be used exclusively
• Hammers utilizes version tracking software
  (Visual SourceSafe) for tracking source changes
  at the module level

43
Special Topic Failure Detection Correction
. . . Tom Feild EO-1 Systems Engineer, GSFC Code
568
44
Failure Detection and Correction

• EO-1 employs 2 methods to autonomously detect
  fault conditions and take appropriate action to
  safe the observatory
• 1) TSM/RTS System
• A Telemetry Statistics Monitor (TSM) is a
  telemetry point or combination of points that are
  used to verify observatory health
• A Relative Time Sequence (RTS) is a list of
  commands that can be executed in response to a
  hazardous condition
• TSMs monitor spacecraft health and when they
  detect a fault condition they can execute an RTS
  and/or issue an event message
• Runs on the ACDS Mongoose V
• 2) FDC System
• The Fault Detection and Correction (FDC) System
  consists of software checks and corrective
  actions that are embedded in the flight software
  running on the various processors on EO-1

45
Failure Detection Correction Philosophy

• Philosophy is Keep it Simple
• Only conditions that threaten observatory health
  are addressed
• Simplest safe action is taken, often to power
  off the device whose health is in question
• The TSM/RTS System is the primary method of
  detecting and correcting hazardous conditions on
  EO-1
• Except for certain instrument fault conditions,
  when one hazard has both a TSM/RTS action and an
  FDC action then the thresholds are set such that
  the TSM/RTS will trip first
• All TSMs, RTSs, and FDCs will be tested prior to
  launch
• Many ACS FDCs can only be tested on the high
  fidelity simulator and many instrument FDCs only
  tested prior to instrument delivery
• All have been verified except updates to
  TSMs/RTSs as noted
• TSMs/RTSs and FDCs are retested after every new
  software load. Many are tested during the CPT.

46
Event Messages

• The Observatory sends down Event Messages to
  provide information on all significant events
  that occur on the observatory such as
• Changes in spacecraft configuration (e.g. data
  rates, filter tables, day/night transitions)
• Anomalous events such as activated TSMs, RTSs,
  and FDCs
• All Event Messages are stored in VR-2
• VR-2 is downloaded during each pass
• Provides Flight Ops Team with immediate access to
  information on spacecraft health and autonomous
  actions
• Event messages are generated by
• ALL TSM activations
• All RTS calls by TSMs
• Most FDCs

47
TSMs

• Current version is TSM Rev I
• Contains 33 TSMs (plus 17 monitor point TSMs)
• Loaded on March 10, 2000
• There was one change to the TSMs for Rev I
• TSM 67 (S/C pointing at sun) now inactive at
  power-up, trips at 18 and not 30 degrees, and
  calls a new RTS (12) to safe the instruments
• Two functions called by TSMs were updated
• TSM 215 Now trips at battery state-of-charge of
  80 (was 0.8)
• TSM 75 Now trips when arrays are 20 deployed
  (was 95)
• All TSMs have been verified
• WARP and instrument TSMs will be reverified after
  they are reintegrated

48
(PROM) RTSs

• There are 256 RTSs
• 1 to 64 are loaded in PROM and EEPROM
• 65 to 256 are RAM only
• 40 to 55 are reserved for the MOC (Not used for
  fault correction)
• 1 to 39 and 56 to 64 are used for Fault
  Correction (48 total)
• 35 are currently used 13 are spares
• 5 and 39 are also used by the MOC. They may
  use others.
• Current EEPROM version is RTS Rev J
• RTS 12 needs to be reloaded Rest loaded on
  March 10, 2000
• RTS 12 is new Significant changes to RTS 6
  Minor changes to RTSs 33 and 38
• RTS Rev J has not been tested following the
  latest MV load

49
FDCs

• Additional Failure Detection and Correction is
  found in the various processors on EO-1
• These are called Fault Detection and Correction
  (FDC) routines
• FDCs contained in ACDS, ACE RSN, PSE RSN, Comm
  RSN, X-Band RSN, WARP MV, Hyperion, ALI, and AC
• Most are enabled at power-up and during nominal
  mission mode
• The ACS FDCs are mode dependent and not all are
  used in any one mode
• Most can be individually enabled or disabled
• Some instrument FDCs cannot be disabled
• Several are disabled for launch

50
FDCs (continued)

• All have been tested
• Many of the ACS FDCs can only be tested on the
  EO-1 hyper-dynamic simulator
• Most of the Instrument FDCs were tested before
  integration onto EO-1
• The rest have been tested on the observatory
• When a new software load is performed, all FDCs
  in that software load are retested
• ACS FDCs are tested on the EO-1 hyperdynamic
  simulator

51
Use By Flight Ops Team

• FDCs
• Autonomous. Can not be called directly by Flight
  Ops
• Updates require new software load
• Most can be enabled and disabled
• TSMs
• Autonomous. Cannot be tripped directly by Flight
  Ops
• Updates require table load or new MV load
• Can be enabled or disabled or placed in watch
  mode
• RTSs
• EEPROM RTSs 40 to 55 and all RAM RTSs (65 to 256)
  are reserved for the MOC. RTSs 5 and 39 are also
  used by the MOC
• All RTSs are run from RAM and can be overwritten
  (updated) by the MOC. EEPROM can be updated by
  table load. PROM cannot be updated from the
  ground
• All RTSs can be enabled or disabled

52
Critical PRs - TSMs

• 681-20-2 TSM 205 kept trying to turn off the
  ALI
• 784-40-2 TSM 116 tripped when ALI was powered
  on
• 828-20-2 RTS 234 trips immediately
• 784-30-2 TSM 234 tripped unexpectedly
• 784-40-8 TSM 233 did not trip and call RTS 6
  as expected
• There was a problem with the data value masks
  used by these TSMs. All these TSMs have been
  updated and verified
• 858-30-1 TSM 75 did not trip with HOP 1 ON (did
  not trip at 20)
• The function that this TSM calls has been changed
  to trip if the Solar Array was 20 deployed
  instead of 95. Has been updated and verified.
• 858-30-4 TSM 215 did not trip when expected
• The function that this TSM calls was updated to
  trip at 80 battery state-of-charge instead of
  0.8. Has been updated and verified

53
Critical PRs - TSMs and RTSs

• 785-20-4 RTS 9, 11 and 27 were started during
  power-up
• 795-20-2 RTS 9 executed during a MV reboot
• Caused by a problem with how timing TSMs were
  handled following a time jam or power-up. The
  function called by the timing-based TSMs was
  updated and all associated TSMs have been
  verified.
• 846-50-3 WARP command sequence is incorrect in
  RTS 6
• Sequence has been updated and will be verified
  after WARP integration
• 723-20-7 Change RTS 47 to not turn off the
  X-Band RSN
• 770-30-5 RTS 47 shouldnt turn off the S-Band
  transmitter
• 846-50-4 RTS 13 turned the X-band off (Should
  call RTS 6)
• 818-20-5 RTS 34 should cancel Solar Array
  override commands
• 785-20-6 RTS 36 should activate TSM 63 earlier
  in the RTS
• The RTSs were updated to accomplish the
  identified actions. All have been verified
  except RTS 6, which cannot be tested w/o the WARP.

54
Critical PRs - STOL Proc Errors

• 784-40-4, 784-40-5, 784-40-7 Commands in
  Test_TSMs too long
• 846-90-1 Test_TSMs used incorrect mnemonic
• 684-20-16 Error during RTS commit due to dump
  at wrong filter rate
• 937-40-1 Command to enable TSM 53 failed
  (wrong command sent)
• These PRs were due to errors in STOL procedures,
  not problems with the TSMs or flight software.
  The ground procedures have been updated
  verified.
• These can be reclassified as noncritical PRs

55
Status

• No open issues exist with EO-1 Failure Detection
  and Correction system
• All changes to the TSMs (Rev I) have been
  verified
• Several unchanged TSMs await WARP instrument
  reintegration for regression testing
• 4 new or modified RTSs (Rev J) need to be
  verified
• All RTSs require regression testing
• After this testing has been completed, the EO-1
  Failure Detection and Correction System will be
  ready for launch
• If software changes are made then re-testing will
  be performed as appropriate, but no changes are
  planned
• Final testing will effectively retire residual
  risk

56
Special Topic Flight Software Sustaining
Engineering
. . . Ray Whitley EO-1 Flight Software Systems
57
FSW Sustaining Engineering

• Facility
• Moving flight software development facility
  components from Swales and GSFC/WARP areas
• Maintenance facility to be augmented with key ACE
  ETU components
• Agreement with MAP to use their PSE ETU test bed
• will have priority for emergencies, otherwise
  must schedule
• Personnel
• Key contractor personnel associated with the
  development S/C level IT will perform
  maintenance
• Personnel participated in recent code
  walk-throughs of 2 mission critical software
  components
• Re-verified software load/dump/verify utilities
  prior to S/C launch
• Have gathered, reviewed, and configured all
  software specification user documentation

58
FSW Sustaining EngineeringFunctional
Organization Chart
59
FSW Sustaining Engineering FSW Facility
60
Special Topic A/D Conversion Anomalies
. . . Mark Perry Swales Aerospace, Inc.
61
Overview of Anomalies

• Two types of anomalies on A/D telemetry
• Conversion errors on passive telemetry that use
  1-milliamp source in the ESN (the same multi-chip
  module that contains the A/D converter)
• Correction is not required
• Single bit-flip or bucket errors
• Modified RSN S/W to eliminate most of the
  anomalies
• Still see some low-amplitude, but unexplained
  noise
• Neither is of operational significance to the
  mission
• Similar anomalies also seen on MAP A/D
  conversions
• This presentation addresses the following items
• Special Topic 9d, and RFA 13.02
• PRs on ACE noise 674-20-23, 677-20-2, 681-20-10,
  681-20-11, 682-20-2, 923-20-10

62
Conversion Errors

• Only seen on some passive telemetry (uses the 1mA
  current source)
• Telemetry value gets stuck, and does not change
  as expected
• During successive approximation, A/D converter
  makes an error, which pegs all remaining
  (less-significant) bits they are all high or all
  low
• The digitized value seems to stick at a
  particular value, becoming accurate only after a
  threshold passes
• To date, only EO-1 evidence is in RCS tank
  temperature
• Caused by instabilities in the current source,
  and the successive-approximation conversion
  process in the MCM
• The conversion process loads the 15-volt supply
  used by the converter
• MAP engineers empirically developed a software
  work-around that involves multiple conversion
  commands and waits
• EO-1 has not changed S/W to eliminate this
  anomaly
• The errors are not of sufficient severity to
  justify the change
• Errors are intermittent and on non-critical
  telemetry temperatures that are not used for
  on-board control.
• S/W changes would be substantial, and could
  adversely affect timing
• Slight residual risk because a few aspects not
  understood
• Only some telemetry affected (not all telemetry
  fully investigated).
• Cannot explain why the S/W fix works.

63
Bit-Flip, Bucket, Other A/D Anomalies

• Characterized by LSB noise or single-cycle errors
  of 2, 4, 8, 16, or 32 counts on some telemetry.
  Was detected on ACE and HK RSN data.
• The noise distribution is not a normal
  distribution
• One source of the noise was insufficient delay
  before reading A/D data
• Detailed timing analysis produced correct S/W
  parameters
• All RSN S/W changed to incorporate correct delays
  for A/D read and write cycles
• Necessary to meet the specifications for the A/D
  converter
• Post-change tests conducted to confirm that
  performance of RSNs not affected by change in A/D
  S/W was particularly concerned about effects on
  timing.
• Benefits of change in RSN S/W
• COMM RSN and PSE RSN data was not noisy, and did
  not get noisier
• Reduced noise on ACS data and HK RSN data.

64
Example of Noise ReductionDue to S/W Change
65
Remaining Noise in A/D Conversions

• Some ACS telemetry adversely affected by the S/W
  change
• Shifts or drifts in telemetry (ex tank pressure
  and 15V tlm on next slide)
• Additional noise (ex 15V tlm on next slide)
• Amplitude of noise is not sufficient to affect
  performance, but there is some residual risk.
• Concern without completely understanding the
  cause, cannot confirm that aging effects such as
  radiation and thermal cycling will have a
  negligible effect.
• Also, noise may delay detection of anomaly by
  masking early effects
• There are several possible sources of the noise
• Successive approximate algorithm rather than
  sample and hold causes unusual noise if voltage
  varies
• Non-optimal board layout
• Analog grounding is not located near the A/D
  converter
• Signal routing may cause cross-talk
• Still investigating. Working with MAP engineers,
  who have better test capability using their ETUs.

66
Some Data Affected by S/W Change

• Shifts the maximum data bin changes from 3568 to
  3576
• Added noise

Before change in RSN S/W
After change in RSN S/W
67
Summary

• The A/D anomalies do not affect EO-1 health,
  safety, or operations
• Noise is below threshold for engineering impact
  to S/C
• Most analog data is not used on-board EO-1
• All FDC has persistence checks to ignore
  single-cycle errors
• Control algorithms using analog data have
  negligible reaction to small, single-cycle
  errors.
• Critical EO-1 ACS data is digital IRU and AST
• Continue investigation to gain confidence that
  noise will not worsen on orbit
• Perform testing on MAP ETUs to identify cause of
  residual noise and data shifts
• Continue to perform in-depth investigation of
  telemetry on other RSNs
• Present additional data at Red Team Follow-up
  Review