May 2004

1
Alternative Access to Mission Control Center
(MCC) Services A Joint Effort Between NASA Centers

• May 2004
• Bryan E. Snook1, David H. Goeken1, Kim Lankford2,
  George Ritter2
• 1National Aeronautics and Space Administration,
  Johnson Space Center, Houston, TX, USA
• 2National Aeronautics and Space Administration,
  Marshall Space Flight Center, Huntsville, AL, USA

2
Background

• On October 1, 2002, the Mission Control Center
  (MCC) at the Johnson Space Center (JSC) was shut
  down in preparation of the Hurricane Lili threat
  to the Houston area. International Space Station
  (ISS) command and control was transferred to the
  Russian Backup Control Center (BCC) in Moscow,
  the first time in history such a transition was
  performed.
• Using Russian assets, Moscow-based BCC flight
  controllers had a very limited capability to
  monitor ISS core system telemetry.
• With the MCC non-operational, remaining US-based
  flight controllers had essentially no ability to
  monitor ISS core system telemetry simultaneously
  with the Moscow-based BCC flight controllers in a
  format that controllers were familiar with.

3
Background (contd)

• In an effort to make high-priority ISS core
  telemetry more widely available to both
  Moscow-based and US-based flight controllers in
  the event of another MCC shut down during the
  2003 Atlantic hurricane season, personnel from
  JSC worked effectively with personnel from the
  Marshall Space Flight Center (MSFC) to rapidly
  develop a solution that would enable select
  flight control team members to remotely and
  securely monitor high-priority ISS core
  telemetry.
• This was accomplished by way of US-based assets
  at the MSFC, Intel-based platforms, select
  commercial off the shelf products, and a custom
  software solution that would "translate"
  MSFC-formatted Custom Data Packets (CDP) into
  JSC-formatted Information Sharing Protocol (ISP)
  data packets, the primary information sharing
  protocol used in the MCC.

4
Background (contd)

• This pitch will give an overview of the
  development, integration efforts, and final
  products that team members from both JSC and MSFC
  were able to put forward in a relatively short
  time frame to prepare for the 2003 Atlantic
  hurricane season.
• The effort ran from April 15, 2003 to September
  10, 2003.

5
Problem Definition

• There are four core services that JSC flight
  controllers utilize while performing their duties
  in the Mission Control Center
• Commanding (via Dec Alpha Consoles)
• Telemetry Monitoring (via Dec Alpha Consoles)
• Voice Communication (via digital intercom system,
  DVIS)
• Data Exchange / View On-line Procedures (via
  OSTPV/MPV)
• Goal of the Summer 2003 joint JSC-MSFC team was
  to provide these core services, except
  commanding, to displaced JSC flight controllers
  by way of MSFC assets
• MSFCs existing distributed infrastructure, used
  primarily for ISS payload operations and support,
  was immediately leveraged
• Intel-based, SecuRemote type of architecture
• Both Voice (via the MSFC IVoDS) and and on-line
  procedure viewing (via MSFC OSPTV/MPV servers)
  were readily available
• The one missing element was JSC ISS high
  priority system telemetry
• Became the main focus of the team Make this
  telemetry available via MSFC

6
Telemetry Lists

• JSC Flight Controllers identified 1942 ISS system
  parameters that were considered high priority
  parameters
• This 1942 set was later expanded to 8,463
  parameters by the end of the summer (2003)

7
Options

• Four solutions were examined to provide the high
  priority parameters to remote, JSC flight
  controllers
• Option 1 100 MSFC Solution
• Add the parameters to the MSFC telemetry database
  servers and then build native EHS/EPC displays
• Utilize existing MSFC secure, remote access
  infrastructure to distribute data
• Option 2 PCDECOM Solution
• Install a JSC PCDECOM server at MSFC to process
  the parameters
• Utilize existing MSFC secure, remote access
  infrastructure to distribute data
• Option 3 Modified MSKWin Solution
• Modify MSKWin such that it would accept MSFC CDP
  packets
• Utilize existing MSFC secure, remote access
  infrastructure to distribute data
• Option 4 IspEPC Solution
• Install a JSC IspEPC server at MSFC to process
  the parameters
• Utilize existing MSFC secure, remote access
  infrastructure to distribute data

8
Pros and Cons

• Option 1 100 MSFC Solution
• Pros
• Highly self-contained solution
• Cons
• For every MSK, would have to create a duplicate
  EPC/EHS display
• Could not reuse existing MSK displays
• Comps not readily available
• Option 2 PCDECOM Solution
• Pros
• Direct access to decommutate the 192 kbps
  s-band telemetry
• Cons
• Create/Recreate a data calibration, configuration
  management process for the PCDECOM server
• Duplicates existing processes and resources at
  both JSC and MSFC
• Considered high overhead

9
Pros and Cons (contd)

• Option 3 Modified MSKWin Solution
• Pros
• 100 software solution
• Cons
• CDP-to-ISP conversation would be limited to
  MSKWin only
• Future programs could not leverage CDP-to-ISP
  conversation process
• Option 4 IspEPC Solution
• Pros
• Data calibration would be handed by the existing
  MSFC infrastructure and configuration management
  processes
• JSC flight controllers would be presented
  telemetry by way of existing, familiar MSK
  displays
• Computed data (comps) could be displayed (by
  way of IspATOM)
• Future ISP client programs could leverage this
  option
• Highly scalable and highly flexible
• Cons
• IspEPC dictionary maintenance

10
Basic Telemetry Architecture
EPC
New Piece developed For 2003 Hurricane Season
CDP
IspEPC
IspATOM
ISP
MSKWin
11
MSKWin, IspEPC, and IspATOM
12
JSC Client Performance

• Configuration Options A, B, and C were
  setup and fully tested at JSC (variants of Option
  3)
• Each configuration was tested over an 8-hour
  period, one after the other, for a combined total
  of 24 hours of testing
• A common benchmark test display of 8000
  parameters was used during testing on each
  machine in order to assess performance
• The reason for the significant bandwidth
  difference between using IspEPC (ISP) and EPC
  (CDP) is that ISP only publishes changes in the
  data to the clients that are attached to it
  whereas CDP publishes all values, all the time,
  regardless if the data has changed

13
Configuration Option A
JSC-DV-WKS1
JSC-DV-WKS2
MSFC I/F
JSC-DV-WKS3
Key
TLM
DELL C600
Voice
Procedures
14
(No Transcript)
15
Configuration Option A
All values are estimates
16
Configuration Option B
JSC-DV-WKS1
JSC-DV-WKS2
MSFC I/F
JSC-DV-WKS3
Key
TLM
DELL C600
Voice
Procedures
17
(No Transcript)
18
Configuration Option B
All values are estimates
19
Configuration Option C
JSC-DV-WKS1
JSC-DV-WKS2
MSFC I/F
JSC-DV-WKS3
Key
TLM
DELL C600
Voice
Procedures
20
(No Transcript)
21
Configuration Option C
All values are estimates
22
(No Transcript)
23
(No Transcript)
24
Telemetry Integrity Test ResultsMCC (Control)
vs. IspEPC via MSFC
JSC units are typically SI w/ very little
English (or no units) MSFC units are typically
English w/ very little SI (or no units) All
values are estimates MSFC units are now
mostly SI
25
Laptop Load
26
Laptop Load (contd)
27
Laptop Load (contd)
28
Laptop Load (contd)
Connectivity
MSFC Tools
JSC Tools
Misc Tools
29
Risks

• Changes to EPC can adversely impact the
  JSC-developed software, to the point of
  completely breaking the JSC software
• Mitigation
• MSFC/JSC should coordinate with each other on any
  changes that may affect the others software
• Efforts in 2003 went into developing IspEPC
  MSKWin application has not been modified and does
  have some issues
• Cannot handle (ridiculously) long enumerated data
  types (e.g. USDG01CC0140J)
• Slight tweaks had to be made to existing
  TITAN/ATLAS displays to get them to work properly
• IspEPC is currently not certified to the same
  level as other MCC capabilities, but has been
  tested
• Mitigation
• Crew confirmation/Other Independent source
  confirmation required prior to making any calls
• Certification is a priority for 2004

30
Conclusions

• The integrated test results were very positive
  and indicate a high probability of future success
  should the tested capability be invoked in time
  of need
• 99.2 match in the data between JSC and MSFC
• Configuration Option C is the winner
• Lowest impact to client resources
• Lowest impact to xwpvt003
• Low bandwidth consumption
• Centralized management of the IspEPC server
• EPC 3.0 not required any more on client machines
  (but can still be retained if desired)
• Can connect directly back to the IspEPC server at
  MSFC using only VPN connectivity

31
Conclusions (contd)

• MSFCs server infrastructure proved to be
  reliable and predictable
• Rock solid performance from xwpvt003
• Rock solid performance from the MSFC IspEPC
  Server
• Keeps up well with the original set (1941)
• Slower with the expanded set (8463)
• Consider 2 GHz processor, more memory as more
  parameters are added
• JSC client machines performed very well, even
  under high load, high stress conditions
• When operating as IspEPC servers, modern systems
  (e.g. JSC-DV-WKS1) can process over 8000
  parameters and still maintain good performance
  margins
• When operating as IspEPC servers, older systems
  (e.g. Dell C600) can also process up to 8000
  parameters, just a little slower than newer
  systems

32
Conclusions (contd)

• The IspEPC design is highly scalable and provides
  great flexibility in bandwidth-limited,
  performance-limited environments
• Older JSC client machines performed just as well
  as the newer machines when the newer machines
  carried the load of processing and distributing
  the data
• Gives new life to old hardware
• IspEPC servers can redistribute s-band data at
  no cost to the original source of publication
  (i.e. xwpvt003)
• IspEPC needs only one connection back to xwpvt003
• IspEPC can publish same data out to many clients
  with very little impact to system resources that
  it is running on
• For low-change rate data, IspEPC is an order of
  magnitude more efficient than EPC CDP when it
  comes to bandwidth consumption
• Makes it possible to distribute data over 56K
  dial-up if no other options are available
• Opens the door for future use of other ISP
  clients, such as IspATOM (for comps) and Birds
  Eye View

33
Conclusions (contd)

• In the end, the mini-MCC on a Laptop became a
  reality
• Telemetry available via MSFC assets and IspEPC
• Comps, Birds Eye View, other ISP clients can
  leverage IspEPC
• Two-Way Voice available via MSFC IVoDS
• On-Line Procedures available via MSFC Terminal
  Services

34
Thanks

• Special Thanks to David Goeken for his excellent
  work on IspEPC
• Special Thanks to everyone at MSFC that has
  helped us out along the way from the Help Desk
  to the programmers and several others
• It has been an excellent, enjoyable team effort!

35
Backup Charts
36
Voice Bandwidth Consumption

• Voice bandwidth consumption was measured by
  joining a test conference loop using one PC at
  JSC, speaking, listening to the return voice on
  a second PC at JSC, and then measuring the
  bandwidth on each of the PCs while voice on the
  test loop was active
• There appeared to be about a 2 to 3 second
  latency interval between forward and return
  voice
• The measured return voice bandwidth on the
  listening PC was on the order of 20 Kbps
  however, a value of 32 Kbps may be used as a more
  conservative number
• The measured forward voice bandwidth on the
  talking PC was on the order of 15 Kbps

37
Voice Bandwidth Consumption (contd)

• Bandwidth for monitor loops is then 20 to 32 Kbps
• Bandwidth for talking loops is then 20 16 36
  Kbps
• One can listen up to 8 voice loops, including the
  ability to speak on one loop (at a time and as
  authorized)
• Knowing the expected performance for one loop,
  values were extrapolated, plotted, and a trend
  line was applied
• y 32x 16z
• The x variable in the equation represents the
  number of active loops
• The z variable in the equation simply takes
  into account the expected overhead for the talk
  loop
• y is the total bandwidth, units are Kbps (not
  KB/s)

38
(No Transcript)
39
(No Transcript)
40
(No Transcript)