Innovations in Process Automation

1
Innovations in Process Automation

• Larry Teng
• teng_at_csr.utexas.edu

2
Some of the Experiments

• Hardware
• RAM Disk Processing
• Linux adjustment for optimization
• Software
• Automated product delivery
• Adapting external algorithms or sources to
  TeraScan
• Networking
• Distributed Computing
• Data Repository/Data Merge

3
RAM Disk Processing
4
Basic Ideas

• RAM Disk Memory can be allocated to simulate
  physical hard-drive in a Linux system.
• Without mechanical movement in physical
  hard-drive, I/O time from RAM is much faster than
  from hard-drive.
• The RAM Disk size is adaptive to the operating
  system. If the system requires more memory, the
  actual storage of RAM Disk can be less than the
  amount allocated.

5
Experiment

• Physical memory size 2GB.
• Limit the memory usage for operation system to
  less than 256MB.
• Choose a single granule short duration MODIS
  (around three minutes) as test sample.
• Process the telemetry to L0 using stps and L0 to
  L1B using DAAC software.

6
RAM Usage
7
Level 0 Processing

• Using Java application, stps, to process the
  telemetry.
• Due to the lack of physical memory, raw telemetry
  is dumped from RAM Disk once PDS is generated.
• During this process, CPU utilization usually runs
  at 97, which means that most computing
  activities consume on processing.

8
Level 1 Processing

• This processing consumes memory intensively,
  especially for MOD_PR02.
• Require more memory for dynamic allocation by the
  processing program.
• Reducing memory size by half causes 55 increase
  of processing time.

9
Conclusion

• Estimate 4GB of memory is needed for processing
  the whole swath of MODIS.
• Except for the use of OS and Java Virtual
  Machine, all the rest memory is allocated as RAM
  Disk and dynamic memory to store and process
  data, respectively.
• Overall processing time drops 25 comparing to
  regular hard-drive processing.

10
Data Repository and Data Merge of OCM
11
Data Repository

• Data acquired by UTCSR are stored at the mass
  storage at Texas Advanced Computing Center.
• Collecting OCM passes from LSU, Rutgers, and
  UTCSR.
• UTCSR hosts a repository to automatically archive
  OCM data from three different sources.

12
OCM Coverage
13
Data Merge

• Low-elevation data over the east coast acquired
  by UTCSR are usually noisy, while Rutgers data
  are clean and integral.
• Evaluate the noisiness by checking the constant
  bytes in the data stream.
• Produce a quality table for each telemetry, then
  compare it with the same pass from other source.

14
OCM Line Fixer

• Ingestion using ocmin does not repair or check
  missing lines in OCM data stream.
• Frame order is essential to achieve merging data
  from different sources.
• Fixing missing lines naturally becomes an
  intermediate step in data merging.

15
Low-Elevation Capture
16
(No Transcript)
17
(No Transcript)
18
Antenna A
Antenna B
Merged Result
19
Utilizing Java RMI

• RMI Remote Method Invocation.
• Client can check the availability and quality of
  certain frame in the data stream at the
  repository remotely.
• Benefits
• Instant response to eliminate the trouble of
  asking repository provider for information.
• Better maintenance and quality control to the
  software.

20
Future Work in Data Merge

• Remotely retrieve required frame from repository
  and merge the data locally.
• GUI for client to visualize the quality of the
  frame and decide whether to retrieve the frame
  interactively.
• Expand the applicable telemetries if necessary.
  (for example, FY-1D)