Ochestrate

1
Ochestrate

• ??? ?? ???
• ?? 1??
• ? ? ?
• (002COG32)

2
Torrent Systems Inc.

• Founded in 1993
• Developing and deploying large-scale analytical
  systems
• Today, making it possible for eBusiness and
  click-and-mortar companies to process and analyze
  unlimited volumes of clickstream and transaction
  data.
• More than 13.5 million in equity and investment
  capital

3
Questions to solve

• Terabytes of transaction, clickstream, and
  historical data.
• Who are my most profitable customers?
• What products do they buy and when?
• How can I keep my customers from switching to
  another vendor?
• How do I find new customers?
• How can I expand my sales?

4
Why Orchestrate?

• Tradeoffs between dada volume, analytical
  complexity, and timely delivery.
• Internet economy
• Torrents Orchestrate simplifies and accelerates
  the development, deployment, and management of
  enterprise-scale analytical applications.

5
Orchestrate

• Unrestricted analytical complexity
• removing the performance limitations associated
  with sequential processing
• Large data volumes
• making it easy to apply the power of
  multiprocessor computers to complex business
  problems
• Solution Timeliness
• reducing the time required to analyze data,
• increasing the frequency with which companies can
  execute high-value applications

6
Orchestra technology

• Processing time is no longer consideration
• large eCommerce site, portals, and others that
  want to increase sales with realtime customer
  profiling, scoring, Web personalization, and
  tailored offers
• click-and-mortar companies that want to integrate
  Web data with demographic and transaction data
  from the data warehouse
• any enterprise that wants to use the power of
  high-velocity analytical systems

7
Build

• Component-Based Development and Assembly
• Visual environment for component development and
  application assembly
• Libraries of prebuilt components for analytics
  and data mining, database operations, integration
  of third-party formats and operations, and common
  data warehousing and web data preparation
  operations
• Scripting and C interfaces
• "Wrapping" of existing, sequential application
  inside Orchestrate parallel-enabling technology
• Wizard-automated generation of C components
• Sequential debugging of parallel programs

8
Deploy

• Application Deployment
• Configuration management
• Validation of data flow before execution
• Consistency checking of metadata
• Single point of control for deployment in MPP
  environments
• Automatic cleanup and shutdown after abnormal
  termination
• Rapid job startup and shutdown

9
Manage

• Application Management
• Performance monitor with continuous graphical and
  statistical presentation of data flow rates,
  record read/write rate, etc.
• Checkpoint-restart
• Centralized collection of error and event
  messages
• Can be integrated with standard monitoring and
  scheduling systems

10
Execute

• Application Execution Engine
• Scalability and Performance
• Partition parallelism
• Pipeline parallelism
• Flow control for movement of large volumes of
  data
• Scalable data storage
• High-performance, parallel interfaces to
  parallel databases
• Record-oriented data processing
• Seamless integration of external data formats,
  hierarchical methods, and mainframe data types
• Architecture independence across workstations,
  SMPs, MPPs
• Portability across processors, operating systems,
  and databases

11
?!
12
Visual Environment

• First, because Orchestrate manages the
  complexities of mapping the application logic to
  the parallel hardware, programmers can develop
  their applications using seqiuential logic.
• Second, Orchestrates open architecture supports
  several development interfaces.

13
(No Transcript)
14
Orchestrate Components

• A functional, reusable code that contains the
  application logic essential for developing
  enterprise-scale analytical applications
• Build Orchestrate applications by selecting from
  a library of components and connecting them
  sequentially in a way that illustrates the flow
  of data from one application process to another

15
Components Function

• Essential data processing functions
• Web clickstream processing
• Data mining, Web mining and other analytical
  processes
• Reading from and writing to databases in parallel
• Flexible interfaces to flat files
• Executing SAS PROC and DATA steps in parallel
• ?????!!

16
Wrapping of Existing Logic within Orchestrate
Parallel Technology

• Wrapping existing sequential application logic
  Orchestrate parallel-enabling technology
• Enterprise can execute existing code in parallel
  without incurring the productivity cots of
  rewriting critical systems
• ?????!

17
Sequential Debugging of Parallel Programs

• Orchestrate simplifies application development
  and debugging by allowing developers to execute
  Orchestrate applications in sequential mode.
• Sequential execution mode runs the Orchestrate
  application on a single processing node, allowing
  developers to run a single debugger session,
  setting breakpoints anywhere within the code.
• ?????!

18
Configuration Management

• Orchestrate speeds the deployment of parallel
  applications by managing the processes of mapping
  the application to system resources, scaling your
  application to accommodate changes in resources,
  partitioning data across multiple processing
  nodes, recognizing data set structure, and other
  processes essential to smooth parallel execution
  within the Orchestrate environment.
• ?????!

19
Consistency Checking of Metadata

• A powerful feature of Orchestrate is its ability
  to handle many data formats and stream the data
  seamlessly between applications in your data or
  Web warehouse, enabling substantial gains in
  performance and productivity.
• This capability allows you to create fully
  integrated systems without the need to write data
  conversion programs or land the data to disk
  between different applications.
• ?????!

20
Performance monitor
21
Partition and Pipeline Parallelism

• Partition parallelism partitions data among the
  processors and streams it through multiple
  instances of the application, taking full
  advantage of the system's parallel processors
  without the need for parallel programming or the
  development of multi-threaded applications.
• Pipeline parallelism allows downstream
  application processes to begin executing as soon
  as they start receiving data, even if preceding
  processes are still processing data.

22
Partition and Pipeline Parallelism (cont.)

• Finally, Orchestrate handles the metadata for all
  application processes, a unique capability that
  increases performance and productivity gains by
  enabling Orchestrate to stream data seamlessly
  among diverse application types?without data
  conversion programs or the need to land data to
  disk.
• ?????!

23
Record-Oriented Data Processing

• Orchestrate's record-oriented data processing
  model extends the handling of record-oriented
  data to applications that execute outside the
  RDBMS.
• ?????!

24
Checkpoint-Restart

• In the event of a system failure, Orchestrate's
  Checkpoint-Restart capability allows you to save
  critical time by resuming execution at a defined
  boundary checkpoint, instead of the beginning of
  the application step.
• ?????!