kajal12

1
Evolution of Netflix Conductor
2
Conductor is a workflow orchestration engine
developed and open-sourced by Netflix. If youre
new to Conductor, this earlier blog post and the
documentation should help you get started and
acclimatized to Conductor.
Over the most recent a long time since
commencement, Conductor has seen wide
appropriation and is instrumental in running
various center work processes at Netflix. A
considerable lot of the Netflix Content and
Studio Engineering administrations depend on
Conductor for proficient handling of their
business streams. The Netflix Media Database
(NMDB) is one such model. In this blog, we
might want to show the most recent updates to
Conductor, address a portion of the much of the
time posed inquiries and thank the network for
their commitments.
3
Visit here for more details office setup today
4

• How were using Conductor at Netflix
• 1.Deployment
• Conductor is one of the most heavily used
  services within Content Engineering at Netflix.
  Of the multitude of modules that can be plugged
  into Conductor as shown in the image below, we
  use the Jersey server module, Cassandra for
  persisting execution data,Dynomite for persisting
  metadata, Dyno Queues as the queuing recipe built
  on top of Dynomite, Elastic search as the
  secondary data store and indexer, and Netflix
  Spectator Atlas for Metrics. Our cluster size
  ranges from 1218 instances of AWS EC2 m4.4xlarge
  instances, typically running at 30 capacity.
• We do not maintain an internal fork of Conductor
  within Netflix. Instead, we use a wrapper that
  pulls in the latest version of Conductor and adds
  Netflix infrastructure components and libraries
  before deployment. This allows us to proactively
  push changes to the open source version while
  ensuring that the changes are fully functional
  and well-tested.
• 2. Adoption
• As of writing this blog, Conductor orchestrates
  600 workflow definitions owned by 50 teams
  across Netflix. While were not (yet) actively
  measuring the nth percentiles, our production
  workloads speak for Conductors performance.
  Below is a snapshot of our Kibana dashboard which
  shows the workflow execution metrics over a
  typical 7-day period.

5

• Use Cases
• Some of the use cases served by Conductor at
  Netflix can be categorized under
• Content Ingest and Delivery
• Content Quality Control
• Content Localization
• Encodes and Deployments
• IMF Deliveries
• Marketing Tech
• Studio Engineering

Whats New gRPC Framework One of the key
features in v2.0 was the introduction of the gRPC
framework as an alternative/auxiliary to REST.
This was contributed by our counterparts at
GitHub, thereby strengthening the value of
community contributions to Conductor.
6
Cassandra Persistence Layer To enable horizontal
scaling of the data store for large volume of
concurrent workflow executions (millions of
workflows/day), Cassandra was chosen to provide
elastic scaling and meet throughput
demands. External Payload Storage External
payload storage was implemented to prevent the
usage of Conductor as a data persistence system
and to reduce the pressure on its backend data
store. Dynamic Workflow Executions For use
cases where the need arises to execute a
large/arbitrary number of varying workflow
definitions or to run a one-time ad hoc workflow
for testing or analytical purposes, registering
definitions first with the metadata store in
order to then execute them only once, adds a lot
of additional overhead. The ability to
dynamically create and execute workflows removes
this friction. This was another great addition
that stemmed from our collaboration with GitHub.
7
Workflow Status Listener Conductor can be
configured to publish notifications to external
systems or queues upon completion/termination of
workflows. The workflow status listener provides
hooks to connect to any notification system of
your choice. The community has contributed an
implementation that publishes a message on a dyno
queue based on the status of the workflow. An
event handler can be configured on these queues
to trigger workflows or tasks to perform specific
actions upon the terminal state of the
workflow. Bulk Workflow Management There has
always been a need for bulk operations at the
workflow level from an operability standpoint.
When running at scale, it becomes essential to
perform workflow level operations in bulk due to
bad downstream dependencies in the worker
processes causing task failures or bad task
executions. Bulk APIs enable the operators to
have macro-level control on the workflows
executing within the system. Decoupling Elastic
search from Persistence This inter-dependency
was removed by moving the indexing layer into
separate persistence modules, exposing a property
(workflow.elasticsearch.instanceType) to choose
the type of indexing engine. Further, the indexer
and persistence layer have been decoupled by
moving this orchestration from within the primary
persistence layer to a service layer through the
Execution DAOFacade.
8
ES5/6 Support Support for Elastic search
versions 5 and 6 have been added as part of the
major version upgrade to v2.x. This addition also
provides the option to use the Elastic search
Rest Client instead of the Transport Client which
was enforced in the previous version. This opens
the route to using a managed Elastic search
cluster (a la AWS) as part of the Conductor
deployment. Task Rate Limiting Concurrent
Execution Limits Task rate limiting helps
achieve bounded scheduling of tasks. The task
definition parameter rateLimitFrequencyInSeconds s
ets the duration window, while rateLimitPerFrequen
cy defines the number of tasks that can be
scheduled in a duration window. On the other
hand, concurrentExecLimitprovides unbounded
scheduling limits of tasks. I.e the total of
current scheduled tasks at any given time will be
under concurrentExecLimit. The above parameters
can be used in tandem to achieve desired
throttling and rate limiting.
9
API Validations Validation was one of the core
features missing in Conductor 1.x. To improve
usability and operability, we added validations,
which in practice has greatly helped find bugs
during creation of workflow and task definitions.
Validations enforce the user to create and
register their task definitions before
registering the workflow definitions using these
tasks. It also ensures that the workflow
definition is well-formed with correct wiring of
inputs and outputs in the various tasks within
the workflow. Any anomalies found are reported to
the user with a detailed error message describing
the reason for failure. Developer Labs, Logging
and Metrics We have been continually improving
logging and metrics, and revamped the
documentation to reflect the latest state of
Conductor. To provide a smooth on boarding
experience, we have created developer labs, which
guides the user through creating task and
workflow definitions, managing a workflow
lifecycle, configuring advanced workflows with
eventing etc., and a brief introduction to
Conductor API, UI and other modules.
10
(No Transcript)