An Introduction to Apache Hadoop Yarn

1
Apache Hadoop Yarn

• What is Yarn
• Problems with Hadoop
• What does Yarn Do ?
• Old Architecture
• New Architecture
• Yarn Example
• Additions

2
Hadoop Yarn What is it ?

• Next Generation MapReduce MRv2
• Split Job Tracker into
• Resource Manager
• Scheduling / Monitoring
• Improves scaling
• Improves resource management
• Already used by Yahoo

3
Problems with Hadoop 1.0

• Problems with large scaling
• gt 4000 nodes
• gt 40k concurrent tasks
• Problems with resource utilization
• Slots only for Map or Reduce
• Single NameNode, single point of failure
• Clients and Cluster must be at same version

4
What does Yarn do ?

• Provides a cluster level resource manager
• Adds application level resource management
• Provides slots for jobs other than Map / Reduce
• Improves resource utilization

5
Old Architecture

• Cluster level Job Tracker, Task Tracker on data
  node

6
New Architecture
7
New Architecture

• Resource Manager
• Cluster level resource manager
• Long life
• Node Manager
• One per data server
• Monitors resources on node
• Application Master
• One per application
• Short life
• Manages task / scheduling

8
Yarn Example
9
Yarn Example

• 1) Client -gt Resource Manager
• Submit App Master
• 2) Resource Manager -gt Node Manager
• Start App Master
• 3) Application Master -gt Resource Manager
• Request and release containers
• 4) Resource Manager -gt Node Manager
• Start tasks in containers

10
Additions

• Consider Weave
• Simplifies the use of Yarn
• Reduced development effort
• Simplified API

11
Contact Us

• Feel free to contact us at
• www.semtech-solutions.co.nz
• info_at_semtech-solutions.co.nz
• We offer IT project consultancy
• We are happy to hear about your problems
• You can just pay for those hours that you need
• To solve your problems