An Introduction to Google Percolator

1
Google Percolator

• What is it ?
• What is it used for ?
• Percolator Vs MapReduce
• Architecture

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
2
Percolator What is it ?

• Incremental updates to Big Data
• Developed by Google
• Based on Google File System ( GFS )
• Provides transactions and locking
• Faster than comparable Map Reduce
• Developed by Google due to MapReduce limitations

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
3
Percolator What is it used for ?

• Iterative updates
• No need to batch process
• Update as data received
• Data in multi petabyte range
• Strong consistency needed
• Improved latency ( 100 x )
• Reduced document age ( 50 )?
• Random access to big data repository

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
4
Percolator Vs MapReduce

• Map Reduce
• Batch Processing
• No transactions
• Latency A
• Run time scales with data
• Code in C
• Open source
• Uses HDFS

• Percolator
• Iterative
• Transactions
• Latency 100 x A
• Incremental updates
• Code in Java ( mainly )?
• Google owned
• Uses GFS

www.semtech-solutions.co.nz info_at_semtech-solutio
ns.co.nz
5
Percolator Architecture

• Applications are a sequence of observers
• An observer is called via a notification
• A notification is triggered when table data
  changes
• Application calls TabletServer via RPC
• TabletServer calls GFS ChunkServer

6
Percolator Architecture

• Applications
• Series of observers
• Observer
• Completes task
• Updates table
• Next Observer called
• Via notification
• Percolator worker
• Scans for changes
• Sends notifications

7
Percolator Architecture

• Actual worker diagram including time stamping and
  locking
• via Chubby lock server

8
Contact Us

• Feel free to contact us at
• www.semtech-solutions.co.nz
• info_at_semtech-solutions.co.nz
• We offer IT project consultancy
• We are happy to hear about your problems
• You can just pay for those hours that you need
• To solve your problems