THE INDUSTRIAL REVOLUTION OF DATA

1
(No Transcript)
2
THE INDUSTRIAL REVOLUTION OF DATA

• logs
• sensors
• cameras
• gps
• ...

3
CRISIS (?) IN COMPUTING

• moores law, derailed
• processor clock rates have stopped growing
• storage, memory, network continue exponentiating
• solution (?) multicore
• many processors on a single chip
• massive parallelism (for the masses)

4
CRISIS AGAIN!

• cant clean up SW thats already out
• future does not look pretty either
• few developers can produce good parallel code

5
HEARD THIS BEFORE?

• dead parallel computer society
• Convex, Encore, Floating Point Systems, INMOS,
  Kendall Square Research, MasPar, nCUBE, Sequent,
  Thinking Machines...
• shared-nothing databases
• Gamma, Bubba, Teradata

6
NASA vs. FEDEX
7
DATAFLOW PARALLELISM

• split up a large set of inputs. not the algorithm.

• SQL (IBM, 1974)
• widely adopted in enterprises
• MapReduce (Google, 2002)
• widely adopted by hackers, students,
  algorithmicists
• very, very similar

master
QD processslice 3
QE processslice 2
QE processslice 2
QE processslice 2
QE processslice 1
QE processslice 1
QE processslice 1
segment1
segment2
segment3
8
SQL OR MAP/REDUCE?

• yes.
• tradeoffs in programmability/usability
• compatibility
• cultural diversity
• mix and match
• see above

9
FROM FLEETS TO SPACESHIPS

• alas, parallel dataflow only works for data
• wait ... nearly everything is data!

10
RETHINKING PROGRAMMING

• data-centric, declarative, distributed
• NWing, robotics, machine learning, NLP, games,
  cloud infrastructure...

11
RESEARCH LINCOLN BOOM

• lincoln
• a data-centric language for parallel computing
• the cloud goes BOOM!
• Berkeley Orders of Magnitude
• OOM bigger systems,OOM less code
• distributed filesystem, parallel dataflow
  infrastructure
• built out of dataflow!
• evolution incremental ilities with minimal
  effort