Introduction to Big Data

1
Big Data

• What is it ?
• What trends in IT support it ?
• Some examples
• What tools are used ?
• The future ?

2
Big Data What is it ?

• Very large data set volumes
• Very long / unacceptable processing times
• Very large data velocity ( inputs / outputs )
• Very large varieties of data
• High level of complexity

3
Big Data Supporting Trends

• Moore's Law
• An observation that the number of transistors on
  integrated circuits doubles every two years.

4
Big Data Supporting Trends

• Kryder's Law
• The density of storage is increasing and the cost
  decreasing at a rate faster than Moore's Law.

5
Big Data Supporting Trends

• Butter's Law
• Relates to network capacity and states that the
  cost of sending data over an optical network
  halves every nine months.

6
Big Data Supporting Trends

• Parallel Processing
• Task parallelism, breaking the task down into its
  constituent parts and processing them
  simultaneously.

7
Big Data Examples

• NASA Climate Simulation
• 32 petabytes
• The Large Hadron Collider
• 25 petabytes annually, 200 petabytes after
  replication
• Wall mart
• 2.5 petabytes per hour

8
Big Data Tools

• Hadoop
• Hadoop is often used at the server level to
  organise the cluster along with a NoSQL database
  for data storage.
• NoSQL
• Databases ( non sql ) that use looser consistency
  models than relational databases. Performance
  gains via simplification using key value stores.
• MPP
• Massively parallel processing and analytics
  databases. Fast for data aggregation but slow for
  data loading.

9
Big Data The Future

• Data sets will continue to grow
• Storage unit costs will continue to decrease
• Processing costs will decrease
• Network capacity will continue to grow
• Data growth may exceed processing capacity

10
Contact Us

• Feel free to contact us at
• www.semtech-solutions.co.nz
• info_at_semtech-solutions.co.nz
• We offer IT project consultancy
• We are happy to hear about your problems
• You can just pay for those hours that you need
• To solve your problems