Managing your Assets with Big Data Tools (1)

1
Managing your Assets with Big Data Tools

• Karthigai Muthu, MachinePulse

2
Agenda

• Big Data value proposition
• Big Data Technology Stack

3
Hype Cycle for Emerging Technologies
Source Wikipedia
4
Sources of data
12 TBs of tweet data every day
? TBs ofdata every day
25 TBs oflog data every day
5
What makes Data Big
Characteristics Description Attributes Drivers
Volume The amount of data generated or intensify that must be ingested, analyzed and managed to make decision based on complete data analysis Exabyte (EB) Zettabyte (ZB) Yottabyte (YB) Increase in data sources Higher resolution sensors Scalable infrastructure
Velocity How fast the data is being produced and changed and the speed at which is transformed into insight Batch Near real time Real time and Streams Rapid feedback loop Improved throughput connectivity Competitive advantage Pre-computed information
Variety The degree of diversity of data from sources both inside and outside an organization Degree of structure Complexity M2M/IoT Social Media Genomics Video and Mobile
Veracity The quality and provenance of data Consistency Completeness Ambiguity Integrity Cost Need of traceability and justification
6
Big Datas Greatest Power Predictive Analytics
7
Whats driving Big Data
8
The Evolution of Business Intelligence
Big Data Real Time Single View Graph
Databases
Interactive Business Intelligence In-memory
RDBMS QlikView, Tableau,HANA
Speed
Scale
BI Reporting OLAP Data warehouse Business
Objects, SAS, Informatica, Cognos other SQL
Reporting Tools
Speed
Scale
Big Data Batch Processing Distributed Data
Store Hadoop/Spark HBase/Cassandra/MongoDB
1990s
2000s
2010s
9
Solving business problem with big data
10
Formulation of big data strategy
11
Companies Market share in Big Data
12
Big Data Investments
13
Priority for big data across industry
14
Are you aware the risk of not implementing Big
Data in your company
15
Big data changed connected things to Internet of
Everything(IoE)
16
How the industry can leverage from big data
17
Challenges in implementing the big data
18
Returns of Investment(ROI)
19
How do companies get MORE from big data

• Merge
• Optimize
• Respond
• Empower

20
Are you planning to launch your new product.
21
Customer 360
Social Media
Banking Finance
Our Known History
Gaming
Purchase
Entertain
Customer
22
Real-Time Analytics/Decision Requirement
Product Recommendations that are Relevant
Compelling
Friend Invitations to join a Game or
Activity that expands business
Influence Behavior
Improving the Marketing Effectiveness of a
Promotion while it is still in Play
Customer
Learning why Customers Switch to competitors
and their offers in time to Counter
Preventing Fraud as it is Occurring
preventing more proactively
23
IoTBig Data IoE(Internet-of-Everything)
24
Role of Big Data in M2M/IoT

• Big Data is a factor that will, to a large
  extent, determine the future growth rate in the
  M2M industry
• M2M will connect increasingly more nodes that
  will provide data from endpoints.
• Data will be more granular, more frequent, and
  more accurate, with bigger data sets or even live
  data streams
• Large volume of endpoint connections IPv4
  addressing scheme cant accommodate
  everything(sensors, smart phones, smart
  factories, smart grids, smart vehicles,
  controllers, meters ) that it requires IPv6
• IoE Convergence of IoT, Big Data Analytics
  ,Cloud Computing and other technologies is
  collectively called as Internet of Everything

25
Challenges of Big Data in M2M/IoT

• Meeting the need for speed
• Data understanding
• Maintaining data quality
• Displaying the meaningful result

26
IoT/M2M Applications..
27
Big Data Use Cases IoT/M2M

• Personal IoT the scope is a single person, such
  as a smartphone equipped with GPS sensor or a
  fitness device that measures the heart rate. This
  is one of the fastest growing, consumer-oriented
  areas of IoT.
• Group IoT the scope is a fairly small group of
  people, such as a family in a smart house,
  co-workers in a van or a group of tourists. This
  is one of the most challenging areas and is still
  in its early phase.
• Community IoT the scope is a large group of
  people, potentially thousands and more usually
  this is in a public infrastructure context, such
  as smart cities or smart roads. This is a young
  and potentially promising IoT area.
• Industrial IoT the scope can be within an
  organization (smart factory) or between
  organizations (retailer supply chain). This is
  arguably the most established and mature part of
  IoT.

28
Big Data Use cases IoT/M2M

• Agriculture - sensors can be deployed on farm
  machinery in order to provide data about the
  equipment, soil temperature, moisture, etc.
• Buildings/Smart Homes - Building sensors be used
  to help facility managers become more proactive
  about ensuring that their buildings operate at
  peak efficiency.
• Communities Smart cities make use of parking
  space availability systems, intelligent traffic
  monitoring systems, intelligent highways,
  weather-adaptive street lighting, and more.
• Healthcare Infant monitors, smart diapers,
  pills with ingestible sensors are just some of
  the IOT-based devices.
• Manufacturing factories with sensors can
  improve operations, product quality, and decrease
  safety hazards.
• Smartphones can control everything from door
  locks, thermostats, light bulbs, vacuum cleaners,
  and more.
• Utilities smart water meters can be used to
  reduce water leaks. Smart electric grids can
  adjust rates depending on usage.
• Wearables Smart watches, fitness trackers and
  health monitors may become primary source for
  human-related data, and can also be used in
  sports, retail, travel and manufacturing.

29
Benefits of Big Data Analytics in M2M/IoT

• Device Maintenance
• a. Time for next patch upgrade
• b. Energy management
• c. Inventory management and track replacement
• 2. Proactive Healthcare
• Capture and analyze real time data from medical
  monitors to predict potential health problems
  before patients manifest clinical signs of
  infection.
• 3. Monetize Machine Data
• a. Monitor performance, usage and capacity
  details to uncover up-sell and cross-sell
  opportunities
• b. Maximize the lifespan and performance of high
  value medical assets

30
Benefits of Big Data Analytics cont..

• 4. Optimize Support Operations
• a. Reduce MTTR and support escalations
• b. Preempt failures with proactive support
• c. Troubleshoot with accurate information
• d. Proactive consultation to customers on
  approaching expiry dates

31
Big Data Analytics Stack
32
Lamda Architecture
33
(No Transcript)
34
Batch vs. Real-Time processing

• Batch processing
• - Gathering of data and processing as a group at
  one time.
• - Jobs run to completion
• - Data might be out of date
• Real-time processing
• - Processing of data that takes place as the
  information is being entered.
• - Run for ever

35
Storm

• Apache Storm is a free and open source
  distributed real-time computation system.
• Storm makes it easy to reliably process unbounded
  streams of data, doing for real-time processing
  what Hadoop did for batch processing

36
Storm Is

• Stream Processing
• Fast
• Scalable
• Fault Tolerant
• Reliable

37
Tuple
38
Streams
39
Spouts
40
Bolts
41
Topologies
42
Reliable Processing
43
Reliable Processing
44
Stream Grouping

• Groupings are used to decide to which task in
  the
• subscribing bolt (group) a tuple is sent.
• Possible Groupings
• - Shuffle
• - Fields
• - All
• - Global
• - None
• - Direct
• - Local or Shuffle

45
Storm Cluster View
46
Fault Tolerance
47
Fault Tolerance
48
Fault Tolerance
49
Fault Tolerance
50
Fault Tolerance
51
Parallelism
52
Parallelism
53
Apache Storm Real-time -Use cases
Segment Prevent Use Cases Optimize Use Cases
Financial Services Securities fraud Operational risks compliance violations Order routing Pricing
Telecom Security breaches Network outages Bandwidth allocation Customer service
Retails Shrinkage Stock outs Offers Pricing
Manufacturing Preventative maintenance Quality assurance Supply chain optimization Reduced plant downtime
Transportation Driver monitoring Predictive maintenance Routes Pricing
Web Application failures Operational Issues Personalized content
54
The End