[Part 2] Automation of Home Appliances using Raspberry Pi – Implementation of Hand Gesture Recognition

1
(No Transcript)
2
Intro

• In our recent blog we have talked about software
  installation guide to setup and begin automation
  of home appliances that can be used to detect
  human hand gesture to control home appliances.
• We are now in the 2nd part of the blog where we
  intend to develop an application that is built
  on RASBERRY PI using java and computer vision
  APIs. It will detect the hand gesture and
  consequently pass the signals to IR device which
  will then perform appropriate home automation.
• This blogs guides you with the complete procedure
  for detecting human hand gesture and interacting
  with hardware/home appliances.

3
The role of the application in implementing hand
gesture recognition for home automation.

• Process
• Application and IR device must be part of same
  network.
• Application capture camera input continuously.
• From captured image frames, Hand gesture is
  detected using OpenCV data structure/algorithms.
• Appropriate signals are found and consequently
  passed on to IR device (HTTP call) with proper
  command (Ex ON/OFF) to external device.

4
Below are the core functionalities and modules
that an application contains

• Gesture detection  The application contains open
  computer vision APIs. OpenCV and JavaCV process
  the camera input and apply algorithms to process
  their frames.
• Smart Config  It is the heart of application
  that can talk with IR device to control external
  device.This module has two modes Learning and
  Execution. During Learning mode, It configures IR
  device using TI Smart config protocol. it
  monitors and records data sequence for different
  infrared sequence for home appliances operations.
  During Execution mode, it can send the recorded
  sequences to IR device, which are converted by IR
  device into infrared sequence to control
  different devices.
• Signal Processing  It is a remote processing
  part of the application, in which application
  stores the signals and passes it to IR device for
  further operation.

5
Technologies used in implementing hand gesture
recognition.

• OpenCV
• In the project we use OpenCV contours for
  recognizing the hand image.
• Contours can be explained simply as a curve
  joining all the continuous points (along the
  boundary), having same color or intensity. The
  contours are a useful tool for shape analysis and
  object detection and recognition. For better
  accuracy, use binary images. So before finding
  contours, apply threshold or canny edge
  detection. findContours function modifies the
  source image. So if you want source image even
  after finding contours, already store it to some
  other variables.
• In OpenCV, finding contours is like finding white
  object from black background. So remember, object
  to be found should be white and background should
  be black.

6
Method / Algorithms to detect hand from video
frame

• Briefly let us look at the several ways to
  process a frame to detect the hand and their
  gestures
• Background Subtraction A simple method to start
  with is we can subtract the pixel values. However
  this will result in negative values and values
  greater than 255, which is the maximum value used
  to store an integer. And what if we have a black
  background? Nothing gets subtracted in that case.
  Instead we use an inbuilt background sub-tractor
  based on a Gaussian Mixture-based
  Background/Foreground Segmentation Algorithm.
  Background subtraction involves calculating a
  reference image, subtracting each new frame from
  this image and threshold the result which results
  is a binary segmentation of the image which
  highlights regions of non-stationary objects .We
  then use erosion and dilation to enhance the
  changes to make it more prominent.

7
(No Transcript)
8

• Color detection  The skin color detection is one
  of the most popular methods. This method is
  simple and depends on skin color that can be
  white, black, or other colors, and the
  environment light conditions, as well as the
  background.

9

• Edge detection It is an edge detection operator
  that uses a multi-stage algorithm to detect  wide
  range of edges in images, Its aim is to
  satisfy the below criteria
• Lower Error Rate
• Good Localization
• Minimal Response

10
(No Transcript)
11

• Note  For our solution we have used color
  detection method to detect hand from frame with
  user skin color. Application itself has GUI to
  detect user hand skin color and persist for
  further application uses.
• After analyzing the application side requirements
  lets move on to middle level device for further
  communication called IR device.

12
IR Device

• IR device works as IR transreceiver. It is used
  to record IR command for different appliances and
  that recorded sequence is used to control
  appliances based on different gestures detected
  in previous phase. IR device can be connected to
  TV, set-top boxes, stereos, air conditioners and
  other appliances and then can pass infrared
  remote control signal and start the home
  appliance.

13

• After understanding the all the devices, lets now
  understand the procedure to develop the
  application.

14
 Guide to develop the Application

• Home automation is the residential extension of
  building automation and involves the control and
  automation of lighting, air conditioning, and
  security, as well as home appliances such as
  washer/dryers, television, ovens or
  refrigerators/freezers use Wi-Fi for remote
  monitoring.
• Modern systems generally consist of switches and
  sensors connected to a central hub sometimes
  called a gateway from which the system is
  controlled with a user interface that is
  interacted either with a wall-mounted terminal,
  mobile phone software, tablet computer or a web
  interface, often but not always via internet
  cloud services.
• The application we have developed has a user
  interface which is able to manage devices and
  communication path. It has a robust feature to
  make device as habituate by their Learning Mode
  and Execution Mode.

15
 SmartConfig

• Its a Wi-Fi setup process that allows multiple
  in-home devices to connect to Wi-Fi networks
  quickly and efficiently. It is specifically for
  applications that typically do not have a display
  or keyboard to enter network name and password.
• SmartConfig interface gives end users the ability
  to easily connect IR devices to an access point.
  Through a simple SmartConfig method/interface,
  consumers can use any programming concept to
  create own application using SmartConfig
  Interface.
• Connect IR device / hardware with application, is
  a first step to connect IR device to application
  by sending Broadcast Datagram packets with
  network (Wi-Fi) details (SSID and Password).

16

• This process is able to send and receive
  broadcast datagram packets and make connection
  with application and IR hardware device.
   Application has a GUI (Graphical User Interface)
  that allow user to connect device.
• Application first check whether any IR device is
  already connected, if not than it will prompt a
  window to connect device with some validations
  as.

17

• Application creates a socket connection with host
  (Router) and sends broadcast datagram packets
  with SSID and password to get connected with IR
  device and wait for acknowledgment from IR smart
  device.

18

• Device learning After IR device is connected
  with the application, the user is able to setup
  commands/signal for home appliances based on
  their hand gesture, and it starts with a learning
  step.
• In learning phase user is able to setup one or
  multiple instructions or command for multiple
  devices using GUI of application.
• Our current implementation has basic gestures
  like 0-5 count, up-down, left-right. Based on
  user configuration, application will operate
  their attached devices.
• For e.g.  Increase TV volume on Up-Down movement
  of hand.

19
(No Transcript)
20

• After learning phase, you are ready to operate
  your home appliances with your hand gestures. Now
  user can Switch On, Off the TV and also manage
  volume in our application implementation
• We have configured 6-7 command to operate TV as
  ON, OFF, Volume Up- Down, Mute, Open menu bar,
  Exit menu bar and their operations.

21

• Signal Processing (Remote service)- Our
  application is able to detect hand gesture and
  pass the IR signal to configured home appliances
  according to human hand gesture.
• Application is capable of managing User
  interaction, IR device communication, signal and
  skin color persistence.
• Application gateway is incorporated to connect
  smart home appliances, and an IOT connection
  management platform is required to enable various
  applications.

22

• These applicable solutions include home
  automation, which idyllically promotes happier,
  and more comfortable and convenient lifestyle.
• The solution will help operators to transform
  from traditional home broadband services to
  provide intelligent home services.

23
(No Transcript)
24
IR Signal Execution (IR signal transmission)
25

• Conclusion
• Home automation using Raspberry pi through hand
  gesture recognition hence is a significant
  undertaking in IoT domain. With improved
  convenience, comfort, energy efficiency and
  security under IoT domain, it has proved to be
  incredibly beneficial for both individual and
  society.
• So Whats next
• In our next blog we are going to look at real
  time working application demo and step by step
  code snippet of the application. So lets grab
  the opportunity and look what the next blog
  brings.

26

• Original Source
• http//www.azilen.com/blog/automation-of-home-appl
  iances-using-raspberry-pi-part-2/

27
Thanks You www.azilen.com Let us know if you
have any Question!