Paper id 173

1
2nd IEEE International Conference on Data
Decision Support Systems ICDDS- 2023
1st - 2nd December 2023
Paper ID 173 Title Artificial Intelligence
Based Enhanced Virtual Mouse Hand Gesture
Tracking Using Yolo Algorithm Authors Karthick
S, Dinesh M, Jeffery Dani Raj Affiliation Christ
University
2
RV College of Engineering
Outline

• Introduction
• Literature Survey
• Research gap
• Objectives Methodology
• Novelty
• Design and Implementation
• Result Analysis
• Conclusion and Future Work
• References

3
RV College of Engineering
Introduction

• Develop an improved hand tracking mouse system.
• Enhance accuracy in hand detection for precise
  cursor movement. Optimize cursor movement speed
  for smooth and responsive control.
• Improve responsiveness to hand gestures for
  intuitive interaction. Improved user experience
  with enhanced accuracy and responsiveness.
• Enhanced productivity and usability across
  various domains.
• Address limitations of existing hand tracking
  mouse systems.
• Problem Statement and Objectives The existing
  hand tracking mouse system suffers from
  limitations in terms of accuracy, speed, and
  efficiency. The current hand detection model has
  a high time complexity of O(n3), resulting in
  slow cursor movement and inaccurate tracking of
  hand movements.
• These drawbacks hinder the usability and
  effectiveness of the hand tracking mouse as an
  alternative input device.

4
RV College of Engineering
Literature Survey
NO Title Problem Statement Solutions
1 Deep Learning-Based Real-Time AI Virtual Mouse System Using Computer Vision to Avoid COVID-19 Spread The COVID-19 pandemic has necessitated the reduction of physical contact with shared surfaces such as computer mice and keyboards. Therefore, there is a need for a contactless system. Develop hand tracking and gesture recognition algorithms to enable users to control a virtual mouse cursor using hand movements and gestures, reducing the risk of virus transmission through shared physical mouse.
2 YOLOv8 An Efficient 80-Class Real-Time Object Detection System. Real-time object detection is essential in various applications, such as autonomous vehicles, surveillance, and robotics. Propose YOLOv8, an optimized deep learning architecture that balances accuracy and speed, allowing for real-time object detection across a wide range of object categories.
3 Computer Vision Based Mouse Control Using Object Detection and Marker Motion Tracking Traditional computer mice can be unsanitary and pose a risk for disease transmission, especially in shared or public computing environments. Combine object detection and marker-based motion tracking in a computer vision system to enable users to control the mouse cursor through hand or marker movements, offering a touchless and precise interaction method.
5
RV College of Engineering
Literature Survey
No Title Problem Statement Solution
4 Virtual Mouse using Hand Gestures Develop a virtual mouse system that allows users to control a computer cursor through hand gestures, addressing the need for intuitive and touchless interaction. Implement computer vision-based hand gesture recognition and mapping algorithms to accurately interpret user hand movements as mouse commands.
5 Vision-Based Interpretation of Hand Gestures for Remote Control of a Computer Mouse the challenge of remote computer control by designing a vision-based system capable of interpreting hand gestures for precise and intuitive cursor manipulation. Utilize deep learning and computer vision techniques to recognize and translate hand gestures into mouse movements and clicks, enhancing accessibility.
6 Examination of eye-hand coordination using computer mouse and hand tracking cursor control Investigate the coordination between eye movements and hand-controlled cursor movements to gain insights into user behavior and performance. Conduct experiments and analyze eye-hand coordination data to understand how users adapt their visual attention and motor control when using traditional computer mice versus hand tracking cursor control systems.
6
RV College of Engineering
Research gap
Real-time Performance Improvement Explore
techniques to enhance the real-time performance
of YOLO-based hand tracking models, especially in
scenarios with multiple hands or complex
interactions. Robustness to Occlusions and
Clutter Investigate methods to improve the
robustness of YOLO models in handling occlusions,
clutter, and challenging environmental
conditions, which are common in real-world
scenarios. Adaptation to Varied Environments
Examine how well YOLO-based hand tracking models
generalize across different environments,
lighting conditions, and camera setups. This is
important for deploying these models in
diverse applications. Temporal Consistency and
Tracking Investigate techniques to improve
temporal consistency in hand tracking. Tracking
hands over time in real-world scenarios involves
dealing with occlusions, abrupt movements, and
changes in hand gestures. Privacy-Preserving Hand
Tracking Address the challenge of privacy
concerns in hand tracking applications. Explore
methods that can perform effective hand tracking
while preserving the privacy of individuals, such
as through the use of privacy-enhancing
technologies like federated learning or on-device
processing without the need for transmitting raw
image data to a central server.
7
RV College of Engineering
Objectives Methodology

• Develop an advanced hand tracking mouse system
• Overcome limitations of the existing model
• Achieve remarkable accuracy and swift cursor
  movement
• Enhance responsiveness to hand movements

8
RV College of Engineering
Novelty
Efficient Architecture Modifications Investigate
modifications to the YOLO architecture to make it
more efficient for hand tracking. This could
involve changes in the number of layers,
introducing skip connections, or utilizing
attention mechanisms tailored for hand detection.
Multi-Scale Feature Fusion Explore techniques
for better multi-scale feature fusion in YOLO for
capturing both global and fine-grained
information about hand poses. This could improve
the accuracy of hand tracking, especially in
scenarios with varying hand sizes and
poses. Temporal Consistency Integrate temporal
consistency into the YOLO model to improve
tracking performance over time. This may involve
incorporating information from previous frames to
enhance the model's ability to handle occlusions
and sudden hand movements. Spatiotemporal
Modelling Extend YOLO for hand tracking into a
spatiotemporal model that considers not only the
current frame but also the temporal context. This
can help in handling dynamic hand movements and
gestures. Domain-Specific Adaptations
Investigate how domain-specific adaptations can
be made to YOLO models for hand tracking. For
example, optimizing the model for specific
applications such as sign language recognition or
human-computer interaction.
9
RV College of Engineering
Design and Implementation
10
RV College of Engineering
Result Analysis
Table Accuracy Comparison Hand tracking
Dataset/models Mediapipe EHT
Sample dataset 1 95.65 98.3
Sample dataset 2 94.63 99.35
Sample dataset 3 98.26 98.54
Sample dataset 4 96.23 97.41
The results show that the EHT model generally
outperforms the Mediapipe model, with higher
accuracy and precision for all four sample
datasets. However, there is some variation in the
results, with the EHT model not always having a
significant advantage over the Mediapipe
model. Overall, the table suggests that the EHT
model is a more accurate and precise model for
the four sample datasets tested. However, the
Mediapipe model may still be a good choice for
some applications, especially if it is less
computationally expensive than the EHT model.
11
RV College of Engineering
Result Analysis
Table Response time of mouse movement
Dataset/models Mediapipe (ms) EHT (ms)
Sample dataset 1 30 20
Sample dataset 2 24 18
Sample dataset 3 40 17
Sample dataset 4 35 29
Response Time Comparison  In Sample Dataset 1,
EHT achieves an accuracy of 98.3, surpassing
Mediapipe's accuracy of 95.65. Sample Dataset 2
shows similar results, with EHT achieving an
accuracy of 99.35, while Mediapipe achieves an
accuracy of 94.63. In Sample Dataset 3, EHT's
accuracy remains high at 98.54, while Mediapipe
achieves an accuracy of 98.26. Table 2 and
Figure 4 is deliberate the response time
comparison of existing and proposed model.
12
RV College of Engineering
Conclusion
In Conclusion, this project represents a
substantial advancement in the field of
human-computer interaction, harnessing the
capabilities of YOLOv8 Hand Tracking for an
Enhanced Virtual Hand Tracking Mouse. The
implemented model has demonstrated outstanding
accuracy, adaptability, and efficiency,
establishing itself as a valuable tool for
enhancing user experiences and accessibility
across diverse digital platforms. The successful
integration of YOLOv8 technology into virtual
hand tracking showcases the potential for
transformative applications in the realm of
human-computer interaction.
13
RV College of Engineering
References

• Duh, E. S., Koceska, N., Koceski, S. Game-based
  learning educational game Azbuka to help young
  children learn writing Cyrillic letters.
  Multimedia tools and applications, 76,
  14091-14105. (2017)
• OConnor, E. Virtual reality bringing education
  to life. Games and Simulations in Teacher
  Education, 155-167. (2020)
• Seinfeld, S., Feuchtner, T., Maselli, A.,
  Müller, J. User representations in human-computer
  interaction. HumanComputer Interaction, 36(5-6),
  400-438. (2021)
• Li, P., Zheng, J., Li, P., Long, H., Li, M.,
  Gao, L. Tomato Maturity Detection and Counting
  Model Based on MHSA-YOLOv8. Sensors, 23(15),
  6701. (2023)
• Hashiguchi, S., Shibanoki, T. A Task Estimation
  Method Based on Image Recognition and Its
  Application to EMG Prosthetic Hand Control. In
  2023 9th International Conference on Virtual
  Reality (ICVR) (pp. 147-154). IEEE. (2023)
• Joson, N. M., Jayakumar, A. Modern Technology
  Usage For Education Field During COVID-19
  Statistical Analysis. In 2022 3rd International
  Conference on Communication, Computing and
  Industry 4.0 (C2I4) (pp. 1-7). IEEE. (2022)
• Liaw, S. Y., Soh, S. L. H., Tan, K. K., Wu, L.
  T., Yap, J., Chow, Y. L., Wong, L. F. Design
  and evaluation of a 3D virtual environment for
  collaborative learning in interprofessional team
  care delivery. Nurse education today, 81, 64-71.
  (2019)
• Bogicevic, V., Seo, S., Kandampully, J. A., Liu,
  S. Q., Rudd, N. A. Virtual reality presence as
  a preamble of tourism experience The role of
  mental imagery. Tourism Management, 74, 55-64.
  (2019)
• Wu, D., Yang, Z., Zhang, P., Wang, R., Yang, B.,
  Ma, X. Virtual-Reality Inter-Promotion
  Technology for Metaverse A Survey. IEEE Internet
  of Things Journal. (2023)

14
RV College of Engineering
References

• Huang, J. M., Ong, S. K., Nee, A. Y.
  Visualization and interaction of finite element
  analysis in augmented reality. Computer-Aided
  Design, 84, 1-14. (2017)
• Shibly, K. H., Dey, S. K., Islam, M. A.,
  Showrav, S. I. Design and development of hand
  gesture based virtual mouse. In 2019 1st
  International Conference on Advances in Science,
  Engineering and Robotics Technology (ICASERT)
  (pp. 1-5). IEEE. (2019)
• Green, T. D., Donovan, L. C. Learning anytime,
  anywhere through technology. The Wiley handbook
  of teaching and learning, 225-256. (2018)
• Xu, M., Ng, W. C., Lim, W. Y. B., Kang, J.,
  Xiong, Z., Niyato, D., Miao, C. A full dive
  into realizing the edge-enabled metaverse
  Visions, enabling technologies, and challenges.
  IEEE Communications Surveys Tutorials. (2022)
• Wang, Z., Hou, Y., Jiang, K., Dou, W., Zhang, C.,
  Huang, Z., Guo, Y. Hand gesture recognition
  based on active ultrasonic sensing of smartphone
  a survey. IEEE Access, 7, 111897-111922. (2019)
• Wu, F., Wu, B., Zhao, D. Real-time measurement
  of individual tree structure parameters based on
  augmented reality in an urban environment.
  Ecological Informatics, 102207. (2023)
• Sree, S. R., Vyshnavi, S. B., Real-world
  application of machine learning and deep
  learning. In 2019 International Conference on
  Smart Systems and Inventive Technology (ICSSIT)
  (pp. 1069-1073). IEEE. (2019)

15
RV College of Engineering
Thank You