Learning Robot with Minimal Requirement

1
Learning Robot with Minimal Requirement

• Wei Qin

2
Introduction

• A real project of Entertainment Robotics Company
• Developing an AI learning system for a toy robot
• The toy robot prototype
• To have more interaction
• with the players

3
Introduction

• The thesis project is to find a good learning
  algorithm for the prototype via software
  simulation.
• Study different learning algorithms
• Select some algorithms for the case
• Implement in software simulation and test
• Transfter to the real robots
• Discuss and draw a conclusion

4
Artificial Intelligence

• Combination of computer science, physiology and
  philosophy
• Concern with the rational action
• AI is viewed as the study and constuction of
  rational agents

5
Adaptive Robotics

• Traditional robotics
• Top-down design (sense-model-plan-act)
• Static environment, world model
• Adaptive robotics
• Approach for animate conditions
• Control architecture
• Artificial Neural Network
• Behavior-based approach
• Reinforcement learning

6
Artificial Neural Network

• Neuron and artificial neuron
• Node characteristics
• Activation function g linear, step, sigmoid

7
Artificial Neural Network

• Connectivity
• Feed-forward network
• Full connected
• Learning alrogithms for ANN
• Supervised leraning algorithm
• Error back propagation
• Nonsupervised learning algorithm
• Evolutionary algorithm

8
Error Back Propagation

• EBP network structure
• Activation function is differentiable

9
Error Back Propagation

• Feed forward processing
• Back propagation of error
• Error for output node
• Error for hidden node
• Modify the weights

10
Genetic Algorithm

• Darwinian Evolution principles
• Genotype
• Process
• Initialize the population P
• Evaluate each individual in P
• Do
• Select the parent individuals from P
• Reproduce with the parent individuals
• Mutate and get new individuals P
• Evaluate each individuals in P
• While (the terminate condition is not
  satisfied)

11
Behavior Based Approach

• Aim at multiple goals, multiple sensors,
  robustness and extensibility
• Subsumption Architecture
• Behaviors are represented as separate layers
• Augmented Finite State Machine model
• Competitive coordination

12
Design with Subsumption Architecture

• Specify all the behaviors needed for the task
• Decompose the qualitative behavior and make them
  independent
• Determine the granularity of the behaviors and
  ground the low level behaviors onto sensors and
  actuators

13
Task of the project

• Interaction with the player
• 1. Let robots play autonomous
• 2. The player has the possibility to change the
  robots behavior
• 3. The player can develope his own robot
  behaviors
• The controller design should be flexible
• The simulation can also present to the users

14
The Prototype

• Multiple Motors
• 2 arm motors, 2 driver motors
• Multiple Sensors
• 3 IR sensors
• Hit sensor
• Microprocessor
• 9 motor commands each
• Sensor data is send in ascii

15
Algorithm Selection

• Genetic Algorithm
• Can realize task 1
• Flexible design
• Behavior-based mode
• A direct way for task 1
• Provide limited interaction with player
• Error Back Propagation
• Select the target predicted sensor data, good
  motor commands or commands from the player

16
Simulator

• Mathematical description (not use)
• Look-up table
• IR sensors
• Driver motors
• Fight and hit behavior
• Framework
• Based on version 0.3.2

17
Genetic Algorithm

• Neural Network Structure
• 
  (j0,1)

18
Genetic Algorithm

• Fitness Formula
• Sum of the sensor data with emphasis on the
  middle
• Extra points for fight and hit behavior
• Elite selection strategy
• Co-evolution
• Mutation rate
• High rate and low rate

19
Genetic Algorithm

• Method 1 training with 1 initial position
• Method 2 training with 3 initial positions
• Initial positions
• position 1 position 2
  position 3
• The difficulty is decreased.
• Population and the developing time

20
Genetic Algorithm

• Different mutation rate with method 1
• GA can find good solutions with the initial
  condition it is trained for
• The training will take some time

21
Behavior-Based

• State Figure

22
Behavior-based

• Subsumption structure

23
Behavior-based

• Level 1 wander
• Random command
• Genetic Algorithm for exploration discard
• Level 2 move to
• Turn to the side with higher sensor data
• Level 3 fight to
• Fight and may move forward or backward
• Level 4 hit
• Fight and not move any more

24
Behavior-based

• Preliminary Test with initial position 1

25
Error Back Propagation

• Target is the command from behavior-based mode
• Neural network structure

26
Error Back Propagation

• Learning rate selection
• Training times
• 5 times per training 20 times per
  training

27
User Interface

• Select algorithms
• Set the algorithms parameters
• Decide initial positions
• Provide good weights for GA and EBP
• View simulation

28
Experiments

• Test from 3 initial positions, each for 10 times
• Compare the average speed of each algorithm in
  each initial position
• Compare the sucessful rate of each algorithm (the
  generalization of the algorithm)
• A special comparison of GA and EBP

29
Test results

• GA method 1
• Training with initial position 1
• Affected by the initial training position

30
Test results

• GA method 1
• Training with initial position 2, 3 have the
  similar results.
• When test with the initial position where it is
  trained the result is good and 100 sucessful.
• When test with other initial position, more than
  50 will be failed.
• GA method 1 will find the solutions specifically
  for the condition that it is trained.

31
Test results

• GA method 2
• Similar testing results in each initial positions

32
Test results

• Behavior-based mode

33
Test results

• Error Back Propagation
• Trained with initial position 1
• Not affected by the initial training position

34
Speed comparison

• Behavior-based method is the fast one. And the
  average speed is decreased from p1 to p3
• GA method 1 the speed is fast in the position it
  is trained, not so well in others
• GA method 2 has similar average speed in 3
  positions
• EBP has a comparative slow speed learn from
  others and may need more training time

35
Generalization comparison

• Behavior-based mode has 100 sucessful rate in
  each positon
• GA method 1 is only good at where it is trained
• GA method 2 gives a similar rate in each position
• EBP gives a similar rate and a little higher than
  GA method 2

36
Conclusion from the comparison

• Behavior-based method is the best one to let the
  robot fight autonomously in simulation.
• In general GA method 2 gives a better solution
  than GA method 1
• Training with multiple initial positions can find
  a more general solution for different conditions
• EBP is a good algorithm to learn from sth

37
GA and EBP Comparison

• EBP has a shorter training time and better
  generalization
• GA method 2 gives a faster solution for the task

38
Behavior-based mode with noise

• Noise is added to the output motor (10)
• Some kind specific to this task and this simulator

39
Transfer to real robot

• Reprogram in C language
• Real robot testing
• The frame is 11 m
• The robots start
• from corresponding
• initial position as in
• simulation

40
Preliminary test

• Results of behavior-based mode
• In similar initial position 3, sometimes the
  robots can come close and fight, may hit once or
  two
• Failed with similar initial position 1 and 2
• Reason gap between the simulation and the real
  world
• Further work
• Reset the fight threshold and the level division
  standard
• Rearranged the commands given in level 2 and 3

41
Preliminary test

• Genetic Algorithm
• Some results of similar initial position 3 are
  good
• Video 1 Video 2

42
Test with real robot

• Video 3
• Failed with initial position 1 and 2
• Further work Train with real robot

43
Analyse the results

• GA gives a better performance in initial position
  3
• For further adjustment, GA need less change
• GA is more adaptive to the environment

44
Discussion

• Results of software test
• Behavior-based approach is the best
• Results of real robot test
• Genetic Algorithm has more advantages
• Interaciton
• EBP can provide an extensive interaction
  possibility
• Further development
• GA and EBP have advantages

45
Conclusion

• GA and EBP are recommended
• Both have good structure for further development
• GA has good adaptivity and not vulnerable to the
  small environment changes
• EBP can provide an extensive interaction
  possibility

46
Thank you!