Dr Changjiu Zhou

1
Learning and Control of Biped Locomotion
Dr Changjiu Zhou School of Electrical
Electronic Engineering Singapore
Polytechnic zhoucj_at_sp.edu.sg www.robo-erectus.org
2
Outline

• Introduction
• Biped Walking Cycles
• How to Control Biped Locomotion
• How to Plan/Learn Biped Gaits
• Biped learning by reinforcement
• Some Research Topics

3
Biped Gait (Frontal Plane)
Biped Gait (Frontal View)
4
Biped Gait (Sagittal Plane)
5
Finite State Machine for Biped Walking Control
6
Static Walking

• In static walking, the biped has to move very
  slowly so that the dynamics can be ignored.

• The bipeds projected center of gravity (PCOG)
  must be within the supporting area.

7
Dynamic Walking

• In dynamic walking, the motion is fast and hence
  the dynamics cannot be negligible.

• In dynamic walking, we should look at the zero
  moment point (ZMP) rather than PCOG.

• The stability margin of dynamic walking is much
  harder to quantify.

8
Why is Biped Robotics Hard?

• Unpowered DOF between the foot and ground

• This constraint limits the trajectory tracking
  approaches used commonly in manipulators research.

9
Biped Control Model-based
Feet position and ZMP (PCOG)
Inverse kinematics model
Desired joint angles
Biped Robot
10
Biped Control Model-based

• Except for certain massless leg models, most
  biped models are nonlinear and do not have
  analytical solutions.
• Massless leg model is the simplest model. The
  body of the robot is usually assumed to be point
  mass and can be viewed to be an inverted
  pendulum.
• When the leg inertia and other dynamics like that
  of the actuator, joint friction, etc. are
  included, the overall dynamic equations can be
  very nonlinear and complex.

11
Example Massless leg model

• The simplest biped model
• Some assumptions, e.g.,
• From DAlemberts principle

12
Biped Control Biologically Inspired

• Since none of the humanoid robots match
  biological humanoids in terms of mobility,
  adaptability, and stability, many researchers try
  to examine biological bipeds so as to extract
  certain algorithms that are applicable to the
  robots.

Reverse Engineering
13
Biped Control Biologically Inspired
Two Main Research Areas

• Central Pattern Generators (CPG)
• Passive Walking

14
ZMP-based Gait Planning

• Plan the hip and ankle trajectories according to
  walking constraints and ground constraints.
• Derive all joint trajectories by inverse
  kinematics.

15
Example Gait Planning for Walking on Slope
- Plan gait using 3rd order Spine which
guarantees the continuity of both 1st derivative
and 2nd derivative.
16
Example Planning Results
Consecutive walking gait along slope
Joint angles
17
IP-based Gait Planning

• The dynamic equation of the IP model

• If the angle is small, it can be simplify as a
  linear homogeneous 2nd order differential
  equation

18
3D Linear Pendulum Model
19
Example IP-based Gait Planning
20
Biped Kicking

• Kicking constraints
• Kicking range
• Friction

21
Kicking Pattern
22
Biped Learning by Reinforcement (1)

• A humanoid robot aims to select a good value for
  the swing leg parameters for each consecutive
  step so that it achieves stable walking.

• A reward function that correctly defines this
  objective is critical for the reinforcement
  learning.

23
Biped Learning by Reinforcement (2)

• The control objective of the gait synthesizing
  for biped dynamic balance can be described as
• To evaluate biped dynamic balance in the frontal
  plane, a penalty signal should be given if the
  biped robot falls down in the frontal plane

24
Biped Learning by Reinforcement (3)
Reinforcement Learning with Fuzzy Evaluative
Feedback
25
The RL Agent

• AEN - the action-state evaluation network
• ASN - the action selection network
• SAM - the stochastic action modifier

• Both the AEN and ASN are initialized randomly.
• Learning starts from scratch.
• It needs a large number of trials for learning.

26
The FRL Agent

• Neural fuzzy networks are used to replace the
  neuron-like adaptive elements.
• The expert knowledge can be directly built into
  the FRL agent as a starting configuration.
• The ASN and/or AEN could house available expert
  knowledge to speed up its learning.

27
The FRL Agent with Fuzzy Evaluative Feedback

• The numerical evaluative feedback is not the
  biological plausible.
• The fuzzy evaluative feedback is much closer to
  the learning environment in the real world.
• The fuzzy evaluative feedback is based on a
  form of continuous evaluation.

28
Comparison of FRL Agents
29
Information Available for Biped Gait Synthesizing
30
The Gait Synthesizer Using Two Independent FRL
Agents
31
Before and After Learning
Ankle joint
Knee joint
32
Results (1)
The ZMP trajectory after FRL (type C)
33
Results (2)
Walk (Backward)
34
Some Research Topics

• Online gait generating
• Online footprint planning
• Constraints
• ZMP constraint for stable walking
• Friction constraint for stable walking
• Current Challenges
• Knee bending
• Body shifting

35
References

• C. Zhou, Robot learning with GA-based fuzzy
  reinforcement learning agents, Information
  Sciences 145 (2002) 45-68.
• C. Zhou, Fuzzy-arithmetic-based Lyapunov
  synthesis to the design of stable fuzzy
  controllers a computing with words approach,
  Int. J. Applied Mathematics and Computer Science
  12(3) (2002) 101-111.
• C. Zhou and Q. Meng, Dynamic balance of a biped
  robot using fuzzy reinforcement learning agents,
  Fuzzy Sets and Systems 134(1) (2003) 169-187.
• C. Zhou, P.K. Yue, Z. Tang and Z. Sun,
  Development of Robo-Erectus A soccer-playing
  humanoid robot, Proc. IEEE-RAS Intl. Conf. on
  Humanoid Robots, CD-ROM, 2003.
• Z. Tang, C. Zhou and Z. Sun, Gait synthesizing
  for humanoid penalty kicking,  Dynamics of
  Continuous, Discrete and Impulsive Systems,
  Series B, (2003) 472-477.
• D. Maravall, C. Zhou and J. Alonso, Hybrid fuzzy
  control of inverted pendulum via vertical
  forces, Int. J. of Intelligent Systems, 2004 (in
  press).

36
Acknowledgements

• Staff Member
• P.K. Yue, F.S. Choy, Nazeer Ahmed
• M.F. Ercan, Mike Wong, H. Li
• Research Associate
• Z. Tang (Tsinghua U.), J. Ni (Shanghai Jiao Tong
  U.)
• Technical Support Officer
• H.M. Tan, W. Ye
• Students
• P.P. Khing, H. W. Yin, H.F. Lu, H.X. Tan, J.X.
  Teo,
• Stephen Quah, H.M. Tan, Y.T. Tan

37
Thanks! Dr Changjiu Zhou School of Electrical
and Electronic Engineering Singapore
Polytechnic zhoucj_at_sp.edu.sg www.robo-erectus.org