IETE 52nd Annual Technical Convention on Technology and Terror Role of ICT in War Against Terror

1
IETE 52nd Annual Technical Convention on
Technology and Terror- Role of ICT in War
Against Terror

• Criminal Information Analysis using Facial
  Expression with ICT
• By
• G R Manza, P L Yannawar, M Karbhari, S C
  Mehrotra
• Human Computer Interaction Lab, Department of
  Computer Science and IT, Dr. Babasaheb Ambedkar
  Marathwada University, Aurangabad (MS), India
• Dy. Superintendent of Police, Osmanabad (MS),
  India
• 26-27 Sept 2009, Mumbai

2
Agenda

• Introduction
• Facial Expression
• Experimental Setup
• Methodology
• Result and Discussion
• References

3
Introduction

• The use of computers is enormous and widespread
  in social life peoples are using computers for
  various purposes
• The research in the computer can be extended now
  a day in such a way that the computer can speak
  as well as understand the voice/speech of the
  user, authenticates user using Biometric signs
• The question is can computer is used to identify
  the expressions and behavioral changes observed
  in subject (person) committing some antisocial
  activities.

4
Cont..

• This question is answered by facial expression
  analysis and recognition method adopted in ICT.
• Use of Human Computer Interaction is an upcoming
  field in Information Communication Technology
  (ICT) and Related Applications

5
Facial Expression

• Facial Expression is the science of expressing
  the emotions/feelings to convey the information
• Facial Expression is the most powerful natural
  and immediate means for human beings to
  communicate their emotions and intentions
• The expression of the Face is the basic mode of
  non verbal communication among people

6
Understanding Face
A
B
C
D
E
F
G
G
H
H
I
J
A Glabella, B Root of Nose , C Eye Cover
Fold, D Lower eyelid Furrow, E Infraorbital
Furrow, F Nostril wing, G Infraorbital
Triangle, H Nasolabial Furrow, IPhiltrum, J
Chin Boss
7
Face Component Description

• Glabella Area of the forehead between eyebrows
• Root of Nose Nasal Root, Beginning of the nose
• Eye Aperture Eye opening
• Eye Cover fold The Skin between the eyebrows and
  palprebral part of the upper eyelid (the part
  that contacts the eye ball)
• Lower eyelid furrow A Place below the lower
  eyelid where a line or wrinkle may appear
• Infrorbital furrow A Place where a line or
  wrinkle may appear parallel to and below the
  lower eyelid running from near the inner corner
  of eye and following the bone laterally

8
Cont..

• Nostril Wings The fleshy skin of the side of the
  nose that forms the outside of each nostril.
• Nasolabial furrow A place where a line or
  wrinkle may appear which begins adjacent to
  nostril wings and runs down and outwards beyond
  the lip corners.
• Philtrum The vertical depression in the center
  of the upper lip directly under the lip of the
  nose
• Elongate The mouth appears to be longer than
  usual in the horizontal plane

9
Cont..

• Narrow The red part of the lip is less visible
  or narrower than usual
• Widen The red part of the lip is revealed more
  or wider than usual
• Flatten The lips appear flattened against the
  teeth
• Protrude The Lips come forward or out away from
  the face more than usual.
• Tighten The Lips appear tight, the lips are not
  relaxed or loose. The muscle with in the lips has
  contracted.

10
Cont..

• Stretch The lips are pulled and skin stretched
  like a rubber band
• Rolled In The lips are turned or rolled,
  inwards, disappearing entirely or almost
  entirely, but they are nor tightened, pressed or
  stretched

11
Facial Action Units

• What do we mean by Facial Action Units ?

• Muscular activity that produces momentary changes
  in facial appearance
• A System which could distinguish all possible
  visually distinguishable facial movements
• It was designed primarily to measure facial
  movements relevant to emotions.
• A Measurement scheme that could distinguish among
  all visible facial behavior
• A Comprehensive system could be obtained by
  discovering how each muscle of the face acts to
  change visible appearance

12
Cont..

• Are primarily composed of
• Upper Face action units affects on Eyebrows,
  forehead and eyelids
• Lower face action unit presented in five groups
• Up/Down
• Horizontal
• Oblique
• Orbital
• Miscellaneous actions

13
Upper Facial Action Units

• Muscles Underlying upper Face Action units

14
Cont..

• AU1 Trace of brow raising at inner corners,
  pull the inner portions of the eyebrows upwards
• AU2 Trace of brow raising at outer corners, pull
  the lateral portion of the eyebrows upwards
• AU4 Trace of brow lowering and/ or trace of
  pulling together, lowers the eyebrow, pushes the
  eye cover fold downwards, pull the eyebrow closer
  together.
• AU6 Raises the infraorbital triangle, lifting
  the chicks upwards, chick raises and lid
  compressor

15
Cont..

• AU7 Tighten eyelids, narrows eye aperture,
  rising of the lower lid, stretching of the lower
  eyelid, pulling the skin below the lower eyelid
  towards root of the nose
• Sample Face showing Upper Action Units

AU1,AU2,AU4,AU6,AU7 Holds
AU4,AU7 Holds
AU4,AU6,AU7 Holds
AU4,AU6,AU7 Holds
16
Lower Facial Action Unit

• Locations of Lower face up/down action unit

17
Cont..

• AU9 Trace of Infraorbital triangle raise with
  skin drawn medially towards the bridge of the
  nose, causing infraorbital furrow to wrinkle,
  pulls the center of the upper lid upwards, nose
  wrinkler
• AU10 Upper lip riser, Center of Upper lip is
  drawn straight up, causes an angular bend in the
  shape of the upper lip.
• AU15 Lip corner depressor, pulls the corner of
  the lip down, angled down at the corner

18
Cont..

• AU16 Trace of the lower lip, skin below the
  lower lip or chin boss actually pulled down, not
  merely a passive lip part.
• AU17 Chin raiser, pushes the chin boss upwards,
  pushes the lower lip upwards, may causes wrinkles
  to appear on the chin boss as skin is stretched,
  may produce depression medially under the lower
  lip.

19
Cont..

• Sample Images representing Lower Action Unit

AU9, AU10, AU15, AU16, AU17 Holds
AU9, AU10, AU15, Holds
AU15, AU16, AU17 Holds
AU15, AU16, AU17 Holds
20
Composed Action Units

• AU5 Trace of Upper lid raising that exposes
  more of the iris or sclera, wideness the eye
  aperture
• AU8 Lips towards each other, the upper lip is
  pulled down towards the lower lip but not pulled
  back over the teeth, lower lip is pulled up
  towards the upper lip
• AU11 Pulls the Upper lip upwards and laterally
  towards small extends at the mid point between
  the philtrum and outer lip corners
• AU12 Lip corner puller, pulls the corner of the
  lip back and upwards

21
Experimental Setup

• The Experimental Setup requires
• Computer
• Camera for Recording Images
• Processing Facial Images
• Identification of Action Units from Face Images
• Recording Action Units and building Mathematical
  Expression

22
Cont..
23
Methodology

• When a person committed anti social activity and
  Investigation agency like police department
  arrest subject (suspect/person), they maintains
  personal history with behavioral characteristics
  of each subject as a record
• Information Recorded as
• Personal Details (Name, Address, Gender, Alias,
  Nationality etc.)
• Physical Examinations
• Socio-economic status

24
Cont..

• With above experimental setup physical features
  related to face has been captured
• These physical feature parameters related to face
  includes Forehead, Eyebrows, Eyes, Nose, Ear,
  Mouth, Lips, Chin, and Nasolabial Furrow.

25
Result and Discussion

• At the time of interrogation, interrogator has
  identified some behavioral changes and same can
  be recorded with above experimental setup as
  Symptoms which plays an important part in
  interrogation, there are some of the symptoms
  that happens on the suspects during the
  interrogation.
• Symptoms like
• Perspiration
• Paleness of the skin,
• Increase or decrease of pulse rate which can be
  seen easily in the visible veins
• Dryness of Mouth and Tongue
• Changes in Respiration
• Muscle spams

26
Cont..
System can Identify (Lie, Involvement in Crime,
Force to perform crime)

• Symptoms Cont..
• Licking lips
• Movements of Eyes
• Shuttering in Speech
• Strong protestation of innocence
• Inability to look at interrogator straight
• Averting Glances
• Faculty memory hesitation
• Long Pauses
• General nervousness affecting whole body

27
Conclusion

• The Experiment carried out in this paper resulted
  that, there will be no difference between normal
  and criminal faces on the basis of N.C.R.B, Form
  No III which has been collected by the
  investigation agency and corresponding Facial
  Action units identified on the basis of FACS
  coding System
• From the above facts it is necessary to capture
  the live feed at the time of interrogation and
  the facial expression variation should be
  recorded and processed through Facial Action unit
  system

28
Future Work

• This kind of system has to be developed in
  greater extent professionally with ICT initiative.

29
References

• Paul Ekman and Terrence J Sejnowski, NSF Report-
  Facial Expression Understanding
• Changbo Hu, Ya Chang, Rogerio Feris and Matthew
  Turk, Manifold Based analysis of facial
  Expression, University of California.
• John Ghent, John McDonald, A Computational Model
  of Facial Expression, A Technical Report Series
  by Department of Computer Science, National
  University of Ireland, NUIM-CS-TR-2004-01.
• Paul Ekman, Wallace V Friesen, Joseph C Hager,
  Facial Action Coding System-Investigators
  guide, ISBN 0-931835-01-1, 2002
• Arrest/Court Surrender form (N.C.R.B, Form No
  III)
• Book on Scientific Interrogation, Study
  material, Central Detective Training School,
  Hyderabad

30
Acknowledgement

• Author would like to acknowledge sincere thanks
  to Dy. Superintendent of Police Mr. Avinash
  Budhawant and his CIB team, Shri Shevgan , PI of
  Crime Branch Police Commissioner Office
  Osmanabad for providing sufficient and relevant
  information, which found to be beneficial for
  constructing this research paper.