Steps towards Integrated Intelligence

1
Steps towards Integrated
Intelligence
Aug. 22nd, 2003

• Naoyuki OKADA
• (Professor Emeritus)
• Kyushu Institute of
  Technology

2

• Progress
• Step 1 Conceptual taxonomy of
• vocabulary
• Step 2 Natural language understanding
• of moving picture patterns
• Step 3 Emotion processing vs
• knowledge processing
• Step 4 Integrated intelligence

3
1.Introduction

• The history of the research of Arti-ficial
  Intelligence(AI) is repetition of diversification
  and specialization of its fields as other
  research does.
• At the beginning of 1960s
• At the beginning of 2000s

• 1960s
• Natural lang. process.
• Pattern recognition
• Learning
• problem solving

4
1.Introduction

• 2000s
• Fundamentals/Theory
• Knowledge representation, reasoning,
  algorithm ,
• fuzzy theory, ---
• Learning/Discovery
• Inductive/deductive learning, example-based
  reasoning, data-mining, ---
• Infrastructure of knowledge
• Knowledge acquisition, knowledge base,
  Web search, -
• AI architecture/language
• Agent/Distributed AI
• Problem solving by collaboration, agent society,
  ---
• Life/Brain system
• Artificial life, genetic algorithm,
  connectionism, ---
• Natural language
• Natural language understanding, dialog
• processing, corpus, speech recognition,
  ----
• Pattern understanding
• Image recognition, scene analysis, image sequence
  processing,---

• The history of the research of Arti-ficial
  Intelligence(AI) is repetition of diversification
  and specialization of its fields as other
  research does.
• At the beginning of 1960s
• At the beginning of 2000s

5

• However, too much diversification and
  specialization weaken the study on the relations
  among subfields.
• Those relations are important above all in
  human intelligence.
• So, we should sometimes stop, look back, put
  various kinds of results in order, and integrate
  them into a system.

6
Approach towards integration

• Multi-modal
• Human intelligence accepts multi-modal inputs.
• - Natural language in letters/voices
• - Picture patterns

7

• Intellect and sensitivity
• Knowledge and emotions are in the
  relationship of both wheels of a cart.

8
2.Conceptual taxonomy of vocabulary

• Language is the window of the mind.
• Semantic contents of language, or the system of
  concepts is the most important objects in making
  clear intelligence.

9

• Research in Early years
• C.J.Fillmore 68 Case grammar
• M.R.Quillian 68 Semantic network
• R.Schank 72 Conceptual dependency
• Y.Wilks 75  Preference semantics

10
Conceptual analysis

• Categories of concepts
• Concepts are formed for all the nature.
• - There are five categories from the linguistic
  viewpoints substance, attribute, event,
  space/time, and miscellaneous
• - But each category is vague.

11

• Computational definition
• - What is substance?
• Individual of which quantity and quality can be
  recognized by sensors
• Fig. 2?1 Substance sensed by eyes
• ---Mountain

12

• - What is state?
• Fundamentally, static relation among several
  substances
• Fig.2?2 State----Man in the car

13

• - What is attribute?
• A special case of state. Fundamentally,
• difference between object and standard

Difference

Object
Standard
Fig. 2?3 Object-standard pair---
The mountain is higher than the tree.
14

• - A measure is necessary for the detection of
  difference
• Measure Height (length in the perpendicular
  direction)
• This measure brings an attribute to the object.

15

• - What is event ?
• Fundamentally, change from a before- state to
  an after-state

Change
???
???
Before-state
After-state
Fig. 2?4 Before-after state pair--- A man gets
out of a car
16

• - What is space and time ?
• Fundamentally, the location of substance,
  attribute or event is identified.
• Spaceposition
• Time passage

17

• Primitive and complex
• - Primitive
• A concept which can not be decomposed
• any more(by referring to its word)
• - Complex
• A concept which can be decomposed
• into one or more primitives

18

• Formation of complex concept
• - Compound
• Type A Two primitives are connected
• with a logical/syntactic relation.
• Type B Primitives are connected with
• a scenario
• - Derivative
• Derived from a primitive

19
Conceptual classification

• Why classification?
• - Verification of the proposed theory
• - Acquisition of conceptual data for machine
  processing
• Target vocabulary
• - About 32,000 words used in everyday language

20

• Results of classification

?(river)
attribute
whole/part
event
N13
N12
N11
flow
stagnate
??
??
??
???
???
???
??
?
?
?
?
?
?
?
?? (upper stream)
??
??
??
?? (brook)
?? (pool)
?
?? (lower course)
?? (main stream)
?? (torrent)
?
??
?? (big river)
?? (tributary)
?? (midstream)
??
??
Fig. 2?5 Network of substance concepts
21

• Table 2?1 Primitives of attribute/event

No. Subcategory (attribute/event) Examples (attribute/event)
0?00 Spirit/change_in_spirit Glad/get anger
0?01 Sense/change_in_sense Cold/hurt
1?00 Location/change_in_location Deep/fall
1?01 Direction/change_in_direction Diagonal/turn over
1?02 Shape/change_in_shape Sharp/bend
1?03 Quality/change_in_quality Soft/rot
1?04 Quantity/change_in_ quantity Many/decrease
1?05 Light/change_in_light Dark/flash
1?06 Color/change_in_color Red/color
22
1?07 Heat/Change_in_heat Hot/cool
1?08 Force?power/change_ in_force?power Strong/strengthen
1?09 Sound/change_in_sound Noisy/sing
1?10 Appearancedisappearance Bare/appear
1?11 Startfinish Sudden/begin
1?12 Time/Change_in_time Quick/pass
2?00 Continuation Constant/continue
2?01 State Fine/tower
3?00 Abstract Equivalent/fit
4?00 Others Eat
23

• Table 2?2 Case-frame of events

Type Example
v(sbj) Fall(leaf)
v(sbj,org) Come(smoke, chimney)
v(sbj,goal) Go(Taro, post office)
v(sbj,ptn) Collide (truck, bus)
v(sbj,std) Resemble(children, parents)
v(sbj,obj) break(boy, cup)
v(sbj,obj,org) Unload(driver, box, truck)
v(sbj,obj,goal) Put(girl, candy, pocket)
v(sbj,obj,inst) Scoop(Hanako, sugar, spoon)
v(sbj,obj,att) Feel(Jiro, breeze, cool)
Others
24

• Number of classified concepts
• Substance 4,200
• Attribute 2,060
• Event 3,720
• Space/time 1,800
• --------------
• Total 11,780

25
Evaluation

• Our theory can cover the 70 of the
• target vocabulary, and almost the
• whole if a little enlarged.
• Fundamental data of concepts was
• obtained, which contributed to the
• construction of EDR concept
• dictionaries later.

26
Main publications

• 1973 N.Okada T.TamatiAnalysis and
  Classification of Simple Matter Concepts for the
  Interpretation of Natural Language and Picture
  Patterns, IECE Trans, Vol.56D,No.9, pp.523-530.
• 1980 N.Okada Conceptual Taxonomy of Japanese
  Verbs for Understanding Natural Language, Proc.
  COLING'80, pp.127-135.

27
3.Natural language understanding of moving
picture patterns

• R.A.Kirsch, Pioneer
• - Kirsch proposed integrated processing through
  the common representation of their meanings
  Kirsch64.
• - But he processed just static picture patterns.

28

• Approaches to moving picture
• patterns in early years
• N.Badler 75 Temporal scene analysis
  Sentence generation as the results
• of temporal scene analysis
• Minsky 75 Frame theory
• Universal data structure, particularly
• representation of event

29

• Our approach
• - Input
• Sequential pictures each of which
• is line drawing by hands
• - Meanings captured
• The events of change_in_location
• which is the biggest in number.
• - Output
• Japanese and English sentences

30

• Flow of processing

start
Picture reading
Bottom up
Noise cleaning
Primitive picture recognition
Reasoning of occurring events
Top down
Structural analysis among primitives
Understanding events
Sentence generation
end
Fig 31 Natural language understanding
of picture sequences
31
Bottom up process

• Picture reading
• A TV camera follows a line segment by
  octagonal scanning.
• Primitive picture recognition
• An input line drawing with graph structure is
  matched with a template just like
  wave-propagation.

32
(a) Octagonal scanning
(b) Line following
Fig 32 Reading a line segment
33
P1
P2
P3
P4
P5
(a) Input drawing
(b) Selected template
Fig. 33 Wave-propagation pattern matching(WPPM)
34
Top down process

• - Context and focus attention
• All the things in a picture are not necessarily
  recognized in a certain context, but some
  attentional objects are focused.

35

• - Attentional rules
• 1.Objects related to a goal in the execution of a
  plan
• 2.Dangerous objects
• 3.Favorite things
• 4. Sudden, big change_in_location/ _shape
• --------

36
((S, thing), time passage, existence's))
move
((OX, thing), existence(OX))
movement_ perpendicularly(S)
(S, movable_ by _oneself)
V3
V1
V2
movement downward (S,(T0,T1))
((S,direction), go_forward(S), come_close(S,OT))
touch(S,OT, T1)
((S ,legs), walking_figure(S))
((OF,inside), inside(S,OF,T0), outside(S,OF,T1))
touch
get out
walk
descend
go
(OF,veihcle))
short_time
collide
get off
Fig.34 Reasoning network of change_in_location
37
Structural analysis

• - Technologies
• Numerical computation
• Logical computation
• Gestalt processing
• Template matching

38

• Logical computation

Sj
Si
Fig.35 Boolean judgment of inside/outside
39

• - Gestalt processing
• Metzgers rule
• Continuation two line segments
• meeting with angle 180
• Enclosure a domain enclosed by
• contours

40
(b) Template
(a) Complex pictures
Fig. 36 Symbol processing
41
Experiments

• Reading and recognition of line drawings by
  hands
• Structural analysis of static pictures
• Natural language understanding (NLU) of
  before-after state pairs
• NLU of picture sequences

42

• (a) Before-state (b)
  After-state
• Generated sentences
• 1)A man(4) moves(1). 6) A man(4)
  heads for a car.
• 2)A man(4) passes(1). 7) A man(4)
  goes to(1) a car.
• 3)A man(4) walks. 8) A man(4)
  comes to(1) a car.
• 4)A man(4) goes forward(1). 9) A man(4)
  gets near a car.
• 5)A man(4) goes out(1)
  -----
• of a house.
• Fig. 37 NLU of a before-after state pair

43
tt0
tt1
tt2
tt3
tt4
tt5
A man(4) is in a house
tt0
A car runs(2).
tt3
- - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
A man(4) goes out(1) of a house
tt1
A car collides with a tree.
tt4
- - - - - - - - - - - - - - - -
tt5
A bird(1) leaves(1) a tree.
A man(4) gets on(1) a car.
tt2
A man(3) get off (1) a car.
- - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
Fig.38 NLU of a picture sequence
44
Evaluation

• Reading and recognition
• About 150 primitive pictures were input, the
  88 of which were correctly recognized and the
  95 of which could be possible by some
  improvement .

45

• Structural analysis of before-after state pairs
• Note that the current image
• processing technology can
• process gray-scale image
• sequences by real-time

46

• Meaning understanding
• Our technology is still useful
• for all the subcategories of
• events except mental one
• Historical significance
• This research took the lead in
• the field of NLU of moving
• picture patterns in 70s.

47
Main publications

• 1976 N.Okada T.TamachiInterpretation of
  Moving Picture Patterns and its Description in
  Natural Language---Semantic Analysis, IEICE
  Trans(D),Vol.J59-D, No.5, pp.331-338.
• 1979 N.Okada SUPP---Under-standing Moving
  Picture Patterns Based on Linguistic Knowledge,
  Proc. IJCAI,pp.690-693.

48
4.Emotion processing vs. knowledge processing

• Why does AI need emotion processing?
• (1) Texts, e.g. social articles in newspapers
  often touch humanity such as glad/sad or
  gain/loss.
• (2) Some intelligent agents should be friendly
  to humans.
• (3) Some kinds of processing need a mechanism
  for evaluation of input information.

49

• Research in early years
• J.G.Carbonell 80
• Story understanding by personality
• Pfeifer Nicholas 85
• Simulation of emotion mechanism
• by interruption
• Okada 87
• Emotion model in NLU

50

• Our approach
• - Evocation and response
• Analysis of general property and algorithm
• - Roles shared by emotion and knowledge

51
Analysis of emotion

• Multi-factor analysis by Plutchik
• Plutchik divided emotions into two
• categories primary and complex
• Plutchik60.
• - We follows this idea, and take the followings
  as primary emotion
• Gladness/sadness, like/dislike,
• surprise, expectancy, anger, and
• fear.

52
(Gladness( the current state is better than the
previous (
physiological (inner pleasure outer pleasure)
psychological ( goal
achievement( information
collection (expected discover
become clear)
plan (planning)
results (completion gain useful))
personal relations( companion
mind (agreement sympathy
collaboration make_friends_again)
superiority/inferiority (superior praise
obedience hospitality
protection))) others))))

• Fig.4?1 Hierarchical features of gladness

53
Evocation of emotion

• - Reflective
• Evoked unconsciously by a sudden stimulus from
  the external world or a remarkable change in the
  internal. Reflective response follows it.
• - Deliberative
• Evoked consciously by a cognitive process.
  Deliberate reasoning mediates between the input
  and its response.

54
Response of emotion

• - General trends
• If one is brought pleasure by an input, one
  promotes the input stimulus through ones
  response,
• otherwise one inhibits it.
• - Type of response
• Free
• Constrained

55

• - Free
• An emotion is evoked straight to a stimulus, and
  a promoting/inhibitory response follows it. The
  response may cause to give up a task under
  execution.
• - Constrained
• Even if a free emotion is evoked internally, some
  task under execution inhibits straight expression

56
Emotion vs. knowledge

• Language expression
• Emotion is adjective whereas
• knowledge is verb
• This implies that emotions are attributes. Since
  an attribute gives a measure to detect the
  difference between an object-standard pair,
  evocation of an emotion is measurement of the
  input stimulus.

57

• Subjective and objective
• Emotion subjective evaluation of
  information
• Knowledge memory of objective information
• Pattern of evaluation
• Formation of personality

58
Experiments

• Simulation of protagonists of fables
• - Free evocation in a series of actions (Shown in
  Chapter 5)
• - Constrained evocation in dialog process

59

• A dialog---invitation
• K1 Hi.
• P2 Hi.
• K3 Where are you going?
• P4 To the river for fishing.
• K5 Sounds good.
• P6 And you?
• K7 Im going to the mansion to drink water of
  the pond. Im very thirsty.
• (continued)

60

• P8 The mansion is dangerous.
• K9 Why?
• P10 Because I heard a voice when I passed it a
  while ago.
• K11 Really?I wonder what shall I do.
• P12 Why dont you come to the river with me ?
• K13 Well, its far, isnt it?
• P14 But the water there is colder and more tasty.
• K15 O.K. Ill come with you.

61
dialogue model
understand (E-PLAN)
persuade_to_abandon(E-Plan)
tentative_ acceptance (R-PLAN)

understand_drawback (R-PLAN)
emphasize_advantage(R-Plan)
utterance planning
intention recognition
accept(R-PLAN)
persuade_to_accept(R-PLAN)
refuse_for_drawback(R-PLAN)
deny_drawback(R-PLAN)
- - -
inform(E-PLAN)
inform_advantage(R-PLAN)
- - -
- - -
action planning
emotion
- - -
language analysis
language generation
message flow
(E7) Im going to the pond in the mansion to - -
-.
top-down prediction
dialogue state tracking
(R12) Why dont you come to the river with me?
(R14) But the water there is colder and more
tasty.
dialogue state transition
(E13) Well, its far.
Fig.4?2 Interaction between discourse and mental
analyses
62
Evaluation

• Conceptual analysis
• The properties of primitive emotions of children
  were made clear.
• Evocation
• The so-called non-logical algorithm was
  clarified.
• Response
• Complicated responses in behavior and dialog
  were verified.

63
Main publications

• 1987 N.Okada Representation of knowledge and
  emotions,Proc. Kyushu Symp. Information
  processing,pp.47-65.
• 1997 M.Tokuhisa N.Okada A Pattern Recognition
  Approach to Emotion Arousal of Intelligent
  Agents,Trans.JSAI,Vol.39, No.8, pp.2440-2451.

64
5.Integrated intelligence

• Intelligence dwells in the mind.
• Recent research in the fields of cognitive
  science(CS) and AI throws light on the
  comprehensive mechanism of the mind.

65
Computer Models of the Mind

• Existent models
• - M.Minsky 85
• System of multi-gents
• - Okada 87
• Mind composed of six domains and five levels
• - P.N.Johnson-Laird 88
• Systematization of the results of research
  in CS

66

• The authors model
• Fundamentally, we follow Minskys multi-agent
  model.
• Micro-processor µ-agent and its
  chain-activation are introduced.

67

• µ-agent(
• name (identifier),
• domain (attached),
• input (premise of activation),
• execution (program),
• memory (data),
• description (result),
• output (message))
• Fig.51 Frame representation of
• µ-agent

68

• - Chain activation
• Various functions of mind is executed by a chain
  activation or a series of activations of
  µ-agents.

Recognition
Reasoning
Behavior
Fig.52 Chain activation
69

• Domains of processing
• The mind consists of six domains which function
  as follows
• (1) Recognition
• (2) ReasoningDesign
• (3) Emotion
• (4) Expression
• (5) Memory
• (6) Language

70
Language
Memory
Emotion
ReasoningDesign
Recognition
Expression
Mind(brain)
Sensors
Actuators
Body
(Thirst,hunger,)
External world
(Scene, speech,)
(Behavior, speech,)
Fig.5.3 Domains of processing
71
Language
Memory
Emotions
ReasoningDesign
Expression
Recognition
Mind(brain )
Sensors
Actuators
Body
(Thirst,hunger)
External world
(Behavior, speech,)
(Scene, speech,)
72

• Levels of data
• Along concept formation process
• Level 5 Connected concept
• 4 Simple concept
• 3 Conceptual feature
• 2 Cognitive feature
• 1 Raw data

73
go
Agent
Origin
Connected concept
Shopping( buy, cash/card, store, . . . .
Inside
Movable
is_a
is_a
Human
Car
Vi
Ni
Ai
Primitive concept
Go
High
House
Composed
Roof, Wall, Room, . . . .
Movement_ from_inside_ to_outside, . . . .
Difference_in_ length, . . . .
Conceptual feature
Associated

,
,
Cognitive feature
Extracted

Raw data
,
,
(Internal)
Visual
(External)

,
,
(Substance)
(Event)
(Attribute)
Fig. 5?4 Levels of data
74
Aesopworld Project

• - Implementation of our theory
• - Simulation of the physical and mental
  activities of the protagonists of Aesop Fables,
• e.g. The Fox and the Grapes

75
Language
desire relieve thirst
plan eat fruits
plan Drink water
Reasoning Design
goal relieve thirst
Emotion
Memory
Controller
Plan- knowledge Nature- reasoning
Planner
physiology thirst
Plan generator
Recognition
Expression
Simulator
Evaluator
Reasoner
reasoning human near pond
reasoning water in pond
reasoning pot in house
Sensors
Actuators
Fig.5?5 Chain activation of µ-agents
76
Language
plan go to mansion to drink water
ReasoningDesign
Emotion
Memory
Plan- knowledge Nature- reasoning
Controller
action movement to mansion
Planner
Plan generator
Recognition
Expression
Simulator
Evaluator
Reasoner
Sensors
Actuators
77
Experiments

• Main system
• Four PCs and fifteen interpreters (subdomains)

Sub- domain 7
Sub- domain 6
Sub- domain 8
Sub-domain 1
Sub- domain 3
Sub- domain 2
Sub- domain 4
Sub- domain 5
PC1 Turbo Linux 8
PC2 Turbo Linux 8
LAN
Message server
Sub- domain 15
Sub- domain 10
Sub- domain 12
Sub- domain 9
Sub- domain 11
Sub- domain 14
Sub- domain 13
PC3 RedHat Linux 5.2J
PC4 RedHat Linux 5.2J )
Fig.56 Composition
78
Fig.57 Snapshot1
79
Fig. 58 Snapshot2
80
Fig.5?9 Animation
81

• Generated monolog by the Fox
• Its very hot today. Im on the animal trail 300
  meters from the intersection. Im very thirsty.
  Id like to relieve my thirst in a safe way in a
  hurry.
• Ill search for and drink water. Ill go home.
  My home is far. I give up going there. Ill go
  under the bridge. Its far. I give up going
  there I study other ways.
• Ill search for a place with water. I remember a
  pond. Ill find it. I remember the B pond. Its
  in the Aesopworld. Ill go there. A hunters
  lodge is close to it. Hell probably be in it. He
  is man. Man is dangerous. I give up going there
• Ill eat watery foods. Ill search for and eat
  fruits

82
Table 5.2 Comparison with Minsky and
Johnson-Laird
Minsky 85 Okada 87 Johnson-Laird 88
Approach Bottom up Top down Top down
Domains Many Six Six
Levels many Five Many
Technology Multi-agents Multi-agents Turing machine
Experi-ment No Yes No
83
Evaluation

• Various mental activities discussed in CS and AI
  could be captured by our six domains and five
  levels.
• An interface to physiology is put at the level of
  raw data.
• This model can be implemented if the number of
  µ-agents is less than ten thousands.
• Our integrated intelligence took the lead in
  verifying its validation by experiments.

84
Main publications

• 1990 N.Okada and T.Endo Story Generation Based
  on Dynamics of the Mind, Computational
  Intelligence, Vol.8, No.1, pp.123-160.
• 1996 N.Okada Integrating Vision, Motion, and
  Language through the Mind, Artificial
  Intelligence Reiview, Vol.10, pp.209-234.

85
6.Residual problems and social applications

• Problems
• - Learning through experiences
• - Implementation to robots
• Applications
• - Support agents for education or diagnosis
• - Partner of handicapped/elder people

86
7.Conclusions

• Concepts of substance, attribute, event, and
  space/time are systematically analyzed and
  classified.
• A system for NLU of picture sequences were
  constructed.
• Primitive emotions were analyzed and implemented
  in the tasks of action and dialog planning.

87

• A computer model of the mind with six domains of
  processing and five levels of data was proposed,
  and was implemented with twelve hundreds µ-agents
  on computers.
• These results led us to a conclusion that an
  infrastructure to construct complex intelligence
  covering many subfields could be obtained.