Relational Learning: from Yesterday to Tomorrow

1
Relational Learning from Yesterday to Tomorrow

• Lorenza Saitta
• Università del Piemonte Orientale
• saitta_at_mfn.unipmn.it

2
Why did I choose this topic?

• This is how I knew Yves existence
• As a team, we have worked on the topic since its
  beginning, knowing thus its hopes and deceptions
• The topic is fascinating and rich of fundamental
  issues for understanding learning in both humans
  and machines
• Warning
• This is NOT an overview of the field, but a
  collection of personal reflexions

3
Relation Learning has a peculiar history

• Negative point it is conceptually hard and
  computationally demanding
• Positive point it is apparently more close to
  human learning, w.r.t. previous Pattern
  Recognition approaches (even the syntactic ones)
• Michalskis  Human comprehensibility principle 
• Instead of a progression  easy ? difficult  in
  the research topics, we have seen, at the
  beginning, researchers launching themselves into
  RL with great enthusiasm, but a scarse sense of
  the feasible

4
The power of names

• The consequence is a history of déjà vu, of dead
  ends, and of difficulties periodically
  re-emerging, that researchers try to solve again
  and again
• Solution Restart and change the label
• FOL Learning ILP
  SRL
• It would be unfair to say that at each restart
  everything is the same. Some novelties are added
  each time
• As restart is an effective technique in search,
  maybe it will be also effective for RL in the
  long term

5
The ambition of the origins

• ML started officially in 1980 Pittsburgh
  workshop
• RL is much older 1970
• Meltzer Inverting deduction E1, E1 V
  E2
• Morgan    
• Plotkin ?-subsumption
• Winston ARCH -gt Near-misses
• Vere TOTH -gt Counterfactuals
• Conceptually interesting and still up-to-date
  today
• Computationally infeasible, but, at the time,
  this was not an issue. The proposed algorithms
  were not meant to work they were ideological and
  explorative in nature.
• It is quite surprising that the first learning
  approaches (except Samuels checker) were
  actually relational.
• RL attracted researchers as  honey attract
  flies  as we say in Italy

6
Keeping feet on the ground

• But learning IS meant to work, even if it is
  relational
• Under Michalskis influence, between 1980 and
  1990 many systems have been designed and
  implemented both in Europe and in the USA
• Kodratoff et al.
• Giordana et al. Learning from a relational
  datbase, 1988
• Esposito et al.
• Morik et al.
• But not just learning systems but source of
  conceptual innovations
• Mitchells and De Jongs EBL
• Ganascia Zuckers  morions 
• Carbonells learning in planning
• Some concrete results were obtained

7
Generality

• Crucial issue how to define the generality
  relation if FOL?
• Michalski
• Buntine
• Niblett
• Flach -gt  Yves lives in France ,  Yves lives
  in Paris 
• Kodratoff
• Console Saitta -gt Freges theory of concepts
• Generality ?
  Information content
• The discussion died, but it is a pity, because
  the field got impoverished received view
  ?-subsumption or covering
• Relationships with abduction and with abstraction

8
Inductive Logic Programming

• Muggletons CIGOL Quinlans FOIL
  
• ILP
• Automatic programming
• Strongly logic-oriented. Mostly based on
  ?-subsumption. High computational complexity
• h(x,y) name of a relation (extension -gt
  intension)
• Attributes -gt Background knowledge
• Needs powerful bias or human help
• A theory of logical learning more than a
  practical approach.
• Some success in applications.
• Almost separated from mainstream ML.
• Extensions towards Reinforcement learning,
  Clustering, Neural Networks.

9
Plateau and Phase Transitions

• The covering test shows a phase transition in a
  range of parameters interesting for practical
  learning approaches
• Concept with more than 4 chained variables cannot
  be learned due to the extremely high
  computational complexity of the test.
• Top-down, hypothesis-and-test-based relational
  learning cannot go beyond stringent limits on the
  complexity of examples and hypotheses.
• Could top-down, data-driven approaches overcome
  the limits?

10
Statistical relational learning
ILP
Probability
SRL

• Started by considering relations among examples
• Evolved toward probabilitic logic. Probabilistic
  logics (PL) did not produce anything usable in
  the past.
• By putting together two difficult subjects, is it
  possible that something simple will come out? By
  interference ?
• Several interesting ideas, but no definite
  solution. Field at the beginning.
• Domingos and Richardson (Markov networks)
• Kersting and De Raedt (Bayest neworks)
• Poole (Lifted inference)
• Koller et al. (Probabilistic nertworks -
  Database-oriented approach)

11
P(X), Q(x) -gt R(X,Y) P(X), Q(x) -gt S(X,Y) Q(X),
Q(Y) -gt V(X,Y) -gt R(X,Y) a, b
12
Future?
13
I believe that the world market can be satured
by maybe five computers
Thomas
Watson, IBM Chairman, 1943 A 640 KB memory
should be sufficient for everyone"

Bill Gates, 1981 "Internet will undergo a
catastrophic collapse in 1996"
Robert
Metcalfe, Ethernets inventor
14
Guesses

• A radically new approach is needed to obtain
  substantial steps ahead
• NO Logic, but get close to (take inspiration
  from)
• Cognitive Sciences
• Human memory and learning
• More complex concept representation (Barsalou)
• Do the limitations to machine learning also apply
  to human learning?
• Complex Systems
• Graphs and networks
• Agent Based Modeling
• Simulation
• Statistical Phisics
• Ensemble phenomena
• Collective learning (emergent phenomenon)
• Abstraction and multi-resolution approaches

15
Guesses

• Investigating the foundations
• Phase transitions
• Kolmogorov complexity
• Chaitins theorem - Algorithmic version of the
  Halt problem
• No program of complexity n can generate a number
  greater than n