Structural Learning

1
Structural Learning

• COMP 8/7745
• November 5 2003

2
Structural Learning

• What is Structural Learning?
• The acquired knowledge is encoded into the
  structure
• Why Structural Learning?
• Good approximation with arbitrary accuracy
• No priliminary knowledge is required
• BUT
• No explanation of the obtained results
• Problems with local minima/ structure selection
• Overtraining can occur

3
Structural Learning in neural networks

• Optimum structure is selected
• Nodes, connections
• Knowledge is generated suring training
• In the form of fuzzy rules
• Interpretation of the obtained results
• Making the black-box transparent
• Improved performance
• Better generalization, robustness to noise

4
Implementing Structural Learning

• Regularization / penalty concept
• Cost function
• J SSE lambda COMPLEXITY
• Possible approaches to regularization
• Entropy
• Optimum brain damage (OBD)
• Forgetting
• Lateral inhibition, etc.

5
Learning with forgetting in neural networks

• Cost function
• J SSELambda ?w_ij
• Learning rule
• ?w_ij ? w_ij lambasign(w_ij)
• Simple to implement
• Very good performance as compared to other
  regularization methods
• (Ishikawa, 1996. Kozma et al. 1996)

6
Rule extraction by structural learning

• Knowledge is generated during training
• In the form of crisp or fuzzy rules
• The NNs skeleton structure represents the rules
• Additional advantages
• Optimum structure is selected
• Improved performance (better generalization and
  robustness to noise)

7
Structural Learning in neural networks

• Original (fully-connected)
• Structured
• Input coordinates e.g., frequency poit, time
  step

8
Representation of the dynamics in the structure

• major cycles identified
• 3T 2T T

9
Embedding in connectionist environment

• Connectionins tools help to
• identify physical parameters
• deeper understanding natural processes
• alternative/complem. to embedology
• extract knowledge on the system behaviour
• in the form of fuzzy rules

10
Discrete representation of rules

• 3 types of weights/nodes after structural
  learning
• Survivors small fraction of all
• Decayed 90
• Strugglers on the interface
• Separation of groups
• Survivorslt-gt decayed by orders of magnitudes
  (100-times or more)
• Discrete / integer representation is feasible
• When environment changes
• 3-level structure diminishing
• Rearrangement among levels