1. Psychology 209 – Winter, 2018 March 6, 2018
Structure and Creativity in Neural Networks: HRL and Generative Models of Sequential Behavior
Psychology 209 – Winter, 2018
March 6, 2018

2. The philosophy behind the approach
Learning vs. Handcrafted
General vs. Specific
Grounded vs. Logic Based
Active vs. Passive

3. Intuition and Creativity
Implicit knowledge acquired through experience but not consciously expressible
The experience and quality of this knowledge can be verified behaviorally
Creativity
The ability to synthesize knowledge to produce a novel or original behavior
Demis’ example from AlphaGo

4. Where does structure come from in behavior?
Hierarchy
A journey of a thousand miles begins with …
Where do hierarchies come from?
Act-R and Soar approaches
Place sub-goals in working memory
Use pattern matching to select next action contingent on goals and WM contents
Create chunks to deal with repeated sub-sequences
Chunks
Subroutines that involve many subactions
In general, these chunks need to be context and content sensitive
E.g. when performing multi-column arithmetic, the routine applied to each column has to take into account the numbers in the column, and can involve some flexibility
Options
Similar to Chunking, but involves RL
Tabular example from Botvinick et al
Issues with the Options approach

6. Tables and subtables (hidden in this diagram)

8. Issues with the options approach
Option discovery
Evolution
Statistics of sequential experience
Analysis of the problem space
Intrinsic reward and pseudo-reward
The social environment
Context sensitivity and ‘shared structure’
Spreading different things on various other things
Grasping different things for different purposes
Finding truly novel solutions

9. DRAW model of Gregor et al

10. Generation process and results