1. Using Hierarchical Reinforcement Learning to Solve a Problem with Multiple Conflicting Sub-problems

2. Reinforcement Learning Involves an agent interacting with an environment The agent can be in one of various states in the environment The agent is not told which action is correct, but is given a measure of an action for a given state After a while the agent develops a policy

3. The curse As complexity of the environment grows, state space increases exponentially We can try to cleverly reduce state space Hierarchical reinforcement learning

4. Hierarchical Reinforcement Learning A complex problem can often be broken up into multiple conflicting sub-problems Hierarchical reinforcement learning can handle this Deals with each sub-problem separately using reinforcement learning Decides which sub-problem to attempt next using reinforcement learning

5. A Practical Example: The Mars Lander Perform Various Conflicting Tasks: Explore the terrain Collect soil samples Return to base for refuelling

6. My Project Apply hierarchical reinforcement learning to a complex problem Consist of an agent existing in an environment where it will have to achieve an overall goal Agent will be a primitive creature trying to survive in the wilderness

7. My Project The overall goal will be for the creature to remain happy or comfortable in the wilderness Overall goal can be divided into sub-goals These sub-goals will be: –Eating food –Drinking water –Resting under a Shelter –Repairing Shelter –Avoiding hazards

8. The Gridworld

9. Motivation for this approach X pos Y pos Hunger Thirst Fatigue Shelter Condition 13 x 13 x 10 x 10 x 10 x 10 = 1690000 Possible states Sub-goals separated out: (Xpos, Ypos, hunger), (Xpos, Ypos, Thirst) (Xpos, Ypos, Fatigue), (Xpos, Ypos, Shelter Condition) (13 x 13 x 10) x 4 =1690 x 4 = 6760 Possible states